A reference implementation for processing the content.log files found at opendata.dwd.de/weather

Related tags

Miscellaneousopendata-content.log-tool
Overview

content.log Tool

A reference implementation for processing the content.log files found at opendata.dwd.de/weather.

Example usage:

wildcards.txt ./get_updated_files.py -w wildcards.txt 2022-01-26T03:00 > updated_files.txt"> 
wget https://opendata.dwd.de/weather/nwp/content.log.bz2
bunzip2 content.log.bz2
echo "./icon-d2/grib/*/t_2m/*" > wildcards.txt
./get_updated_files.py -w wildcards.txt 2022-01-26T03:00 > updated_files.txt

The produced file updated_files.txt will hold all pathnames as given by content.log that are updated since 2022-01-26 03:00:00 UTC according to the file's modification date. Remember that those paths are relative to the directory the content.log.bz2 was in, in this case https://opendata.dwd.de/weather/nwp/.

Also mind that there are multiple servers behind https://opendata.dwd.de which might not be exactly in sync with each other regarding file modification timestamps. Look into the code of get_updated_files.py for a suggestion on how to deal with that.

While this program relies on the file modification timestamp dumped into content.log.bz2, it might be more feasible to process the data reference time that is contained in the filenames instead.

$ ./get_updated_files.py --help
usage: get_updated_files.py [-h] [--content-log CONTENT_LOG_FILE] [--wildcards WILDCARDS_FILE] [--min-delta MIN_DELTA] [--version] UPDATED_SINCE

Filters paths of a DWD Open Data content.log file for entries that have been updated.

positional arguments:
  UPDATED_SINCE         Last time files were checked for updates

optional arguments:
  -h, --help            show this help message and exit
  --content-log CONTENT_LOG_FILE, -f CONTENT_LOG_FILE
                        The decompressed content.log file (default: content.log)
  --wildcards WILDCARDS_FILE, -w WILDCARDS_FILE
                        Filter results by a set of wildcards
  --min-delta MIN_DELTA, -d MIN_DELTA
                        Minimum number of seconds a file needs to be younger than UPDATED_SINCE (default: 60)
  --version             show program's version number and exit
You might also like...
Script de monitoramento das teclas do teclado, salvando todos os dados digitados em um arquivo de log juntamente com os dados de rede.

listenerPython Script de monitoramento das teclas do teclado, salvando todos os dados digitados em um arquivo de log juntamente com os dados de rede.

Vinícius Azevedo 4 Nov 27, 2022
Python Common things by Problem Fighter Library, (Exception, Debug Log, etc.)

In the name of God, the Most Gracious, the Most Merciful. PF-PY-Common Documentation Install and update using pip: pip install -U xxxx Please find the

Problem Fighter 3 Jan 15, 2022
A log likelihood fit for extracting neutrino oscillation parameters

A-log-likelihood-fit-for-extracting-neutrino-oscillation-parameters Minimised the negative log-likelihood fit to extract neutrino oscillation paramete

Vid Homsak 1 Jan 23, 2022
A Python Based Utility for Processing GST-Return JSON Files to Multiple Formats
A Python Based Utility for Processing GST-Return JSON Files to Multiple Formats

GSTR 1/2A Utility by Shan.tk Open Source GSTR 1/GSTR 2A JSON to Excel utility based on Python. Useful for Auditors in Verifying GSTR 1 Return Invoices

Sudharshan TK 1 Oct 8, 2022
A python script that changes your desktop background based on current weather and time of the day.

Desktop background wallpaper, based on current weather and time A python script that changes your computer's desktop background based on current weath

Maj Gaberšček 1 Nov 16, 2021
A simple weather app.

keather A simple weather app. This is currently not finished. Dependencies: yay -S python-beautifulsoup4 tk

null 1 Jan 9, 2022
Use a real time weather API to apply wind to your mouse cursor.

wind-cursor Use a real time weather API to apply wind to your mouse cursor. Requirements PyAutoGUI pyowm Usage This program uses the OpenWeatherMap AP

Andreas Schmid 1 Feb 7, 2022
Python package for reference counting native pointers
Python package for reference counting native pointers

refcount master: testing: This package is primarily for managing resources in native libraries, written for instance in C++, from Python. While it boi

CSIRO Hydroinformatics 2 Nov 3, 2022
Project Faros is a reference implimentation of Red Hat OpenShift 4 on small footprint, bare-metal clusters.
Project Faros is a reference implimentation of Red Hat OpenShift 4 on small footprint, bare-metal clusters.

Project Faros Project Faros is a reference implimentation of Red Hat OpenShift 4 on small footprint, bare-metal clusters. The project includes referen

project: Faros 9 Jul 18, 2022
Comments
  • changing urljoin to os.path.join

    changing urljoin to os.path.join

    The behavior of urljoin is on my opionion not constructive, because of its description "Construct a full (“absolute”) URL by combining a “base URL” (base) with another URL (url). Informally, this uses components of the base URL, in particular the addressing scheme, the network location and (part of) the path, to provide missing components in the relative URL." (https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urljoin)

    opened by nklever 0
Releases(v2.0.0)

  • v2.0.0(Jan 27, 2022)

    This new major release introduces support for timezones. Also, the deprecated feature --wildcards was removed and instead the use of external tools like bzgrep on the compressed file content.log.bz2 are recommended. The content.log file was promoted to the

    Changes

    • CONTENT_LOG_FILE is now a positional argument which defaults to STDIN when omitted
    • --updated-since is now a required option

    New features

    • supply multiple CONTENT_LOG_FILEs as positional arguments to parse their combined content
    • --updated-since now supports timezones when specified as +HH:MM while assuming the system's time zone if not specified; date-times in content.log are always treated as UTC
    Source code(tar.gz)
    Source code(zip)

  • v1.1.0(Jan 27, 2022)

  • v1.0.0(Jan 27, 2022)

    This is the first fully functional version. It implements minimal features with focus on readability to also act as a form of documentation around the quirks around DWD's Open Data emerging from having two seperate servers sitting behind opendata.dwd.de.

    Features

    • List the paths contained in a content.log file for entries that have likely been updated since a given date-time
    • Filter those results by one or more --wildcards
    Source code(tar.gz)
    Source code(zip)
Owner
Deutscher Wetterdienst (DWD)
Deutscher Wetterdienst (DWD)
GitHub Repository
A Blender addon to enable reloading linked libraries from UI.

library_reload_linked_libraries A Blender addon to enable reloading linked libraries from UI.

3 Nov 27, 2022
an opensourced roblox group finder writen in python 100% free and virus-free

Roblox-Group-Finder an opensourced roblox group finder writen in python 100% free and virus-free note : if you don't want install python or just use w

mollomm1 1 Nov 11, 2021
Gerenciador de processos e registros pessoais do Departamento de Fiscalização de Produtos Controlados.

CRManager Gerenciador de processos e registros pessoais do Departamento de Fiscalização de Produtos Controlados. Descrição Este projeto tem como objet

Wolfgang Almeida 1 Nov 15, 2021
An universal linux port of deezer, supporting both Flatpak and AppImage

Deezer for linux This repo is an UNOFFICIAL linux port of the official windows-only Deezer app. Being based on the windows app, it allows downloading

Aurélien Hamy 154 Jan 06, 2023
Convex Optimisation MVA course - Assignment

Convex Optimisation MVA course - Assignment This repository contains the coding files of the third assignment in the MVA Convex Optimisation course. U

1 Nov 27, 2021
SQL centered, docker process running game

REQUIREMENTS Linux Docker Python/bash set up image "docker build -t game ." create db container "run my_whatever/game_docker/pdb create" # creating po

1 Jan 11, 2022
Tiling manager which runs on top of EWMH window managers.

PyTyle is an extremely versatile and extensible tiling manager that is meant to be used on top of EWMH window managers. Its feature set was modeled af

55 Jul 29, 2021
Data wrangling & common calculations for results from qMem measurement software

qMem Datawrangler This script processes output of qMem measurement software into an Origin ® compatible *.csv files and matplotlib graphs to quickly v

Julian 1 Nov 30, 2021
TimeWizard - A script that generates every single Time Wizard EDOPRO lflist possible

EDOPRO F&L list generator This project is just a script that generates every sin

Diamond Dude 2 Sep 28, 2022
A comprensive software collection for nmea manipulation

nmeatoolkit A comprensive software collection for nmea manipulation; it includes a library and a collections of command line tools. Library pipes: con

Davide Gessa 1 Sep 14, 2022
Pulse sequence builder and compiler for q1asm

q1pulse Pulse sequence builder and compiler for q1asm. q1pulse is a simple library to compile pulse sequence to q1asm, the assembly language of Qblox

Sander de Snoo 3 Dec 14, 2022
2 Way Sync Between Notion Database and Google Calendar

Notion-and-Google-Calendar-2-Way-Sync 2 Way Sync Between a Notion Database and Google Calendar WARNING: This repo will be undergoing a good bit of cha

248 Dec 26, 2022
A tool for generating skill map/tree like diagram

skillmap A tool for generating skill map/tree like diagram. What is a skill map/tree? Skill tree is a term used in video games, and it can be used for

Yue 98 Jan 07, 2023
A Web app to Cross-Seed torrents in Deluge/qBittorrent/Transmission

SeedCross A Web app to Cross-Seed torrents in Deluge/qBittorrent/Transmission based on CrossSeedAutoDL Require Jackett Deluge/qBittorrent/Transmission

ccf2012 76 Dec 19, 2022
Control your gtps with gtps-tools!

Note Please give credit to me! Do not try to sell this app, because this app is 100% open source! Do not try to reupload and rename the creator app! S

Jesen N 6 Feb 16, 2022
K2HASH Python library - NoSQL Key Value Store(KVS) library

k2hash_python Overview k2hash_python is an official python driver for k2hash. Install Firstly you must install the k2hash shared library: curl -o- htt

Yahoo! JAPAN 3 Oct 19, 2022
Run CodeServer on Google Colab using Inlets in less than 60 secs using your own domain.

Inlets Colab Run CodeServer on Colab using Inlets in less than 60 secs using your own domain. Features Optimized for Inlets/InletsPro Use your own Cus

2 Dec 30, 2021
Shows VRML team stats of all players in your pubs

VRML Team Stat Searcher Displays Team Name, Team Rank (Worldwide), and tier of all the players in your pubs. GUI WIP: Only username search works (for

Hamish Burke 2 Dec 22, 2022
Mpis-ex7 - Implementation of tasks 1, 2, 3 for Metody Probabilistyczne i Statystyka Lista 7

Implementations of task 1, 2 and 3 from here Author: Maciej Bazela Index: 261743 Each task was implemented in Python 3. I've used Cython to speed up e

Maciej Bazela 1 Feb 27, 2022
Draw random mazes in python

a-maze Draw random mazes in python This program generates and draws a rectangular maze, with an entrance on one side and one on the opposite side. The

Andrea Pasquali 1 Nov 21, 2021