A reference implementation for processing the content.log files found at opendata.dwd.de/weather

Last update: Nov 26, 2022

Related tags

Miscellaneous opendata-content.log-tool

Overview

content.log Tool

A reference implementation for processing the content.log files found at opendata.dwd.de/weather.

Example usage:

wildcards.txt ./get_updated_files.py -w wildcards.txt 2022-01-26T03:00 > updated_files.txt">

wget https://opendata.dwd.de/weather/nwp/content.log.bz2
bunzip2 content.log.bz2
echo "./icon-d2/grib/*/t_2m/*" > wildcards.txt
./get_updated_files.py -w wildcards.txt 2022-01-26T03:00 > updated_files.txt

The produced file updated_files.txt will hold all pathnames as given by content.log that are updated since 2022-01-26 03:00:00 UTC according to the file's modification date. Remember that those paths are relative to the directory the content.log.bz2 was in, in this case https://opendata.dwd.de/weather/nwp/.

Also mind that there are multiple servers behind https://opendata.dwd.de which might not be exactly in sync with each other regarding file modification timestamps. Look into the code of get_updated_files.py for a suggestion on how to deal with that.

While this program relies on the file modification timestamp dumped into content.log.bz2, it might be more feasible to process the data reference time that is contained in the filenames instead.

$ ./get_updated_files.py --help
usage: get_updated_files.py [-h] [--content-log CONTENT_LOG_FILE] [--wildcards WILDCARDS_FILE] [--min-delta MIN_DELTA] [--version] UPDATED_SINCE

Filters paths of a DWD Open Data content.log file for entries that have been updated.

positional arguments:
  UPDATED_SINCE         Last time files were checked for updates

optional arguments:
  -h, --help            show this help message and exit
  --content-log CONTENT_LOG_FILE, -f CONTENT_LOG_FILE
                        The decompressed content.log file (default: content.log)
  --wildcards WILDCARDS_FILE, -w WILDCARDS_FILE
                        Filter results by a set of wildcards
  --min-delta MIN_DELTA, -d MIN_DELTA
                        Minimum number of seconds a file needs to be younger than UPDATED_SINCE (default: 60)
  --version             show program's version number and exit

Script de monitoramento das teclas do teclado, salvando todos os dados digitados em um arquivo de log juntamente com os dados de rede.

listenerPython Script de monitoramento das teclas do teclado, salvando todos os dados digitados em um arquivo de log juntamente com os dados de rede.

4 Nov 27, 2022

Python Common things by Problem Fighter Library, (Exception, Debug Log, etc.)

In the name of God, the Most Gracious, the Most Merciful. PF-PY-Common Documentation Install and update using pip: pip install -U xxxx Please find the

3 Jan 15, 2022

A log likelihood fit for extracting neutrino oscillation parameters

A-log-likelihood-fit-for-extracting-neutrino-oscillation-parameters Minimised the negative log-likelihood fit to extract neutrino oscillation paramete

1 Jan 23, 2022

A Python Based Utility for Processing GST-Return JSON Files to Multiple Formats

GSTR 1/2A Utility by Shan.tk Open Source GSTR 1/GSTR 2A JSON to Excel utility based on Python. Useful for Auditors in Verifying GSTR 1 Return Invoices

1 Oct 8, 2022

A python script that changes your desktop background based on current weather and time of the day.

Desktop background wallpaper, based on current weather and time A python script that changes your computer's desktop background based on current weath

1 Nov 16, 2021

A simple weather app.

keather A simple weather app. This is currently not finished. Dependencies: yay -S python-beautifulsoup4 tk

1 Jan 9, 2022

Use a real time weather API to apply wind to your mouse cursor.

wind-cursor Use a real time weather API to apply wind to your mouse cursor. Requirements PyAutoGUI pyowm Usage This program uses the OpenWeatherMap AP

1 Feb 7, 2022

Python package for reference counting native pointers

refcount master: testing: This package is primarily for managing resources in native libraries, written for instance in C++, from Python. While it boi

2 Nov 3, 2022

Project Faros is a reference implimentation of Red Hat OpenShift 4 on small footprint, bare-metal clusters.

Project Faros Project Faros is a reference implimentation of Red Hat OpenShift 4 on small footprint, bare-metal clusters. The project includes referen

9 Jul 18, 2022

Comments

changing urljoin to os.path.join

The behavior of urljoin is on my opionion not constructive, because of its description "Construct a full (“absolute”) URL by combining a “base URL” (base) with another URL (url). Informally, this uses components of the base URL, in particular the addressing scheme, the network location and (part of) the path, to provide missing components in the relative URL." (https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urljoin)

opened by nklever 0

Releases(v2.0.0)

v2.0.0(Jan 27, 2022)
This new major release introduces support for timezones. Also, the deprecated feature --wildcards was removed and instead the use of external tools like bzgrep on the compressed file content.log.bz2 are recommended. The content.log file was promoted to the

Changes

CONTENT_LOG_FILE is now a positional argument which defaults to STDIN when omitted

--updated-since is now a required option

New features

supply multiple CONTENT_LOG_FILEs as positional arguments to parse their combined content

--updated-since now supports timezones when specified as +HH:MM while assuming the system's time zone if not specified; date-times in content.log are always treated as UTC

Source code(tar.gz)
Source code(zip)
v1.1.0(Jan 27, 2022)
Changes

added --url-base option

deprecated --wildcards

Source code(tar.gz)
Source code(zip)
v1.0.0(Jan 27, 2022)
This is the first fully functional version. It implements minimal features with focus on readability to also act as a form of documentation around the quirks around DWD's Open Data emerging from having two seperate servers sitting behind opendata.dwd.de.

Features

List the paths contained in a content.log file for entries that have likely been updated since a given date-time

Filter those results by one or more --wildcards

Source code(tar.gz)
Source code(zip)

Owner

Deutscher Wetterdienst (DWD)

GitHub Repository

Pylexa - Artificial Assistant made with Python

Pylexa - Artificial Assistant made with Python Alexa is a famous artificial assistant used massively across the world. It is a substitute of Alexa whi

$\_ＰＲＯＴＩＫ_/$ 4 Nov 03, 2021

CPython extension implementing Shared Transactional Memory with native-looking interface

21 Jul 22, 2022

Gerenciador de processos e registros pessoais do Departamento de Fiscalização de Produtos Controlados.

CRManager Gerenciador de processos e registros pessoais do Departamento de Fiscalização de Produtos Controlados. Descrição Este projeto tem como objet

1 Nov 15, 2021

This package tries to emulate the behaviour of syntax proposed in PEP 671 via a decorator

Late-Bound Arguments This package tries to emulate the behaviour of syntax proposed in PEP 671 via a decorator. Usage Mention the names of the argumen

0 Feb 06, 2022

FindUncommonShares.py is a Python equivalent of PowerView's Invoke-ShareFinder.ps1 allowing to quickly find uncommon shares in vast Windows Domains.

FindUncommonShares The script FindUncommonShares.py is a Python equivalent of PowerView's Invoke-ShareFinder.ps1 allowing to quickly find uncommon sha

184 Jan 03, 2023

Object-oriented programming (OOP) is a method of structuring a program by bundling related properties and behaviors into individual objects. In this tutorial, you’ll learn the basics of object-oriented programming in Python.

06_Python_Object_Class Introduction 👋 Objected oriented programming as a discipline has gained a universal following among developers. Python, an in-

239 Dec 20, 2022

API wrapper for VCS hosting system.

PythonVCS API wrapper for VCS hosting system. Supported platforms Gitea Github, Gitlab, Bitbucket support will not, until that packages is not updated

1 Apr 02, 2022

📙 Super lightweight function registries for your library

catalogue: Super lightweight function registries for your library catalogue is a tiny, zero-dependencies library that makes it easy to add function (o

139 Jan 02, 2023

CoreSE - basic of social Engineering tool

Core Social Engineering basic of social Engineering tool. just for fun :) About First of all, I must say that I wrote such a project because of my int

7 Jun 10, 2022

Analyzes crypto candles over a set time period and then trades based on winning patterns found

patternstrade Analyzes crypto candles over a set time period and then trades based on winning patterns found. Heavily customizable. Warning: This was

14 May 29, 2022

Logging-monitoring-instrumentation - A brief repository on logging monitoring and instrumentation in Python

logging-monitoring-instrumentation A brief repository on logging monitoring and

6 Feb 17, 2022

The Blinker Herald includes helpers to easily emit signals using the excellent blinker library.

Blinker Herald The Blinker Herald includes helpers to easily emit signals using the excelent blinker library. Decorate a function or method with @blin

7 Nov 03, 2022

Apilytics for Python - Easy API analytics for Python backends

apilytics-python Installation Sign up and get your API key from https://apilytics.io - we offer a completely free trial with no credit card required!

6 Sep 29, 2022

A browser login credentials thief for windows and Linux

Thief 🦹🏻 A browser login credentials thief for windows and Linux Python script to decrypt login credentials from browsers in windows or linux Decryp

1 Dec 13, 2021

use Notepad++ for real-time sync after python appending new log text

FTP远程log同步工具使用Notepad++配合来获取实时更新的log文档效果适用于FTP协议的log远程同步工具,配合MT管理器开启FTP服务器使用,通过Notepad++监听文本变化,更便捷的使用电脑查看方法注入打印后的信息功能过滤器对每行要打印的文本使用回调函数筛选，支持链式调用

1 Oct 17, 2021

poro is a LCU interface to change some lol's options.

poro is a LCU interface to change some lol's options. with this program you can: change your profile icon change your profiel background image ch

2 Jan 05, 2022

A python package for batch import of resume attachments to be parsed in HrFlow.

HrFlow Importer Description A python package for batch import of resume attachments to be parsed in HrFlow. hrflow-importer is an open-source project

3 Nov 15, 2022

The fetch of the delegator list and the input of the epoch nonce need to be done independently

raffle The fetch of the delegator list and the input of the epoch nonce need to be done independently. Get the list of delegators at the epoch change.

1 Dec 15, 2021

Programming in Bioinformatics, Block 3

Programming in Bioinformatics - Block 3 I. Setting up Environment and Running the Code Create the environment using the pibi_block3.yml file with the

2 Dec 10, 2021

Pyrmanent - Make all your classes permanent in a flash 💾

Pyrmanent A base class to make your Python classes permanent in a flash. Features Easy to use. Great compatibility. No database needed. Ask for new fe

4 Jan 07, 2022