A reference implementation for processing the content.log files found at opendata.dwd.de/weather

Last update: Nov 26, 2022

Related tags

Miscellaneous opendata-content.log-tool

Overview

content.log Tool

A reference implementation for processing the content.log files found at opendata.dwd.de/weather.

Example usage:

wildcards.txt ./get_updated_files.py -w wildcards.txt 2022-01-26T03:00 > updated_files.txt">

wget https://opendata.dwd.de/weather/nwp/content.log.bz2
bunzip2 content.log.bz2
echo "./icon-d2/grib/*/t_2m/*" > wildcards.txt
./get_updated_files.py -w wildcards.txt 2022-01-26T03:00 > updated_files.txt

The produced file updated_files.txt will hold all pathnames as given by content.log that are updated since 2022-01-26 03:00:00 UTC according to the file's modification date. Remember that those paths are relative to the directory the content.log.bz2 was in, in this case https://opendata.dwd.de/weather/nwp/.

Also mind that there are multiple servers behind https://opendata.dwd.de which might not be exactly in sync with each other regarding file modification timestamps. Look into the code of get_updated_files.py for a suggestion on how to deal with that.

While this program relies on the file modification timestamp dumped into content.log.bz2, it might be more feasible to process the data reference time that is contained in the filenames instead.

$ ./get_updated_files.py --help
usage: get_updated_files.py [-h] [--content-log CONTENT_LOG_FILE] [--wildcards WILDCARDS_FILE] [--min-delta MIN_DELTA] [--version] UPDATED_SINCE

Filters paths of a DWD Open Data content.log file for entries that have been updated.

positional arguments:
  UPDATED_SINCE         Last time files were checked for updates

optional arguments:
  -h, --help            show this help message and exit
  --content-log CONTENT_LOG_FILE, -f CONTENT_LOG_FILE
                        The decompressed content.log file (default: content.log)
  --wildcards WILDCARDS_FILE, -w WILDCARDS_FILE
                        Filter results by a set of wildcards
  --min-delta MIN_DELTA, -d MIN_DELTA
                        Minimum number of seconds a file needs to be younger than UPDATED_SINCE (default: 60)
  --version             show program's version number and exit

Script de monitoramento das teclas do teclado, salvando todos os dados digitados em um arquivo de log juntamente com os dados de rede.

listenerPython Script de monitoramento das teclas do teclado, salvando todos os dados digitados em um arquivo de log juntamente com os dados de rede.

4 Nov 27, 2022

Python Common things by Problem Fighter Library, (Exception, Debug Log, etc.)

In the name of God, the Most Gracious, the Most Merciful. PF-PY-Common Documentation Install and update using pip: pip install -U xxxx Please find the

3 Jan 15, 2022

A log likelihood fit for extracting neutrino oscillation parameters

A-log-likelihood-fit-for-extracting-neutrino-oscillation-parameters Minimised the negative log-likelihood fit to extract neutrino oscillation paramete

1 Jan 23, 2022

A Python Based Utility for Processing GST-Return JSON Files to Multiple Formats

GSTR 1/2A Utility by Shan.tk Open Source GSTR 1/GSTR 2A JSON to Excel utility based on Python. Useful for Auditors in Verifying GSTR 1 Return Invoices

1 Oct 8, 2022

A python script that changes your desktop background based on current weather and time of the day.

Desktop background wallpaper, based on current weather and time A python script that changes your computer's desktop background based on current weath

1 Nov 16, 2021

A simple weather app.

keather A simple weather app. This is currently not finished. Dependencies: yay -S python-beautifulsoup4 tk

1 Jan 9, 2022

Use a real time weather API to apply wind to your mouse cursor.

wind-cursor Use a real time weather API to apply wind to your mouse cursor. Requirements PyAutoGUI pyowm Usage This program uses the OpenWeatherMap AP

1 Feb 7, 2022

Python package for reference counting native pointers

refcount master: testing: This package is primarily for managing resources in native libraries, written for instance in C++, from Python. While it boi

2 Nov 3, 2022

Project Faros is a reference implimentation of Red Hat OpenShift 4 on small footprint, bare-metal clusters.

Project Faros Project Faros is a reference implimentation of Red Hat OpenShift 4 on small footprint, bare-metal clusters. The project includes referen

9 Jul 18, 2022

Comments

changing urljoin to os.path.join

The behavior of urljoin is on my opionion not constructive, because of its description "Construct a full (“absolute”) URL by combining a “base URL” (base) with another URL (url). Informally, this uses components of the base URL, in particular the addressing scheme, the network location and (part of) the path, to provide missing components in the relative URL." (https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urljoin)

opened by nklever 0

Releases(v2.0.0)

v2.0.0(Jan 27, 2022)
This new major release introduces support for timezones. Also, the deprecated feature --wildcards was removed and instead the use of external tools like bzgrep on the compressed file content.log.bz2 are recommended. The content.log file was promoted to the

Changes

CONTENT_LOG_FILE is now a positional argument which defaults to STDIN when omitted

--updated-since is now a required option

New features

supply multiple CONTENT_LOG_FILEs as positional arguments to parse their combined content

--updated-since now supports timezones when specified as +HH:MM while assuming the system's time zone if not specified; date-times in content.log are always treated as UTC

Source code(tar.gz)
Source code(zip)
v1.1.0(Jan 27, 2022)
Changes

added --url-base option

deprecated --wildcards

Source code(tar.gz)
Source code(zip)
v1.0.0(Jan 27, 2022)
This is the first fully functional version. It implements minimal features with focus on readability to also act as a form of documentation around the quirks around DWD's Open Data emerging from having two seperate servers sitting behind opendata.dwd.de.

Features

List the paths contained in a content.log file for entries that have likely been updated since a given date-time

Filter those results by one or more --wildcards

Source code(tar.gz)
Source code(zip)

Owner

Deutscher Wetterdienst (DWD)

GitHub Repository

Simple Python tool to check if there is an Office 365 instance linked to a domain.

o365chk.py Simple Python script to check if there is an Office365 instance linked to a particular domain.

37 Jan 02, 2023

Media Cloud Outlet Filtering

Using ABYZ and Media-Bias Fact-Check outlet databases, I've provided outlet CSV files for both and scripts to intended to match Media Cloud files to respective outlets.

1 Feb 02, 2022

A compiler for ARM, X86, MSP430, xtensa and more implemented in pure Python

277 Dec 26, 2022

A PDM plugin to publish to PyPI

PDM Publish A PDM plugin to publish to PyPI NOTE: Consider if you need this over using twine directly Installation If you installed pdm via pipx: pipx

20 Aug 06, 2022

清晰易读的7x7像素点阵中文字体和取模工具

FontChinese7x7 上古神器 III : 7x7像素点阵中文字体想要在低分辨率屏幕上显示中文, 却发现中文字体实在是太大? 找了全网发现字体库最小也只有12x12? 甚至是好不容易找到了一个8x8字体, 结果发现字体收费且明确说明不得以任何形式嵌入到软件当中? 那就让这个项目来解决你的问

72 Dec 12, 2022

Understanding the field usage of any object in Salesforce

Understanding the field usage of any object in Salesforce One of the biggest problems that I have addressed while working with Salesforce is to unders

1 Dec 14, 2021

Prototype application for GCM bias-correction and downscaling

dodola Prototype application for GCM bias-correction and downscaling This is an unstable prototype. This is under heavy development. Features Nothing!

9 Dec 27, 2022

Example platform plugin that fixes fentry calls in Binja

Example Binja Platform Plugin This is an example Binja platform plugin which fixes up linux kernel module calls to __fentry__. __fentry__ is the linux

2 Oct 07, 2021

Convert Roman numerals to modern numerals and vice-versa

Roman Numeral Conversion Utilities This is a utility module for converting from and to Roman numerals. It supports numbers upto 3,999,999, using the v

1 Dec 17, 2021

serological measurements from multiplexed ELISA assays

pysero pysero enables serological measurements with multiplexed and standard ELISA assays. The project automates estimation of antibody titers from da

5 Aug 06, 2022

Mdisk - 🚧 On Construction 🚧

Mdisk Install For Package pip install mdisk pip install git+https://github.com/HeimanPictures/Mdisk.git Usage You can use this as python module or via

6 Aug 08, 2022

Turn a raspberry pi into a Bluetooth Midi device

PiBluetoothMidSetup This will change serveral system wide packages/configurations Do not run this on your primary machine or anything you don't know h

40 Sep 19, 2022

Gaia: a chrome extension that curates environmental news of a company

Gaia - Gaia: Your Environment News Curator Call for Code 2021 Gaia: a chrome extension that curates environmental news of a company Explore the docs »

4 Mar 19, 2022

Rufus port to linux, writed on Python3

Rufus-for-Linux Rufus port to linux, writed on Python3 Программа будет иметь тот же интерфейс что и оригинал, и тот же функционал. Программа создается

10 May 12, 2022

The refactoring tutorial I wrote for PyConDE 2022. You can also work through the exercises on your own.

Refactoring 101 planet images by Justin Nichol on opengameart.org CC-BY 3.0 Goal of this Tutorial In this tutorial, you will refactor a space travel t

9 Jun 10, 2022

Woltcheck - Python script to check if a wolt restaurant is ready to deliver to your location

woltcheck Python script to check if a wolt restaurant is ready to deliver to you

30 Sep 13, 2022

A pure-Python codified rant aspiring to a world where numbers and types can work together.

Copyright and other protections apply. Please see the accompanying LICENSE file for rights and restrictions governing use of this software. All rights

28 Sep 04, 2022

App and Python library for parsing, writing, and validation of the STAND013 file format.

python-stand013 python-stand013 is a Python app and library for parsing, writing, and validation of the STAND013 file format. Features The following i

3 Nov 09, 2022

This is a project for socks card label validation where the socks card is validated comparing with the correct socks card whose coordinates are stored in the database. When the test socks card is compared with the correct socks card(master socks card) the software checks whether both test and master socks card mathches or not.

Automation_in_socks_label_validation THEME: MACHINE LEARNING This is a project for socks card label validation where the socks card is validated compa

3 Apr 21, 2022

Integration of Hotwire's Turbo library with Flask.

turbo-flask Integration of Hotwire's Turbo library with Flask, to allow you to create applications that look and feel like single-page apps without us

240 Jan 06, 2023