A reference implementation for processing the content.log files found at opendata.dwd.de/weather

Related tags

Miscellaneousopendata-content.log-tool
Overview

content.log Tool

A reference implementation for processing the content.log files found at opendata.dwd.de/weather.

Example usage:

wildcards.txt ./get_updated_files.py -w wildcards.txt 2022-01-26T03:00 > updated_files.txt"> 
wget https://opendata.dwd.de/weather/nwp/content.log.bz2
bunzip2 content.log.bz2
echo "./icon-d2/grib/*/t_2m/*" > wildcards.txt
./get_updated_files.py -w wildcards.txt 2022-01-26T03:00 > updated_files.txt

The produced file updated_files.txt will hold all pathnames as given by content.log that are updated since 2022-01-26 03:00:00 UTC according to the file's modification date. Remember that those paths are relative to the directory the content.log.bz2 was in, in this case https://opendata.dwd.de/weather/nwp/.

Also mind that there are multiple servers behind https://opendata.dwd.de which might not be exactly in sync with each other regarding file modification timestamps. Look into the code of get_updated_files.py for a suggestion on how to deal with that.

While this program relies on the file modification timestamp dumped into content.log.bz2, it might be more feasible to process the data reference time that is contained in the filenames instead.

$ ./get_updated_files.py --help
usage: get_updated_files.py [-h] [--content-log CONTENT_LOG_FILE] [--wildcards WILDCARDS_FILE] [--min-delta MIN_DELTA] [--version] UPDATED_SINCE

Filters paths of a DWD Open Data content.log file for entries that have been updated.

positional arguments:
  UPDATED_SINCE         Last time files were checked for updates

optional arguments:
  -h, --help            show this help message and exit
  --content-log CONTENT_LOG_FILE, -f CONTENT_LOG_FILE
                        The decompressed content.log file (default: content.log)
  --wildcards WILDCARDS_FILE, -w WILDCARDS_FILE
                        Filter results by a set of wildcards
  --min-delta MIN_DELTA, -d MIN_DELTA
                        Minimum number of seconds a file needs to be younger than UPDATED_SINCE (default: 60)
  --version             show program's version number and exit
You might also like...
Script de monitoramento das teclas do teclado, salvando todos os dados digitados em um arquivo de log juntamente com os dados de rede.

listenerPython Script de monitoramento das teclas do teclado, salvando todos os dados digitados em um arquivo de log juntamente com os dados de rede.

Vinícius Azevedo 4 Nov 27, 2022
Python Common things by Problem Fighter Library, (Exception, Debug Log, etc.)

In the name of God, the Most Gracious, the Most Merciful. PF-PY-Common Documentation Install and update using pip: pip install -U xxxx Please find the

Problem Fighter 3 Jan 15, 2022
A log likelihood fit for extracting neutrino oscillation parameters

A-log-likelihood-fit-for-extracting-neutrino-oscillation-parameters Minimised the negative log-likelihood fit to extract neutrino oscillation paramete

Vid Homsak 1 Jan 23, 2022
A Python Based Utility for Processing GST-Return JSON Files to Multiple Formats
A Python Based Utility for Processing GST-Return JSON Files to Multiple Formats

GSTR 1/2A Utility by Shan.tk Open Source GSTR 1/GSTR 2A JSON to Excel utility based on Python. Useful for Auditors in Verifying GSTR 1 Return Invoices

Sudharshan TK 1 Oct 8, 2022
A python script that changes your desktop background based on current weather and time of the day.

Desktop background wallpaper, based on current weather and time A python script that changes your computer's desktop background based on current weath

Maj Gaberšček 1 Nov 16, 2021
A simple weather app.

keather A simple weather app. This is currently not finished. Dependencies: yay -S python-beautifulsoup4 tk

null 1 Jan 9, 2022
Use a real time weather API to apply wind to your mouse cursor.

wind-cursor Use a real time weather API to apply wind to your mouse cursor. Requirements PyAutoGUI pyowm Usage This program uses the OpenWeatherMap AP

Andreas Schmid 1 Feb 7, 2022
Python package for reference counting native pointers
Python package for reference counting native pointers

refcount master: testing: This package is primarily for managing resources in native libraries, written for instance in C++, from Python. While it boi

CSIRO Hydroinformatics 2 Nov 3, 2022
Project Faros is a reference implimentation of Red Hat OpenShift 4 on small footprint, bare-metal clusters.
Project Faros is a reference implimentation of Red Hat OpenShift 4 on small footprint, bare-metal clusters.

Project Faros Project Faros is a reference implimentation of Red Hat OpenShift 4 on small footprint, bare-metal clusters. The project includes referen

project: Faros 9 Jul 18, 2022
Comments
  • changing urljoin to os.path.join

    changing urljoin to os.path.join

    The behavior of urljoin is on my opionion not constructive, because of its description "Construct a full (“absolute”) URL by combining a “base URL” (base) with another URL (url). Informally, this uses components of the base URL, in particular the addressing scheme, the network location and (part of) the path, to provide missing components in the relative URL." (https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urljoin)

    opened by nklever 0
Releases(v2.0.0)

  • v2.0.0(Jan 27, 2022)

    This new major release introduces support for timezones. Also, the deprecated feature --wildcards was removed and instead the use of external tools like bzgrep on the compressed file content.log.bz2 are recommended. The content.log file was promoted to the

    Changes

    • CONTENT_LOG_FILE is now a positional argument which defaults to STDIN when omitted
    • --updated-since is now a required option

    New features

    • supply multiple CONTENT_LOG_FILEs as positional arguments to parse their combined content
    • --updated-since now supports timezones when specified as +HH:MM while assuming the system's time zone if not specified; date-times in content.log are always treated as UTC
    Source code(tar.gz)
    Source code(zip)

  • v1.1.0(Jan 27, 2022)

  • v1.0.0(Jan 27, 2022)

    This is the first fully functional version. It implements minimal features with focus on readability to also act as a form of documentation around the quirks around DWD's Open Data emerging from having two seperate servers sitting behind opendata.dwd.de.

    Features

    • List the paths contained in a content.log file for entries that have likely been updated since a given date-time
    • Filter those results by one or more --wildcards
    Source code(tar.gz)
    Source code(zip)
Owner
Deutscher Wetterdienst (DWD)
Deutscher Wetterdienst (DWD)
GitHub Repository
Object-oriented programming (OOP) is a method of structuring a program by bundling related properties and behaviors into individual objects. In this tutorial, you’ll learn the basics of object-oriented programming in Python.

06_Python_Object_Class Introduction 👋 Objected oriented programming as a discipline has gained a universal following among developers. Python, an in-

Milaan Parmar / Милан пармар / _米兰 帕尔马 239 Dec 20, 2022
Plux - A dynamic code loading framework for building plugable Python distributions

Plux plux is the dynamic code loading framework used in LocalStack. Overview The

LocalStack 65 Dec 20, 2022
プレヤフHackUチーム「キャット・タン」が作成したアプリ「illustection」

cat_tongue_illustection プレヤフHackUチーム「キャット・タン」が作成した， プライバシー保護アプリ「illustection」です！ デモ動画 https://youtu.be/z3I7LuB_i58 機能 アップロードされた画像をいい感じのイラストやの素材に置き換える（

4 Jul 03, 2021
Submission from Team OMR for the TRI-NIT Hackathon

Submission from Team OMR for the TRI-NIT Hackathon

0 Feb 01, 2022
Скрипт позволяет выгрузить участников чатов/каналов(по чату для комментариев) и сообщения в различные форматы файлов.

TG-Parser Парсер участников и сообщений из ТГ-Чатов и чатов для комментариев в ТГ-Каналах Возможности Выгрузка участников групп/каналов(по чату для ко

50 Jan 06, 2023
Easy, clean, reliable Python 2/3 compatibility

Overview: Easy, clean, reliable Python 2/3 compatibility python-future is the missing compatibility layer between Python 2 and Python 3. It allows you

Python Charmers 1.2k Jan 08, 2023
YourX: URL Clusterer With Python

YourX | URL Clusterer Screenshots Instructions for running Install requirements

ARPSyndicate 1 Mar 11, 2022
With the initiation of the COVID vaccination drive across India for all individuals above the age of 18, I wrote a python script which alerts the user regarding open slots in the vicinity!

cowin_notifier With the initiation of the COVID vaccination drive across India for all individuals above the age of 18, I wrote a python script which

13 Aug 01, 2021
Use `forge` and `cast` commands in Python scripts

foundrycli.py ( 🔥 , 🐍 ) foundrycli.py is a Python library I've made for personal use; now open source. It lets you access forge and cast CLIs from P

Zero Ekkusu 17 Jul 17, 2022
Python script to autodetect a base set of swiftlint rules.

swiftlint-autodetect Python script to autodetect a base set of swiftlint rules. Installation brew install pipx

Jonathan Wight 24 Sep 20, 2022
API Rate Limit Decorator

ratelimit APIs are a very common way to interact with web services. As the need to consume data grows, so does the number of API calls necessary to re

Tomas Basham 574 Dec 26, 2022
Click2call for asterisk with python

Click2call para Asterisk com Python Este projeto disponibiliza uma API construíd

Benedito Marques 1 Jan 17, 2022
A Non profit app built on top of Frappe framework & ERPNext

Non Profit A Non profit app built on top of Frappe framework & ERPNext. People who change the world need the tools to do it! The Non Profit Modules of

Frappe 16 Nov 17, 2022
Reload all Blender add-on modules

Reload-Addon This add-on creates a list of the modules that the add-on selected in the drop-down menu contains and reloads them with the keyboard shor

2 Dec 02, 2021
Python binding to rust zw-fast-quantile

zw_fast_quantile_py zw-fast-quantile python binding Installation pip install zw_fast_quantile_py Usage import zw_fast_quantile_py

Paul Meng 1 Dec 30, 2021
Demo Python project using Conda and Poetry

Conda Poetry This is a demonstration of how Conda and Poetry can be used in a Python project for dev dependency management and production deployment.

Ryan Allen 2 Apr 26, 2022
A Github Action for sending messages to a Matrix Room.

matrix-commit A Github Action for sending messages to a Matrix Room. Screenshot: Example Usage: # .github/workflows/matrix-commit.yml on: push:

3 Sep 11, 2022
Logo DYS (Doküman Yönetim Sitemi) API Python Implementation

dys-connector Logo DYS (Dokuman Yonetim Sistemi) API Python Implementation Python Package: https://pypi.org/project/dys-connector Quick Start from dys

Logo Group 8 Mar 19, 2022
ColabFold / AlphaFold2_advanced on your local PC (or macOS)

LocalColabFold ColabFold / AlphaFold2_advanced on your local PC (or macOS) Installation For Linux Make sure curl and wget commands are already install

Yoshitaka Moriwaki 207 Dec 22, 2022
A lightweight solution for local Particle development.

neopo A lightweight solution for local Particle development. Features Builds Particle projects locally without any overhead. Compatible with Particle

Nathan Robinson 19 Jan 01, 2023