Consider the case of downloading data for 6 months. This script does it month by month. If data does not exist for the last month the return code was -10 before this fix, indicating that no data at all is available.

Now we only get a -10 return code if indeed no data at all is available in the requested period

Fixes https://github.com/TUW-GEO/ecmwf_models/issues/26

If the CDS API does not have any data available we catch this with the error callback. Only the string Reason: Request returned no data is available to catch this. If the CDS package would ever change their error messages this would break.

As a return code the python errno.ENODATA is used if no data is present. I've not been able to find more widely used default error codes so I would argue that also just using -10 by convention would be ok instead.

Other changes

I've also removed the usage of the * import in this MR.

Downloading e.g. era5-land data of the last month gives an error like the following

era5_download /home/cpa/debugging/2021-10-20_era5_download/era5_land/ -s 2021-10-01 -e 2021-10-10 -p era5-land -var "volumetric_soil_water_layer_1" --h_steps 2
Downloading era5-land netcdf files between 2021-10-01T00:00:00 and 2021-10-10T00:00:00 into folder /home/cpa/debugging/2021-10-20_era5_download/era5_land/
2021-10-22 13:00:31,336 INFO Welcome to the CDS
2021-10-22 13:00:31,337 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-land
2021-10-22 13:00:31,453 INFO Request is queued
2021-10-22 13:12:55,443 INFO Request is failed
2021-10-22 13:12:55,444 ERROR Message: no data is available within your requested subset
2021-10-22 13:12:55,444 ERROR Reason:  Request returned no data
2021-10-22 13:12:55,445 ERROR   Traceback (most recent call last):
2021-10-22 13:12:55,445 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/handler.py", line 55, in handle_request
2021-10-22 13:12:55,445 ERROR       result = cached(context.method, proc, context, context.args, context.kwargs)
2021-10-22 13:12:55,445 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/caching.py", line 108, in cached
2021-10-22 13:12:55,445 ERROR       result = proc(context, *context.args, **context.kwargs)
2021-10-22 13:12:55,445 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 118, in __call__
2021-10-22 13:12:55,446 ERROR       return p(*args, **kwargs)
2021-10-22 13:12:55,446 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 59, in __call__
2021-10-22 13:12:55,446 ERROR       return self.proc(context, *args, **kwargs)
2021-10-22 13:12:55,446 ERROR     File "/home/cds/cdsservices/services/mars/mars.py", line 45, in internal
2021-10-22 13:12:55,446 ERROR       return mars(context, request, **kwargs)
2021-10-22 13:12:55,446 ERROR     File "/home/cds/cdsservices/services/mars/mars.py", line 17, in mars
2021-10-22 13:12:55,447 ERROR       execute_mars(context, requests)
2021-10-22 13:12:55,447 ERROR     File "/home/cds/cdsservices/services/mars/execute_mars.py", line 25, in execute_mars
2021-10-22 13:12:55,447 ERROR       raise NoDataException("Request returned no data", '')
2021-10-22 13:12:55,447 ERROR   cdsinf.exceptions.NoDataException: Request returned no data

In the current implementation that means that the downloading is retried 5 times after which the program fails when trying to open the downloaded NetCDF or GRIB file.

Ideally the CLI would show a error message and return with a negative exit code

I think we could catch that by using an error_callback that cdsapi provides. But maybe another solution will be necessary.

Hey! I was just fixing some CI issues. All python versions are passing now, but I saw that there are CI environments for pygrib version 1.9.9 and version 2.0.1. Or at least there should be, because until now only version 2.0.1 was tested (which was installed last from the requirements.txt via pip install -r requirements.txt in .travis.yml).

Now I have the 2 versions working, and tests fail for version 1.9.9 as some of the metadata stuff is handled differently it seems. e.g grbs.message(1)['levels'] returnsu'0-7' for v2.0.1 but u'7' for v1.9.9 (on the same file)

Should we support and test v1.9.9? Otherwise I would just set pygrib>=2.0.1 as the required version if thats ok for you?

I think supporting ERA 5 download and ts generation would be a good idea (so that it works when the full data set is available in a few months). https://www.ecmwf.int/en/newsletter/147/news/era5-reanalysis-production

Tried to separate ERA Interim and ERA5 more, so that we can focus on developing the ERA5 part better in the future. era5_download uses the cds api (instead of ecmwf api). There are more options for downloading now, and we can add more if we want, also the download functions are tested now.

Looks like download of era5 from february on has to be done via a new api.

"Access through ECMWF Web-API to dataset ERA5 and C3S Seasonal will be phased out, please migrate to the Climate Data Store (https://cds.climate.copernicus.eu/)."

https://confluence.ecmwf.int/display/CKB/C3S+ERA5%3A+Web+API+to+CDS+API

Can we release this as a new version? I think there were some significant changes:

• Add ERA5 support (download, reading, reshuffling)
• Add netcdf support for ERA5 and ERA-Interim download (regular grid)
• Netcdf download in regular grid, grib in gaussian grid -- download allows passing parameters for spatial resampling now (grib and nc)
• Rework GRIB message storing (dont create a new file for each message, but save messages for each day in a distinct file)
• Add more tests

We should consider adding:

• Grid reducion to land mask when creating TS (Land mask is already being downloaded automatically)
• Lookup table for parameter ids and variable names (download now via parameter ids, reshuffling via variable names necessary as param ids not saved when downloading netcdf files)

• Option for a different stepsize than full months for downloads. For hourly data, the requests are too large otherwise
• Removal of temporary files created by CDO after regridding, otherwise the temporary directory will be filled up when using jupyter notebooks.

This adds the option to regrid data directly after downloading it using CDO. The regridding is done using pre-computed weights in a separate thread in order to not block the download.

• Fix bug when creating 0.1 deg grid cells (floating point precision)
• Missing variables in grib files are now replaced by empty images.
• Read variable names from grib files from cfVarNameECMF instead of short_name field