Create time-series datacubes for supervised machine learning with ICEYE SAR images.

Trying to read an SLC .h5 downloaded from ICEYE archive (id 10499) and get 'RPC does not exist':

cube_config = CubeConfig()
slc_datacube = SLCDatacube.build(cube_config, '/Users/sstrong/bin/test_data_icecube/slcs')

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/var/folders/7r/fyfh8zx51ls6yt8t_jppnz3c0000gq/T/ipykernel_11546/2087236712.py in <module>
      1 cube_config = CubeConfig()
----> 2 slc_datacube = SLCDatacube.build(cube_config, '/Users/sstrong/bin/test_data_icecube/slcs')

~/Documents/github/icecube/icecube/bin/sar_cube/slc_datacube.py in build(cls, cube_config, raster_dir)
     52     def build(cls, cube_config: CubeConfig, raster_dir: str) -> SARDatacube:
     53         slc_datacube = SLCDatacube(cube_config, RASTER_DTYPE)
---> 54         ds = slc_datacube.create(cls.PRODUCT_TYPE, raster_dir)
     55         slc_datacube.xrdataset = ds
     56         return slc_datacube

~/Documents/github/icecube/icecube/utils/common_utils.py in time_it(*args, **kwargs)
    111     def time_it(*args, **kwargs):
    112         time_started = time.time()
--> 113         return_value = func(*args, **kwargs)
    114         time_elapsed = time.time()
    115         logger.info(

~/Documents/github/icecube/icecube/bin/sar_cube/sar_datacube.py in create(self, product_type, raster_dir)
     43         """
     44         metadata_object = SARDatacubeMetadata(self.cube_config)
---> 45         metadata_object = metadata_object.compute_metdatadf_from_folder(
     46             raster_dir, product_type
     47         )

~/Documents/github/icecube/icecube/bin/sar_cube/sar_datacube_metadata.py in compute_metdatadf_from_folder(self, raster_dir, product_type)
    116         )
    117 
--> 118         self.metadata_df = self._crawl_metadata(raster_dir, product_type)
    119         logger.debug(f"length metadata from the directory {len(self.metadata_df)}")
    120 

~/Documents/github/icecube/icecube/bin/sar_cube/sar_datacube_metadata.py in _crawl_metadata(self, raster_dir, product_type)
     68 
     69     def _crawl_metadata(self, raster_dir, product_type):
---> 70         return metadata_crawler(
     71             raster_dir,
     72             product_type,

~/Documents/github/icecube/icecube/utils/metadata_crawler.py in metadata_crawler(raster_dir, product_type, variables, recursive)
     36     _, raster_paths = DirUtils.get_dir_files(raster_dir, fext=fext)
     37 
---> 38     return metadata_crawler_list(raster_paths, variables)
     39 
     40 

~/Documents/github/icecube/icecube/utils/metadata_crawler.py in metadata_crawler_list(raster_paths, variables)
     43 
     44     for indx, raster_path in enumerate(raster_paths):
---> 45         metadata = IO.load_ICEYE_metadata(raster_path)
     46         parsed_metadata = _parse_data_row(metadata, variables)
     47         parsed_metadata["product_fpath"] = raster_path

~/Documents/github/icecube/icecube/utils/analytics_IO.py in load_ICEYE_metadata(path)
    432         are converted from bytedata and read into the dict for compatability reasons.
    433         """
--> 434         return read_SLC_metadata(h5py.File(path, "r"))
    435 
    436     elif path.endswith(".tif") or path.endswith(".tiff"):

~/Documents/github/icecube/icecube/utils/analytics_IO.py in read_SLC_metadata(h5_io)
    329 
    330     # RPCs are nested under "RPC/" in the h5 thus need to be parsed in a specific manner
--> 331     RPC_source = h5_io["RPC"]
    332     meta_dict["RPC"] = parse_slc_rpc_to_meta_dict(
    333         RPC_source=RPC_source, meta_dict=meta_dict

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

/opt/homebrew/anaconda3/envs/icecube_env/lib/python3.8/site-packages/h5py/_hl/group.py in __getitem__(self, name)
    303                 raise ValueError("Invalid HDF5 object reference")
    304         elif isinstance(name, (bytes, str)):
--> 305             oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
    306         else:
    307             raise TypeError("Accessing a group is done with bytes or str, "

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/h5o.pyx in h5py.h5o.open()

KeyError: "Unable to open object (object 'RPC' doesn't exist)"

Can't install all requirements for icecube on an M1 chip. This may present a future problem, just documenting for awareness. scikit-image cannot seem to be compiled/installed/etc on the M1. I have not tested the conda installation, as perhaps that does work. But i use brew/pip (and conda can create conflicts with those)

Summary includes:

• Making xr.dataset structure coherent for labels and SAR (added time coords for labels)
• For labels datacube, product_fpath are used compared to previously
• small typo fixed
• tests added for merging sar cubes with labels cube
• instructions/cell added to install ml requirements for notebook#5
• release notes added to mkdocs
• steup.py updated with ml requirements and version

Changes involve:

• Introduced a new markdown file called "overview.md" that talks about the structure of examples under docs/
• Added a new notebook : CreatingDatacube that walks a user how to create datacubes with different methods
• Other notebooks updated and improved.

related to issue: https://github.com/iceye-ltd/icecube/issues/11 Some of old ICEYE images can have RPC information missing. If that happens RPC key will be missing and pipeline does not work. RPC is now set to None if it's missing with a user warning generated.

Following changes introduced:

• metadata constraints loosen up to allow merging general SAR data (rasterio/HDF5 compatible). But this means that cube configuration is not available for such rasters
• .tiff support added for GRDs
• code refactoring in SARDatacubeMetadata to avoid repetitive code

Changes include:

• Updating SLC metadata reader to avoid key values stored as HDF5 dataset
• Enabling cube generation from labels.json that have masks/labels for subset rasters (i.e., number of masks ingested into labels cube don't necessarily have to be same as number of rasters)
• CHUNK_SIZE have been reduced to provide more optimized performance for creating massive datacubes

After installing from github using !pip install git+https://github.com/iceye-ltd/icecube.git it imports well icecube, but it throws this error for module bin ModuleNotFoundError: No module named 'icecube.bin'

Any advice, thanks

FileNotFoundError: [Errno 2] No such file or directory: './resources/labels/dummy_mask_labels.json'

Could you upload it? is it available? Thanks in advance,