🗂️ 🔍 Geospatial Data Management and Search API - Django Apps

Overview

logo

codecov ci PyPI

Geospatial Data API in Django

Resonant GeoData (RGD) is a series of Django applications well suited for cataloging and searching annotated geospatial imagery, shapefiles, and full motion video datasets.

A publicly deployed instance of this application is available at https://www.resonantgeodata.com - find the deployment code for this at ResonantGeoData/RD-OpenGeo

Search Interface Raster Viewer with ROI Extraction Tool
homepage roi-extraction

Highlights

  • Faceted searching of spatiotemporal data
  • Supports normal imagery and overhead raster imagery
  • Supports image annotations: polygons, bounding boxes, keypoints, and run-length-encoded masks
  • Built-in image tile server
  • 3D viewer for point clouds and mesh files
  • Extract Regions of Interest from imagery

Documentation

Each app's README file contains an overview of functionality.

Documentation is currently a work in progress.

For general questions about the project, its applications, or about software usage, please create an issue directly in this repository. You are also welcome to send us an email at [email protected] with the subject line including Resonant GeoData.

Connections

  • Resonant GeoData is built on top of Kitware's Girder 4 platform.
  • GeoJS: we leverage GeoJS for our interactive map view.
  • large_image: we leverage large_image to serve image tiles and extract thumbnails.
  • VTK.js: we use VTK.js for the client-side 3D viewer for 3D data.
  • KWIVER: we leverage KWIVER's Full Motion Video (FMV) processing capabilities to extract spatial information from FMV files.
  • KWCOCO: KWCOCO is an extension of the COCO image annotation format which we support for ingesting annotated imagery.

Contributing

Please see the adjacent DEVELOPMENT.md file for all development instructions.

Comments
  • Dev/xdoctest

    Dev/xdoctest

    Builds on #493

    Adds declaration of xdoctest as a test dependency and adds the flags to tox. I also added a second simple doctest to spatial_search_params.

    opened by Erotemic 12
  • Redesign geodata models to support non-spatial data

    Redesign geodata models to support non-spatial data

    Per team discussion, we need to support non-spatial imagery as well.

    I will brainstorm a new model hierarchy and post it here, then sync up with David about the implementation. I already have a few ideas about how we can make this work in a general fashion while keeping the SpatialEntry concept that informed our initial model design.

    opened by banesullivan 12
  • STAC Browser

    STAC Browser

    This imports version 2.0.0 of the STAC Browser. The build process was difficult to automate (a couple patches), so I made the decision to build and commit the result into the tree. STAC Browser 3.0.0 is the next release, so we can automate the build process there.

    The existing STAC-API was fleshed out a bit more to handle the browser.

    The STAC Browser is visible at /rgd_imagery/stac_browser/.

    Here is an example browsing the user's "default" collection:

    Capture

    opened by mcovalt 10
  • Allow lazy random access to files in Storage on S3

    Allow lazy random access to files in Storage on S3

    When a Django FieldFile (the object referenced by a FileField) is opened, both in development with MinIO and in production with AWS S3, the Storage layers download a whole file write it to a SpooledTemporaryFile (which is in-memory for small content and on-disk for larger content)

    • S3Boto3Storage: https://github.com/jschneier/django-storages/blob/770332b598712da27ecdba75c9e202ad6a1a8722/storages/backends/s3boto3.py#L132
    • MinioStorage: https://github.com/py-pa/django-minio-storage/blob/8ce77cd8eae0d86fa3200e74cde64144c0338f37/minio_storage/files.py#L130

    However, many operations need random access to only a small part of the file's content.

    If all that's necessary is a Python file-like object (which is the same interface that a Django FieldFile naturally provides), we could look into developing a new Storage wrapper which lazily loads the content of the file-like object, internally using HTTP Range requests to only download the required parts just-in-time to be read from. This would likely also include some caching of the parts once they are retrieved.

    If external tools (like GDAL) need access via a filesystem path (i.e. what's provided by field_file_to_local_path), then FUSE may be the most appropriate abstraction. However, FUSE could not be run on Heroku and might generally be more difficult to install for new developers.

    In either case (Python file-like object or FUSE), the underlying logic for how to fetch and cache portions of file content may be the same.

    opened by brianhelba 10
  • psycopg2.errors.UniqueViolation: duplicate key value error

    psycopg2.errors.UniqueViolation: duplicate key value error

    When running the demo data commands that I have, if the celery worker is set up to run in the background, an integretiy error for duplicate keys happens on the image_entry.save() call here:

    https://github.com/ResonantGeoData/ResonantGeoData/blob/998a6c3995b4421c3632979a249fb78d66e1108f/rgd/geodata/models/imagery/etl.py#L69

    The error:

    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
        return self.cursor.execute(sql, params)
    psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "geodata_imageentry_image_file_id_key"
    DETAIL:  Key (image_file_id)=(14) already exists.
    

    This is making me think that when we create a new ImageEntry in the tasks, there is some sort of race condition between jobs for the same ImageFile... which shouldn't happen? I'm not really sure what is going on here.

    Steps to reproduce

    1. Clear the database volume
    2. Apply migrations: docker-compose run --rm django ./manage.py migrate
    3. In one session, launch the celery worker: docker-compose up celery and wait until ready
    4. In another session, run the Landsat demo data command: docker-compose run --rm django ./manage.py landsat_data -c 3
      • Use the changes from #296
    5. Observe the error

    Error Message

    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
        return self.cursor.execute(sql, params)
    psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "geodata_imageentry_image_file_id_key"
    DETAIL:  Key (image_file_id)=(14) already exists.
    
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "./manage.py", line 28, in <module>
        main()
      File "./manage.py", line 24, in main
        execute_from_command_line(sys.argv)
      File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
        utility.execute()
      File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
        self.execute(*args, **cmd_options)
      File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
        output = self.handle(*args, **options)
      File "/opt/django-project/rgd/geodata/management/commands/landsat_data.py", line 49, in handle
        helper.load_raster_files(_get_landsat_urls(count))
      File "/opt/django-project/rgd/geodata/management/commands/_data_helper.py", line 80, in load_raster_files
        imentries = load_image_files(
      File "/opt/django-project/rgd/geodata/management/commands/_data_helper.py", line 56, in load_image_files
        result = load_image_files(imfile)
      File "/opt/django-project/rgd/geodata/management/commands/_data_helper.py", line 60, in load_image_files
        read_image_file(entry)
      File "/opt/django-project/rgd/geodata/models/imagery/etl.py", line 129, in read_image_file
        _read_image_to_entry(image_entry, file_path)
      File "/opt/django-project/rgd/geodata/models/imagery/etl.py", line 69, in _read_image_to_entry
        image_entry.save()
      File "/opt/django-project/rgd/geodata/models/common.py", line 51, in save
        super(ModifiableEntry, self).save(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 726, in save
        self.save_base(using=using, force_insert=force_insert,
      File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 763, in save_base
        updated = self._save_table(
      File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 868, in _save_table
        results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
      File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 906, in _do_insert
        return manager._insert(
      File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
        return getattr(self.get_queryset(), name)(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1268, in _insert
        return query.get_compiler(using=using).execute_sql(returning_fields)
      File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1410, in execute_sql
        cursor.execute(sql, params)
      File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
        return super().execute(sql, params)
      File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
        return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
      File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
        return executor(sql, params, many, context)
      File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
        return self.cursor.execute(sql, params)
      File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
        raise dj_exc_value.with_traceback(traceback) from exc_value
      File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
        return self.cursor.execute(sql, params)
    django.db.utils.IntegrityError: duplicate key value violates unique constraint "geodata_imageentry_image_file_id_key"
    DETAIL:  Key (image_file_id)=(14) already exists.
    
    bug 
    opened by banesullivan 9
  • Large image

    Large image

    Closes #260

    This is a simple tile server endpoint. It's not an app as was imagined. There's a 💯abstract™ app in 701293890f75b646d042da877289646ebac88a7f. It started feeling silly for just one endpoint, so it was reverted. It may be useful down the road.

    I didn't want to deal with the context manager to download/cleanup S3 files, so instead I'm using /vsis3/ and /vsicurl/. You can see the implementation in 8b4d2fc6f162447c128b1a4fdb17e8353853dd09. I think it's simpler and luckily results in a pretty quick tile server. It was tested with other ETL routines in ed9bd8ae57e3d9c9ff9f891847a051073280b2ce, which seemed to work fine, but that was reverted to stay on topic with this PR.

    New endpoints:

    • api/geodata/imagery/image_entry/<pk>/tiles: get an ImageEntry's metadata
    • api/geodata/imagery/image_entry/<pk>/tiles/<z>/<x>/<y>.jpg: get an ImageEntry's x, y tile at zoom-level z
    opened by mcovalt 9
  • Rgdc usability improvements

    Rgdc usability improvements

    This MR contains 3 small improvements to rgdc.Rgdc.

    1. Add 5 automatic retries on all GET requests. This is to guard against random 503 errors from rate-limiting that would be fixed by simply rerunning the command. The implementation is to bake this into the Session object; see https://www.peterbe.com/plog/best-practice-with-retries-with-requests for the approach.
    2. Add overwrite=False kwarg to download_raster_entry. This will skip files already on disk - handy for speeding up tests and making scripts idempotent.
    3. Perhaps a more controversial one: let download_raster_entry and download_raster_entry_thumbnail directly parse a dict returned by search to find the id, rather than the user having to make another request manually as in the README example or study that dict to find it. I believe this will improve usability for the current use cases, as the users of this client should need to know as little as possible about the internals of RGD's data representation (and web programming in general, for that matter) to get started. But I am open to ideas to implement this in a more principled and future-proof way.
    opened by matt-bernstein 8
  • fix thumbnails and outline geometry

    fix thumbnails and outline geometry

    Resolve #176

    For future reference, use the following example rasters to verify that the thumbnails are working.

    • LC08_L1TP_033032_20200422_20200508_01_T1_pixel_qa.tif
    • landcover_sample_2000.tif
    • vegdri_emodis_week34_082320.tif

    Summary of change:

    • new admin actions to reprocess image entries and raster entries (calls save to trigger save event and recreate the thumbnail or spatial reference)
    • added a helper method to reproject an open rasterio raster to a given ESPG
    • During image thumbnail generation routine: reproject raster to Web Mercator before extracting if the image file has a spatial reference
    • During raster outline/valid footprint calculation:
      • first, reproject the raster to the DB's spatial reference (something was off when transforming the extracted geometry)
      • use rasterio/GDAL to extract a mask and create the GeoJSON feature (as opposed to our own convex hull approach)
    • Raster detail view page now limits the view bounds on the map
    • Update centroids in the raster tests since we are extracting the geometry properly now
    opened by banesullivan 8
  • `yield_checksumfiles` function doesn't handle `ChecksumFiles` with file paths as their `name` correctly

    `yield_checksumfiles` function doesn't handle `ChecksumFiles` with file paths as their `name` correctly

    To reproduce, run this management script and then try to view the Raster that it creates. The tile requests will fail with 500 errors with errors about wrong file paths in the server log.

    I came across this bug when working on Danesfield. On that project we encode the file path of a given ChecksumFile inside the name column - for example, foo/bar/foobar.txt represents a file hierarchy of

    foo/
      bar/
        foobar.txt
    

    This line in RGD is incompatible with this approach, since it assumes the ChecksumFile name is a flat file name instead of a file path. For example, if we have a ChecksumFile with the name foo.tiff, the current code would work:

    path = '/tmp/rgd/file_cache/foo.tiff'
    with yield_checksumfiles([self], path.parent):
        yield path
    

    Note that path.parent would be /tmp/rgd/file_cache/foo.tiff, which is correct. But if the file name is a/b/c/foo.tiff:

    path = '/tmp/rgd/file_cache/a/b/c/foo.tiff'
    with yield_checksumfiles([self], path.parent):
        yield path
    

    the call to path.parent would evaluate to /tmp/rgd/file_cache/a/b/c, which is incorrect and leads to 500 errors since the server can't find the files in that location.

    bug 
    opened by mvandenburgh 7
  • Add more base maps

    Add more base maps

    Add the base maps from here: https://carto.com/help/building-maps/basemap-list/#carto-vector-basemaps

    and this one for google imagery: https://mt1.google.com/vt/lyrs=s&x={x}&y={y}&z={z}

    feature-request 
    opened by banesullivan 7
  • large_image tiling issue with specific dataset

    large_image tiling issue with specific dataset

    Recently, large_image started failing to serve tiles for this dataset: https://data.kitware.com/#item/5f9c316650a41e3d1920ef90

    We were originally using this raster for some testing of the large_image tile serving endpoint, so I know it worked at one point.

    Recently, I started seeing this numpy error when trying to serve tiles for this file:

    ...
                                      File "/usr/local/lib/python3.8/site-packages/nu
    django_1    |                     mpy/core/getlimits.py", line 523, in __init__
    django_1    |                         raise ValueError("Invalid integer data type
    django_1    |                     %r." % (self.kind,))
    django_1    |                     ValueError: Invalid integer data type 'f'.
    
    django_1    | [16:42:08] ERROR    Internal Server Error: /api/geodata/imagery/image log.py:224
    django_1    |                     _entry/159/tiles/12/830/1565.png
    django_1    |                     Traceback (most recent call last):
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/dj
    django_1    |                     ango/core/handlers/exception.py", line 47, in
    django_1    |                     inner
    django_1    |                         response = get_response(request)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/dj
    django_1    |                     ango/core/handlers/base.py", line 181, in
    django_1    |                     _get_response
    django_1    |                         response = wrapped_callback(request,
    django_1    |                     *callback_args, **callback_kwargs)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/dj
    django_1    |                     ango/views/decorators/csrf.py", line 54, in
    django_1    |                     wrapped_view
    django_1    |                         return view_func(*args, **kwargs)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/dj
    django_1    |                     ango/views/generic/base.py", line 70, in view
    django_1    |                         return self.dispatch(request, *args,
    django_1    |                     **kwargs)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/re
    django_1    |                     st_framework/views.py", line 509, in dispatch
    django_1    |                         response = self.handle_exception(exc)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/re
    django_1    |                     st_framework/views.py", line 469, in
    django_1    |                     handle_exception
    django_1    |                         self.raise_uncaught_exception(exc)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/re
    django_1    |                     st_framework/views.py", line 480, in
    django_1    |                     raise_uncaught_exception
    django_1    |                         raise exc
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/re
    django_1    |                     st_framework/views.py", line 506, in dispatch
    django_1    |                         response = handler(request, *args, **kwargs)
    django_1    |                       File
    django_1    |                     "/opt/django-project/rgd/geodata/api/tiles.py",
    django_1    |                     line 36, in get
    django_1    |                         tile_binary = tile_source.getTile(x, y, z)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/la
    django_1    |                     rge_image/cache_util/cache.py", line 102, in
    django_1    |                     wrapper
    django_1    |                         v = func(self, *args, **kwargs)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/la
    django_1    |                     rge_image_source_gdal/__init__.py", line 692, in
    django_1    |                     getTile
    django_1    |                         return self._outputTile(tile,
    django_1    |                     TILE_FORMAT_NUMPY, x, y, z,
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/la
    django_1    |                     rge_image/tilesource/base.py", line 1486, in
    django_1    |                     _outputTile
    django_1    |                         tile, mode = self._outputTileNumpyStyle(tile,
    django_1    |                     applyStyle, x, y, z, kwargs.get('frame'))
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/la
    django_1    |                     rge_image/tilesource/base.py", line 1444, in
    django_1    |                     _outputTileNumpyStyle
    django_1    |                         tile = self._applyStyle(tile, self.style, x,
    django_1    |                     y, z, frame)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/la
    django_1    |                     rge_image/tilesource/base.py", line 1406, in
    django_1    |                     _applyStyle
    django_1    |                         min = self._getMinMax('min', entry.get('min',
    django_1    |                     'auto'), image.dtype, bandidx, frame)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/la
    django_1    |                     rge_image/tilesource/base.py", line 1318, in
    django_1    |                     _getMinMax
    django_1    |                         self._scanForMinMax(dtype, frame)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/la
    django_1    |                     rge_image_source_gdal/__init__.py", line 264, in
    django_1    |                     _scanForMinMax
    django_1    |                         self._bandRanges[frame]['max'], numpy.iinfo(s
    django_1    |                     elf._bandRanges[frame]['max'].dtype).max)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/nu
    django_1    |                     mpy/core/getlimits.py", line 523, in __init__
    django_1    |                         raise ValueError("Invalid integer data type
    django_1    |                     %r." % (self.kind,))
    django_1    |                     ValueError: Invalid integer data type 'f'.
    django_1    |            ERROR    Internal Server Error: /api/geodata/imagery/image log.py:224
    django_1    |                     _entry/159/tiles/12/831/1566.png
    django_1    |                     Traceback (most recent call last):
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/dj
    django_1    |                     ango/core/handlers/exception.py", line 47, in
    django_1    |                     inner
    django_1    |                         response = get_response(request)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/dj
    django_1    |                     ango/core/handlers/base.py", line 181, in
    django_1    |                     _get_response
    django_1    |                         response = wrapped_callback(request,
    django_1    |                     *callback_args, **callback_kwargs)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/dj
    django_1    |                     ango/views/decorators/csrf.py", line 54, in
    django_1    |                     wrapped_view
    django_1    |                         return view_func(*args, **kwargs)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/dj
    django_1    |                     ango/views/generic/base.py", line 70, in view
    django_1    |                         return self.dispatch(request, *args,
    django_1    |                     **kwargs)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/re
    django_1    |                     st_framework/views.py", line 509, in dispatch
    django_1    |                         response = self.handle_exception(exc)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/re
    django_1    |                     st_framework/views.py", line 469, in
    django_1    |                     handle_exception
    django_1    |                         self.raise_uncaught_exception(exc)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/re
    django_1    |                     st_framework/views.py", line 480, in
    django_1    |                     raise_uncaught_exception
    django_1    |                         raise exc
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/re
    django_1    |                     st_framework/views.py", line 506, in dispatch
    django_1    |                         response = handler(request, *args, **kwargs)
    django_1    |                       File
    django_1    |                     "/opt/django-project/rgd/geodata/api/tiles.py",
    django_1    |                     line 36, in get
    django_1    |                         tile_binary = tile_source.getTile(x, y, z)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/la
    django_1    |                     rge_image/cache_util/cache.py", line 102, in
    django_1    |                     wrapper
    django_1    |                         v = func(self, *args, **kwargs)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/la
    django_1    |                     rge_image_source_gdal/__init__.py", line 692, in
    django_1    |                     getTile
    django_1    |                         return self._outputTile(tile,
    django_1    |                     TILE_FORMAT_NUMPY, x, y, z,
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/la
    django_1    |                     rge_image/tilesource/base.py", line 1486, in
    django_1    |                     _outputTile
    django_1    |                         tile, mode = self._outputTileNumpyStyle(tile,
    django_1    |                     applyStyle, x, y, z, kwargs.get('frame'))
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/la
    django_1    |                     rge_image/tilesource/base.py", line 1444, in
    django_1    |                     _outputTileNumpyStyle
    django_1    |                         tile = self._applyStyle(tile, self.style, x,
    django_1    |                     y, z, frame)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/la
    django_1    |                     rge_image/tilesource/base.py", line 1407, in
    django_1    |                     _applyStyle
    django_1    |                         max = self._getMinMax('max', entry.get('max',
    django_1    |                     'auto'), image.dtype, bandidx, frame)
    django_1    |                       File "/usr/local/lib/python3.8/site-packages/la
    django_1    |                     rge_image/tilesource/base.py", line 1335, in
    django_1    |                     _getMinMax
    django_1    |                         value =
    django_1    |                     self._bandRanges[frame]['max'][bandidx]
    django_1    |                     IndexError: index 1 is out of bounds for axis 0
    django_1    |                     with size 1
    

    @manthey, any insight here?

    FYI, this raster is a part of our demo_data command.

    bug 
    opened by banesullivan 7
  • kwiver test failing

    kwiver test failing

    I cannot reproduce this error outside of the tox testing environment:

    https://github.com/ResonantGeoData/ResonantGeoData/runs/5993156571?check_suite_focus=true#step:7:60

    I added a skip for the failing test in

    https://github.com/ResonantGeoData/ResonantGeoData/commit/cd01ca20c99be4e69500d1bc1732b29a92c2214e

    The error logs are: stderr.txt

    the most likely culprit is

    terminate called after throwing an instance of 'pybind11::error_already_set'
      what():  ImportError: /usr/local/lib/python3.8/lib-dynload/_posixsubprocess.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyTuple_Type
    

    I cannot figure out why this is all of a sudden failing - no obvious packages have recently been upgraded

    bug 
    opened by banesullivan 1
  • Persistent S3 test file

    Persistent S3 test file

    We need a persistent file on S3 to test our file handling.

    At present, we have the following fixture:

    https://github.com/ResonantGeoData/ResonantGeoData/blob/f41be685481d3f62457aba1001faa68012760ffc/testing-utils/rgd_testing_utils/data_fixtures.py#L51-L53

    but that is currently failing with object nor found errors.

    This is not the first time this has happened and I've updated that fixture. I will temporarily patch this with another file from that bucket, but we really need to find a file, any file on S3 that we can rely on persisting.

    opened by banesullivan 0
  • migrate collection to FileSet

    migrate collection to FileSet

    • [ ] Add concrete definitions for all models in rgd and rgd_imagery
    • [ ] Make sure no two images point to the same FileSet (ref #591)
    • [ ] Change ancillary_files field on Raster to be a manyToManyField to FileSet
      • [ ] make sure this fileset is not used for an Image
    • [ ] add annotation on Image to point to FileSet
    • [ ] Change all relationships to ChecksumFile to point to FileSet per new definitions (Image is the exception and will be well documented why)

    Other things to address/start thinking about:

    • [ ] How does a user upload data? First, create a FileSet, then add ChecksumFiles? how do we implement that in the REST endpoints and Python client cleanly? Can we add some server-rendered pages to handle make this easier for users?
    • [ ] when getting images from an ImageSet, check each Image's FileSet collection before yielding
      • think of the use case where two users privately uploaded processed imagery to that ImageSet (through the ProcessedImage models)

    initial prototyping done in #566

    opened by banesullivan 0
  • Blog: Making a Schema Crosswalk between Django Models and STAC API

    Blog: Making a Schema Crosswalk between Django Models and STAC API

    Title: Making a Schema Crosswalk between Django Models and STAC API

    Description: Kitware offers ResonantGeoData: a Django web application for visualizing and searching geospatial datasets. We added support for interacting with these datasets via the STAC API. This lightning talk will discuss how we made a schema crosswalk from our current API and data models to comply with the STAC API.

    blog 
    opened by banesullivan 0
Releases(0.2.7)
Owner
Resonant GeoData
Geospatial data cataloging with Django
Resonant GeoData
Drf-stripe-subscription - An out-of-box Django REST framework solution for payment and subscription management using Stripe

Drf-stripe-subscription - An out-of-box Django REST framework solution for payment and subscription management using Stripe

Oscar Y Chen 68 Jan 07, 2023
This is a simple Todo web application built Django (back-end) and React JS (front-end)

Django REST Todo app This is a simple Todo web application built with Django (back-end) and React JS (front-end). The project enables you to systemati

Maxim Mukhin 5 May 06, 2022
A Django Webapp performing CRUD operations on Library Database.

CRUD operations - Django Library Database A Django Webapp performing CRUD operations on Library Database. Tools & Technologies used: Django MongoDB HT

1 Dec 05, 2021
Tools to easily create permissioned CRUD endpoints in graphene-django.

graphene-django-plus Tools to easily create permissioned CRUD endpoints in graphene-django. Install pip install graphene-django-plus To make use of ev

Zerosoft 74 Aug 09, 2022
Silk is a live profiling and inspection tool for the Django framework.

Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before presenting them in a user interface for further inspection:

Jazzband 3.7k Jan 02, 2023
Realtime data read and write without page refresh using Ajax in Django.

Realtime read-write with AJAX Hey,this is the basic implementation type of ajax realtime read write from the database. where you can insert or view re

Mehedi Hasan 3 Dec 13, 2022
Alt1-compatible widget host for RuneScape 3

RuneKit Alt1-compatible toolbox for RuneScape 3, for Linux and macOS. Compatibility macOS installation guide Running This project use Poetry as packag

Manatsawin Hanmongkolchai 75 Nov 28, 2022
A simple polling app made in Django and Bootstrap

DjangoPolls A Simple Polling app made with Django Instructions Make sure you have Python installed Step 1. Open a terminal Step 2. Paste the given cod

Aditya Priyadarshi 1 Nov 10, 2021
Exemplo de biblioteca com Django

Bookstore Exemplo de biblioteca feito com Django. Este projeto foi feito com: Python 3.9.7 Django 3.2.8 Django Rest Framework 3.12.4 Bootstrap 4.0 Vue

Regis Santos 1 Oct 28, 2021
Website desenvolvido em Django para gerenciamento e upload de arquivos (.pdf).

Website para Gerenciamento de Arquivos Features Esta é uma aplicação full stack web construída para desenvolver habilidades com o framework Django. O

Alinne Grazielle 8 Sep 22, 2022
A Django/Python web app that functions as a digital diary

My Django Diary Full-stack web application that functions as a digital diary using Django, Python, SQLite, HTML & CSS. Things I learned during this pr

1 Sep 30, 2022
Easily share data across your company via SQL queries. From Grove Collab.

SQL Explorer SQL Explorer aims to make the flow of data between people fast, simple, and confusion-free. It is a Django-based application that you can

Grove Collaborative 2.1k Dec 30, 2022
REST API with Django and SQLite3

REST API with Django and SQLite3

Luis Quiñones Requelme 1 Nov 07, 2021
Keep track of failed login attempts in Django-powered sites.

django-axes Axes is a Django plugin for keeping track of suspicious login attempts for your Django based website and implementing simple brute-force a

Jazzband 1.1k Dec 30, 2022
django social media app with real time features

django-social-media django social media app with these features: signup, login and old registered users are saved by cookies posts, comments, replies,

8 Apr 30, 2022
Send logs to RabbitMQ from Python/Django.

python-logging-rabbitmq Logging handler to ships logs to RabbitMQ. Compatible with Django. Installation Install using pip. pip install python_logging_

Alberto Menendez Romero 38 Nov 17, 2022
Django Phyton Web Apps template themes

Django Phyton Web Apps template themes Free download source code project for build a modern website using django phyton web apps. Documentation instal

Mesin Kasir 4 Dec 15, 2022
Django datatables and widgets, both AJAX and traditional. Display-only ModelForms.

Django datatables and widgets, both AJAX and traditional. Display-only ModelForms. ModelForms / inline formsets with AJAX submit and validation. Works with Django templates.

Dmitriy Sintsov 132 Dec 14, 2022
Simple Login Logout System using Django, JavaScript and ajax.

Djanog-UserAuthenticationSystem Technology Use #version Python 3.9.5 Django 3.2.7 JavaScript --- Ajax Validation --- Login and Logout Functionality, A

Bhaskar Mahor 3 Mar 26, 2022
Loguru is an exceeding easy way to do logging in Python

Django Easy Logging Easy Django logging with Loguru Loguru is an exceeding easy way to do logging in Python. django-easy-logging makes it exceedingly

Neutron Sync 8 Oct 17, 2022