Refererences: GNIP 89: Architecture Design - Resource and Storage Manager Modules

Checklist

Reviewing is a process done by project maintainers, mostly on a volunteer basis. We try to keep the overhead as small as possible and appreciate if you help us to do so by completing the following items. Feel free to ask in a comment if you have troubles with any of them.

For all pull requests:

• [x] Confirm you have read the contribution guidelines
• [x] You have sent a Contribution Licence Agreement (CLA) as necessary (not required for small changes, e.g., fixing typos in the documentation)
• [x] Make sure the first PR targets the master branch, eventual backports will be managed later. This can be ignored if the PR is fixing an issue that only happens in a specific branch, but not in newer ones.

The following are required only for core and extension modules (they are welcomed, but not required, for contrib modules):

• [x] There is a ticket in https://github.com/GeoNode/geonode/issues describing the issue/improvement/feature (a notable exemption is, changes not visible to end-users)
• [ ] The issue connected to the PR must have Labels and Milestone assigned
• [ ] PR for bug fixes and small new features are presented as a single commit
• [x] Commit message must be in the form "[Fixes #<issue_number>] Title of the Issue"
• [x] New unit tests have been added covering the changes, unless there is an explanation on why the tests are not necessary/implemented
• [x] This PR passes all existing unit tests (test results will be reported by travis-ci after opening this PR)
• [x] This PR passes the QA checks: flake8 geonode
• [ ] Commits changing the settings, UI, existing user workflows, or adding new functionality, need to include documentation updates
• [ ] Commits adding new texts do use gettext and have updated .po / .mo files (without location infos)

Submitting the PR does not require you to check all items, but by the time it gets merged, they should be either satisfied or inapplicable.

I'm working on migrating data for John Jediny's nepanode.anl.gov site, which is Geonode 2.0, to a new site that will use the latest version Geonode 2.4. From my perspective, there are three parts to this process:

1. Migrate Geonode 2.0's application data, typically found in Postgresql database "geonode", schema "geonode" to Geonode 2.4's application database "geonode", schema "public".
2. Migrate GeoServer's upload data store, typically found in Postgresql database "geonode_upload", schema "public" to Geonode 2.4's application database "datastore", schema, "public".
3. Migrate GeoServer's data directory.

At this point in time, I'm working on step 1. To that end, I've written a Python program named "create_migrate", that opens two connections, one for the source (Geonode 2.0) database and a second for the destination (Geonode 2.4) database. Next it compares the tables in each, providing a list of tables in the source, but not in the destination, a second list of tables that exist in the destination, but not in the source, and a third list of tables that exist in both. The it writes a set of "migrate_<table_name>.py" Python programs that can perform the migration. Why write a set of programs? Because each program has a section where select values are assigned to insert values, and it is there that a programmer can customize each program, if needed, to perform translations, ID substitutions, assign default values, etc..

For Geonode 2.0 to 2.4 here is the lists it produced:

Tables in src not in dst:

auth_user auth_user_groups auth_user_user_permissions base_thumbnail django_comment_flags django_comments notification_noticequeuebatch notification_noticesetting notification_noticetype people_role people_role_permissions security_genericobjectrolemapping security_objectrole security_objectrole_permissions security_userobjectrolemapping south_migrationhistory tagging_tag tagging_taggeditem taggit_templatetags_amodel

I've removed from this list captcha and zinnia tables that are unique modifications to NEPAnode.

Tables in dst not in src:

account_signupcodeextended base_license celery_taskmeta celery_tasksetmeta djcelery_crontabschedule djcelery_intervalschedule djcelery_periodictask djcelery_periodictasks djcelery_taskstate djcelery_workerstate groups_groupinvitation groups_groupmember groups_groupprofile guardian_groupobjectpermission guardian_userobjectpermission layers_layerfile layers_uploadsession maps_mapsnapshot people_profile_groups people_profile_user_permissions services_service services_servicelayer services_serviceprofilerole services_webserviceharvestlayersjob services_webserviceregistrationjob tastypie_apiaccess tastypie_apikey

Tables in both:

account_account account_accountdeletion account_emailaddress account_emailconfirmation account_signupcode account_signupcoderesult actstream_action actstream_follow agon_ratings_overallrating agon_ratings_rating announcements_announcement announcements_dismissal auth_group auth_group_permissions auth_permission avatar_avatar base_contactrole base_link base_region base_resourcebase base_resourcebase_regions base_restrictioncodetype base_spatialrepresentationtype base_topiccategory dialogos_comment django_admin_log django_content_type django_session django_site documents_document layers_attribute layers_layer layers_layer_styles layers_style maps_map maps_maplayer people_profile taggit_tag taggit_taggeditem upload_upload upload_uploadfile user_messages_message user_messages_thread user_messages_userthread

First issue found: I have found that Geonode 2.0's auth_user and people_profile tables have been merged in Geonode 2.4, and accordingly, I modified the generated program migrate_people_profile.py.

Second issue found: I have found that there are a set of configuration and/or code tables, where the "meaning" of the ID values has not been preserved from Geonode 2.0 to Geonode 2.4. May I suggest that in the future greater attention be paid to the reuse of such IDs? The only solution I have at this time is to build a mapping table from the old values to the new values, and swapping the values on-the-fly in the migration programs. That is a lot of extra work that may negative impact the adoption of Geonode going forward, and I don't want to see that happen.

Third issue: concerning the list of Geonode 2.0 tables that are not in Geonode 2.4, do you have any documentation or explanation as to why? Were these table renamed, merged, or did they become obsolete?

Fourth issue: concerning the list of Geonode 2.4 tables that are not in Geonode 2.0, do you have any documentation or explanation as to why? Are these table renamed, merged, or really new?

John Jediny and I would like to share our migration work so others who are in a similar situartion may benefit. How do we go about doing that within the structure of your project?

When trying to print a map, the following error message is given

Communication Trouble: Status undefined [object, Object]

Verified the yaml file is correct. Issue is reproducible on alpha dev server.

After previouse success lunching geonode 2.01 on ubuntu 12.04 and making project for custom theming, for using latest version, jumped to 2.4, but some error appeared.. 500 Internal Server Error !

after tracking changes from apache to new version 2.4, error steel exist.

from error.log of apache :

[Tue Mar 03 02:45:09.631507 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] mod_wsgi (pid=1299): Exception occurred processing WSGI script '/home/dary/my_geonode/my_geonode/wsgi.py'. [Tue Mar 03 02:45:09.631563 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] Traceback (most recent call last): [Tue Mar 03 02:45:09.631596 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] File "/usr/lib/python2.7/dist-packages/django/core/handlers/wsgi.py", line 187, in call [Tue Mar 03 02:45:09.631638 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] self.load_middleware() [Tue Mar 03 02:45:09.631654 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 47, in load_middleware [Tue Mar 03 02:45:09.631674 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] mw_instance = mw_class() [Tue Mar 03 02:45:09.631687 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] File "/usr/lib/python2.7/dist-packages/django/middleware/locale.py", line 24, in init [Tue Mar 03 02:45:09.631707 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] for url_pattern in get_resolver(None).url_patterns: [Tue Mar 03 02:45:09.631720 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] File "/usr/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 365, in url_patterns [Tue Mar 03 02:45:09.631739 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module) [Tue Mar 03 02:45:09.631752 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] File "/usr/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 360, in urlconf_module [Tue Mar 03 02:45:09.631770 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] self._urlconf_module = import_module(self.urlconf_name) [Tue Mar 03 02:45:09.631783 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] File "/usr/lib/python2.7/dist-packages/django/utils/importlib.py", line 40, in import_module [Tue Mar 03 02:45:09.631802 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] import(name) [Tue Mar 03 02:45:09.631815 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] File "/home/dary/my_geonode/my_geonode/urls.py", line 4, in [Tue Mar 03 02:45:09.631833 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] from geonode.urls import * [Tue Mar 03 02:45:09.631846 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] File "/usr/local/lib/python2.7/dist-packages/geonode/urls.py", line 30, in [Tue Mar 03 02:45:09.631865 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] from geonode.api.urls import api [Tue Mar 03 02:45:09.631878 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] File "/usr/local/lib/python2.7/dist-packages/geonode/api/urls.py", line 3, in [Tue Mar 03 02:45:09.631910 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] from .api import TagResource, TopicCategoryResource, ProfileResource,
[Tue Mar 03 02:45:09.631935 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] File "/usr/local/lib/python2.7/dist-packages/geonode/api/api.py", line 21, in [Tue Mar 03 02:45:09.631958 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] from tastypie.resources import ModelResource [Tue Mar 03 02:45:09.631972 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] File "/usr/lib/python2.7/dist-packages/tastypie/resources.py", line 18, in [Tue Mar 03 02:45:09.631993 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] from tastypie.authentication import Authentication [Tue Mar 03 02:45:09.632022 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] File "/usr/lib/python2.7/dist-packages/tastypie/authentication.py", line 13, in [Tue Mar 03 02:45:09.632045 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] from tastypie.compat import User, username_field [Tue Mar 03 02:45:09.632072 2015] [:error] [pid 1299:tid 140148981966592] [remote 127.0.0.1:39424] ImportError: cannot import name username_field

please help to removing errors..

Closes #4346

This PR is now a major component of GNIP-75 and enables spatial Django datastores for GeoNode.

Previous Description

This PR aims to enable full functionality of the 'search by extent' feature when filtering layers.

The previous implementation stored bounding box coordinates as decimal fields in the database, performing complex custom methods to infer coordinates and search queries such as filtering by layer extent.

To address issues in #4346, this PR:

• Uses the PostGIS extension in the GeoNode database (as specified by DATABASE_URL= in geonode/settings.py)
• Replaces existing decimal-based bounding fields with a PostGIS polygon (includes data migration)
• Relies on PostGIS to perform any queries related to bounding boxes

As a side effect, bounding boxes not provided in EPSG4326 can now be reprojected on-the-fly.

Previous decimal bbox_ fields have not been removed yet, but such a migration can be added once people are happy with our implementation.

Checklist

Reviewing is a process done by project maintainers, mostly on a volunteer basis. We try to keep the overhead as small as possible and appreciate if you help us to do so by completing the following items. Feel free to ask in a comment if you have troubles with any of them.

For all pull requests:

• [x] Confirm you have read the contribution guidelines
• [x] You have sent a Contribution Licence Agreement (CLA) as necessary (not required for small changes, e.g., fixing typos in the documentation)
• [x] Make sure the first PR targets the master branch, eventual backports will be managed later. This can be ignored if the PR is fixing an issue that only happens in a specific branch, but not in newer ones.

The following are required only for core and extension modules (they are welcomed, but not required, for contrib modules):

• [x] There is a ticket in https://github.com/GeoNode/geonode/issues describing the issue/improvement/feature (a notable exemption is, changes not visible to end-users)
• [x] The issue connected to the PR must have Labels and Milestone assigned
• [x] PR for bug fixes and small new features are presented as a single commit
• [x] Commit message must be in the form "[Fixes #<issue_number>] Title of the Issue"
• [ ] New unit tests have been added covering the changes, unless there is an explanation on why the tests are not necessary/implemented
• [x] This PR passes all existing unit tests (test results will be reported by travis-ci after opening this PR)
• [x] This PR passes the QA checks: flake8 geonode
• [x] Commits changing the settings, UI, existing user workflows, or adding new functionality, need to include documentation updates

Submitting the PR does not require you to check all items, but by the time it gets merged, they should be either satisfied or inapplicable.

GNIP-84: Upload Page Enhancements #7154

Checklist

Reviewing is a process done by project maintainers, mostly on a volunteer basis. We try to keep the overhead as small as possible and appreciate if you help us to do so by completing the following items. Feel free to ask in a comment if you have troubles with any of them.

For all pull requests:

• [x] Confirm you have read the contribution guidelines
• [x] You have sent a Contribution Licence Agreement (CLA) as necessary (not required for small changes, e.g., fixing typos in the documentation)
• [x] Make sure the first PR targets the master branch, eventual backports will be managed later. This can be ignored if the PR is fixing an issue that only happens in a specific branch, but not in newer ones.

The following are required only for core and extension modules (they are welcomed, but not required, for contrib modules):

• [x] There is a ticket in https://github.com/GeoNode/geonode/issues describing the issue/improvement/feature (a notable exemption is, changes not visible to end-users)
• [x] The issue connected to the PR must have Labels and Milestone assigned
• [ ] PR for bug fixes and small new features are presented as a single commit
• [x] Commit message must be in the form "[Fixes #<issue_number>] Title of the Issue"
• [x] New unit tests have been added covering the changes, unless there is an explanation on why the tests are not necessary/implemented
• [x] This PR passes all existing unit tests (test results will be reported by travis-ci after opening this PR)
• [x] This PR passes the QA checks: flake8 geonode
• [ ] Commits changing the settings, UI, existing user workflows, or adding new functionality, need to include documentation updates
• [x] Commits adding new texts do use gettext and have updated .po / .mo files (without location infos)

Submitting the PR does not require you to check all items, but by the time it gets merged, they should be either satisfied or inapplicable.

Hi eb, I'm here, Ubuntu 20.04 Core installation Install and configure NGINX https://docs.geonode.org/en/master/install/advanced/core/#install-and-configure-nginx

When I restart the uwsgi service: sudo service uwsgi restart

I get: Job for uwsgi.service failed because the control process exited with error code. See "systemctl status uwsgi.service" and "journalctl -xe" for details.

The service status is: ● uwsgi.service - LSB: Start/stop uWSGI server instance(s) Loaded: loaded (/etc/init.d/uwsgi; generated) Active: failed (Result: exit-code) since Sun 2021-01-03 20:00:21 CET; 2min 51s ago Docs: man:systemd-sysv-generator(8) Process: 25449 ExecStart=/etc/init.d/uwsgi start (code=exited, status=1/FAILURE)

Thanks

GNIP 75 - Enable spatial data store for Django DB

Overview

This GNIP proposes the transition to spatial data stores such as PostGIS and Spatialite in place of the existing Django database. This will enable more streamlined and easier implementation of features involving geospatial queries such as 'search by extent'. The implication of this transition is the migration of existing data sets. The implementation requires adjustments to all installation sources and fixes to the Travis CI and paver test suites.

Proposed By

@mtnorthcott @GeoTob

Assigned to Release

This proposal is for GeoNode 3.x

State

• [ ] Under Discussion
• [ ] In Progress
• [x] Completed
• [ ] Rejected
• [ ] Deferred

Existing work

This GNIP started as a fix for the 'search by extent' feature, proposed in #6064 which serves as a basis. In this state, the feature works with the Docker, paver and SPCGeoNode installation methods. The focus is now on getting Travis CI tests to pass, documenting and advertising all changes thoroughly.

Pull Requests:

• GeoNode/geonode
• GeoNode/postgis-docker (Merged)
• GeoNode/geonode-selenium (Merged)
• GeoNode/documentation (Merged)

Motivation

• Easier storage and handling of geospatial data
• Removal of unnecessary bbox_* decimal fields
• More simplistic implementation of features like 'search by extent'
• On-the-fly reprojection of non-EPSG4326 bounding boxes

Originally, an issue (#4346) was raised over the 'search by extent' feature not working properly. The issue was mitigated in a commit, however, the end goal is to transition to spatial databases exclusively. Work towards this goal has begun in effort to enable full functionality to the 'search by extent' feature in #6064.

Proposal

• [x] Update paver installation method and test accordingly
• [x] Update Docker installation method and test accordingly
• [x] Update Django test suite
• [x] Update Travis CI
• [x] Update all documentation accordingly, especially the setup parts
• [ ] Advertise all new changes accordingly

Backwards Compatibility

• We can maintain access to bounding box fields but calculate these upon request via Python @property decorated access methods
• Tests require adjustment to be compatible with spatial data stores

Future evolution

Feedback

Some initial thoughts have been discussed in the original issue (#4346) and the 'search by extent' pull request (#6064).

Voting

Project Steering Committee:

• Giovanni Allegi: ✔
• Alessio Fabiani: ✔
• Francesco Bartoli: abstained
• Simone Dalmasso:
• Toni Schoenbuchner: ✔
• Florian Hoedt: ✔

Links

• Issue #4346 - search by extent broken
• Commit 6624790
• Pull request #6064 - Filter by extent not working

Expected Behavior

Connect to resources based on access permissions set in GeoNode

Actual Behavior

QGIS is unable to see any available layers.

Steps to Reproduce the Problem

1. Uploaded a "Test_Layer" and changed permissions:
2. Attempt to connect to WFS resource using QGIS @ http://<geonode-url>/geoserver/ows?service=wfs&version=2.0.0&request=GetCapabilities and basic authentication for the testuser account.
3. QGIS is showing no layers for the connected WFS resource.
4. Geofence rules created by GeoNode are visible in Geoserver:

Specifications

• GeoNode version: 2.10.1
• Installation method (manual, GeoNode Docker, SPCGeoNode Docker): manual
• Platform: Ubuntu Server 18.04 LTS
• Additional details:

Expected Behavior

I installed GeoNode with Docker with this doc. Geonode is installed on production mode and every part of application works. I want to import a shapefile to GeoNode as an vector layer.

Actual Behavior

It has an error after geoserver configuration step during adding new vector layers.

java.lang.NoSuchMethodError: org.locationtech.jts.geom.Polygon.getExteriorRing()Lorg/locationtech/jts/geom/LinearRing; org.locationtech.jts.geom.Polygon.getExteriorRing()Lorg/locationtech/jts/geom/LinearRing; In the database, shapefile data, is stored in the geonode_data database ,and in the Geoserver,layer is publish from PostGIS store, But the in Geoserver I can't see the layer, and I also face with above error.

Steps to Reproduce the Problem

1. GeoNode Installed completely.
2. Raster Layer Upload worked.
3. Vector Layer Upload Face with an error.

Specifications

• GeoNode version: 3.1.0.dev1598252669
• Installation method (manual, GeoNode Docker, SPCGeoNode Docker): GeoNode Docker
• Platform: Ubuntu Server 18.04 LTS
• Additional details: I didn't changed any configuration ,and followed steps that were in this documentation.

Expected Behavior

You can add a new group to a map like in mapstore2 demo: https://dev.mapstore.geo-solutions.it/mapstore/#/viewer/openlayers/6200

Actual Behavior

The add group button is missing in GeoNode

Steps to Reproduce the Problem

1. open master.demo.geonode.org and login
2. create a map
3. try to create a new group

Specifications

I tested this on my dev instance with SPC 2.10.1 and master.demo.geonode.org

• GeoNode version: master.demo.geonode.org ? / 2.10.1
• Installation method (manual, GeoNode Docker, SPCGeoNode Docker): ? / SPC
• Platform: some Linux
• Additional details:

Hi eb and Happy New Year. When in a Dashboard I try to create a chart widget from a vector layer I get this error:

Can not create the widget for the selected layer Please try to select another layer or widget type The server doesn't provide the needed services for the layer and the widget type selected Possible causes are:

• The selected layer is a raster layer
• WFS service is not available
• The WPS process gs:aggregate is not available

The vector layer is a very popular layer and can be downloaded from here: https://www.naturalearthdata.com/http//www.naturalearthdata.com/download/50m/cultural/ne_50m_admin_0_countries.zip

When I try to create the chart widget in a Map instead of a Dashboard, using the same ne_50m_admin_0_countries layer, I have no problem and I can create the chart correctly. It seems that the chart widget creation problem is only when I'm in a Dashboard. Thanks, Happy 2023

Bumps django-storages from 1.13.1 to 1.13.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps coverage from 7.0.0 to 7.0.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps boto3 from 1.26.32 to 1.26.37.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

GNIP 90 - New importer module

Overview

The upload workflow is complex and requires revision, updates, and improvements. Currently, the critical points of the upload workflow are:

• The steps are hardcoded and shared between all the resources
• It's dependent on the GeoServer importer
• It's not 100% async

With this proposal, we want to resolve the critical aspects of the actual upload flow and improve the reliability and speed of the import by:

• relying on celery and async at 100%
• removing the dependency with GeoServer
• encapsulating the upload flow for a single resource
• letting GeoNode be responsible for the data source
• having the available resources as a registry
• relying on ogr2ogr for the data import into the datastore DB

Proposed By

@mattiagiupponi @giohappy

Assigned to Release

This proposal is for GeoNode.

State

• [x] Under Discussion
• [x] In Progress
• [ ] Completed
• [ ] Rejected
• [ ] Deferred

**NOTICE**: GeoNode master demo is already configured with the new importer, which is also activated for ESRI Shapefiles and Geotiffs. Feel free to test it!

Motivation

The upload workflow needs some revisions even if over the years it had some evolution, but it relies on a set of predefined steps that are inflexible and difficult to extend/adapt for more advanced flows or different input formats (e.g. GPKG, etc.). GeoNode cannot expose the imported data since it is not managed by the app, and the integration with the GeoServer importer (which also runs asynchronously can create some inconsistency with the resource state.

Proposal

The new importer is based on a few strong concepts:

• orchestrator: the main object which is responsible for starting the import process and evaluating its progression
• dynamic models: dynamic creation of the Django model for the resource (vector only)
• handlers: objects entitled to handle the resource

Handlers

The handler is the core component of the new importer. It represents an object that handles all the aspects of the resource imported in GeoNode.

Each handler has the responsibility to:

• validate the input data
• define the upload step required for importing the resource
• manage the lifecycle of the data behind a resource (e.g. copy, delete, etc.)

A few handlers are already available that encapsulate the shared operations for a generic resource type (Vector or Raster file),. However, each handler can override the functionalities by defining the specific behavior for an input format.

Additional documentation that describes how to create a handler, will be available soon on the geonode-importer repository.

Examples for handlers code can be found here

Handlers registry

A new setting is added and named IMPORTER_HANDLERS. This new setting registers in the application scope the handlers available.

IMPORTER_HANDLERS = os.getenv('IMPORTER_HANDLERS', [
    'importer.handlers.geojson.handler.GeoJsonFileHandler',
    'importer.handlers.shapefile.handler.ShapeFileHandler'
])

At the moment, if the new importer module is installed and configured it will operate along with the old importer. The new importer will be triggered for input formats listed inside the IMPORTER_HANDLERS list only.

Vector handlers

The following vector extensions are supported:

• GeoPackage
• GeoJson
• Shapefiles
• KML & KMZ

Raster handlers

The following raster extension is supported:

• GeoTiff

NB: Since we are not relying anymore on GeoServer and GeoServer must have access to the raster data, both GeoNode and Geoserver must share a long-term storage folder where the raster data is stored.

Non-spatial data

Thanks to the versatility of the handlers the geonode-importer can handle non-spatial data like documents, pdf, CSV, json, etc...

Dynamic Models

One of the limits of GeoNode is not being aware of the base data that is imported and not being able to navigate it. One of the features introduced with the geonode-importer makes GeoNode able to perform these actions. During the import process, the handler (for the vector file type only) will read the file schema with ogr2ogr, define a dynamic model based on that specification and save its configuration in the datastore database.

Orchestrator

This object calls the next step defined in the handler. Is also the core point where the upload progression is updated and estimated.

Backwards Compatibility

The new workflow is an external third-party library and isn't contained in the core code atm.

Future evolution

Thanks to the dynamic model available, GeoNode could be able to expose CRUD API on the data itself. Progressively the current upload flow will be transferred to the new one.

Feedback

Update this section with relevant feedback, if any.

Voting

Project Steering Committee:

• Alessio Fabiani: 👍
• Francesco Bartoli:
• Giovanni Allegri: 👍
• Toni Schoenbuchner:
• Florian Hoedt:

Links

Remove unused links below.

• Email Discussion
• Pull Request
• Mail Discussion
• Linked Issue

References: #10472

Checklist

Reviewing is a process done by project maintainers, mostly on a volunteer basis. We try to keep the overhead as small as possible and appreciate if you help us to do so by completing the following items. Feel free to ask in a comment if you have troubles with any of them.

For all pull requests:

• [x] Confirm you have read the contribution guidelines
• [x] You have sent a Contribution Licence Agreement (CLA) as necessary (not required for small changes, e.g., fixing typos in the documentation)
• [x] Make sure the first PR targets the master branch, eventual backports will be managed later. This can be ignored if the PR is fixing an issue that only happens in a specific branch, but not in newer ones.

The following are required only for core and extension modules (they are welcomed, but not required, for contrib modules):

• [x] There is a ticket in https://github.com/GeoNode/geonode/issues describing the issue/improvement/feature (a notable exemption is, changes not visible to end-users)
• [x] The issue connected to the PR must have Labels and Milestone assigned
• [x] PR for bug fixes and small new features are presented as a single commit
• [x] Commit message must be in the form "[Fixes #<issue_number>] Title of the Issue"
• [x] New unit tests have been added covering the changes, unless there is an explanation on why the tests are not necessary/implemented
• [x] This PR passes all existing unit tests (test results will be reported by travis-ci after opening this PR)
• [x] This PR passes the QA checks: flake8 geonode
• [ ] Commits changing the settings, UI, existing user workflows, or adding new functionality, need to include documentation updates
• [ ] Commits adding new texts do use gettext and have updated .po / .mo files (without location infos)

Submitting the PR does not require you to check all items, but by the time it gets merged, they should be either satisfied or inapplicable.