I think the standard should define polygon orientation.

1. Spherical edges case

With spherical edges on sphere, there is an ambiguity in polygon definition, if the system allows polygons larger than hemisphere.

A sequence of vertices that define a polygon boundary can define either polygon to the left of that line, or to the right of the line. E.g. global coastal line can define either continents or oceans. Systems that support polygons larger than hemisphere usually use orientation rule to solve this ambiguity. E.g. MS SQL, Google BigQuery interpret the side to the left of the line as the content of the ring.

2. Planar edges case

Planar case does not have such ambiguity, but it is still good idea to have specific rule.

E.g. GeoJson RFC defines a rule consistent with the rule above:

   o  Polygon rings MUST follow the right-hand rule for orientation
      (counterclockwise external rings, clockwise internal rings).

@cholmes asked me to write a script to update the nz-building-outlines file to Parquet version 0.4.0. This reads from the GeoPackage version of the data, which you can download from here (1.3GB).

Part of this is derived from the GeoPandas code here. But this additionally ~~uses pyogrio~~ (reverted because it failed to build on CI) and pygeos to try and speed things up a little bit. It's probably not a bad thing to have a Python script here because the GeoPandas release schedule is likely slower than our release schedule (at least thus far).

You can install the new dependencies with poetry install and run it with

poetry run python write_nz_building_outline.py \
    --input nz-building-outlines.gpkg \
    --output nz-building-outlines.parquet \
    --compression SNAPPY

This takes about 5 minutes on my computer. With Snappy compression, the 1.3GB GeoPackage file became 410MB in Parquet. (375MB with ZSTD compression).

To see the CLI options you can run

poetry run python write_nz_building_outline.py --help

Closes https://github.com/opengeospatial/geoparquet/issues/42

Implements https://github.com/opengeospatial/geoparquet/issues/23.

This PR adds a draft for a basic validator using Python. It checks the metadata of a geoparquet file using JSON Schema, but it can be extended to include specific custom validations.

Example

Try to validate this wrong metadata

metadata = {
    "version": "0.x.0",
    "primary_column": "geom",
    "columns": {
        "geometry": {
            "encoding": "WKT",
            "edges": "",
            "bbox": [180, 90, 200, -90],
        },
    },
}

$ python3 validator/validate.py examples/example-wrong.parquet
Validating file...
- [ERROR] $.version: '0.x.0' does not match '^0\\.1\\.[0-9]+$'
          INFO: The version of the geoparquet metadata standard used when writing.
- [ERROR] $.columns.geometry.encoding: 'WKT' is not one of ['WKB']
          INFO: Name of the geometry encoding format. Currently only 'WKB' is supported.
- [ERROR] $.columns.geometry.edges: '' is not one of ['planar', 'spherical']
          INFO: Name of the coordinate system for the edges. Must be one of 'planar' or 'spherical'. The default value is 'planar'.
- [ERROR] $.columns.geometry.bbox[2]: 200 is greater than the maximum of 180
          INFO: The eastmost constant longitude line that bounds the rectangle (xmax).
- [ERROR] $.columns.geometry: 'crs' is a required property
- [ERROR] $.primary_column: must be in $.columns
This is an invalid GeoParquet file.

The output for a correct GeoParquet file

$ python3 validator/validate.py examples/example.parquet
Validating file...
This is a valid GeoParquet file.

In https://github.com/opengeospatial/geoparquet/pull/45 we explicitly allowed 3D geometries, but in that PR I forgot to update the bbox description to reflect this.

A common use case if for a geometry column to hold a single geometry type (Point, LineString, Polygon, ...) for all its records. It could be good to have an optional "type" field name under https://github.com/opengeospatial/geoparquet/blob/main/format-specs/geoparquet.md#column-metadata to capture that when it is known.

This would help for example conversion between GeoParquet and GIS formats (shapefiles, geopackage) that have typically this information in their metadata.

Values for type could be the ones accepted by GeoJSON: Point, LineString, Polygon, MultiPoint, MultiLineString, MultiPolygon, GeometryCollection (that would be extended to CircularString, CompoundCurve, CurvePolygon, MultiCurve, MultiSurface, PolyhedralSurface, TIN if we support ISO WKB, and with Z, M or ZM suffixes for other dimensionalities)

What to do when there is mixed geometry types ?

• do not set "type"
• set "type": "mixed"
• set "type": array of values, e.g. [ "Polygon", "MultiPolygon" ] (this one would be typical when converting from shapefiles where the polygonal type can hold both polygons and multipolygons )

There are a lot of options for how we approach coordinate reference systems.

• GeoJSON only allows 4326 data. They started with more options, but then narrowed it down.
• Simple Features for SQL defines an 'SRID table' where you are supposed to map number id's to crs well known text. PostGIS uses the same srid's as epsg, but oracle doesn't.
• WKT doesn't include projection info, but that's seen as a weakness, and one main reason why ewkt came. I believe ewkt just assumes epsg / postgis default srid table, so it's not fully interoperable with Oracle.
• STAC uses epsg, but you can set it to null and provide projjson or crs-wkt
• OGC API - Features core specifies just WGS-84 (long, lat), using a URI like http://www.opengis.net/def/crs/OGC/1.3/CRS84, see crs info

And there's obviously more.

My general take is that we should have a default, and expect most things to use that. But should specify it in a way that it could be an extension in the future. So we shouldn't just say 'everything is 4326' just in the spec, but should have a field that says this field is always 4326 for the core spec, but in the future that field could have other values.

So I think we do the first version with just 4326, and then when people ask for more we can have an extension.

One thing I'm not sure about is whether we should use 'epsg' as the field. EPSG covers most projections people want, but not all. In geopackage they just create a whole srid table to then refer to, so the SRID's used are defined. Usually the full epsg database is included, but then users can add other options.

One potential option would be to follow ogc api - features and use URI's. I'm not sure how widely accepted that approach is, like if the full epsg database is already referenced online. So instead of 'epsg' as the field we'd have 'crs', and it's a string URI.

It looks like there are a couple options for the case where edges is spherical:

• If orientation is spherical than counterclockwise orientation is required. (Or say that if it is not set then the default is counterclockwise instead of null - effectively the same, but maybe slightly better?)
• If spherical and orientation is left blank then have implementations use the 'smaller-of-two-possible' rule, as used by bigquery, sqlserver.

We could also just 'recommend' its use, and not mention the smaller-of-two-possible rule. Though that seems far from ideal for me, as it doesn't tell implementations what to do if they get spherical data without it set.

Currently in main it does say use the smaller-of-two-possible rule, but it is likely poorly described, as I wrote it and was just trying to capture I don't 100% understand.

In #80 @jorisvandenbossche removed the rule of two possible thing. Which I think is totally fine. But I'd like us to make an explicit decision about it.

Originally posted by @mentin in https://github.com/opengeospatial/geoparquet/issues/46#issuecomment-1105505534

Closes https://github.com/opengeospatial/geoparquet/issues/135 (rework of https://github.com/opengeospatial/geoparquet/pull/134 to focus on json schema for now)

This renames the geometry_type property to geometry_types and updates the JSON type to be an array of strings (instead of a string or array of strings). The spec language has been updated to reflect that an empty array indicates that the geometry types are not known. The schema now enforces that the items in the list are one of the expected types and allows the Z suffix ~for everything except GeometryCollection~.

I updated the example.parquet file to use geometry_types instead of geometry_type. (I followed the readme, but am struggling to run the validator, so admit I haven't yet done that.)

I bumped the schema version to 0.5.0-dev. Ideally this would happen as part of the process for releasing a tag (create release branch, update version, create tag, generate release, bump version back to X.Y.Z-dev, merge branch).

Fixes #132. Fixes #133.

Has there been discussion around including an id_column or something similar in the file metadata? I think it would assist in round-tripping features from other formats if it were known which column represented the feature identifier.

It looks like GDAL has a FID layer creation option. But I'm assuming that the information about which column was used when writing would be lost when reading from the parquet file (@rouault would need to confirm).

I grant that this doesn't feel "geo" specific, and there may be existing conventions in Parquet that would be appropriate.

From https://github.com/opengeospatial/geoparquet/pull/25#issuecomment-1059016020. The spec has a required crs field that stores a WKT2:2019 string representation of the Coordinate Reference System.

We currently recommend using EPSG:4326 for the widest interoperability of the written files. However, this is a dynamic CRS, and in addition uses an ensemble datum. See https://gdal.org/user/coordinate_epoch.html for some context. In summary, when using coordinates with a dynamic CRS, you also need to know the point in time of the observation to know the exact location.

Some discussion topics related to this:

• How do we deal with a dynamic CRS? We should probably give the option to include the "coordinate epoch" in the metadata (the point in time at which the coordinate is valid)
  • This coordinate epoch is not part of the actual CRS definition, so I think the most straightforward option is to specify an additional (optional) "epoch" field in the column metadata (next to "crs") that holds the epoch value as a decimal year (eg 2021.3).
  • This means we would only support a constant epoch per file. This is in line with the initial support for other formats in GDAL, and we can always later think about expanding this (eg additional field in the parquet file that has a epoch per geometry, or per vertex)

From experience in STAC and openEO, it seems some implementations/programming languages have a hard time to distinguish between "not present" and "null". Specifying different meanings for null and not present as for crs (unknown / CRS:84) might be a bad idea. Therefore, I'm putting out for discussion whether it's a good idea to do this and maybe instead use "unknown" as string or so instead of null?

Related to https://github.com/opengeospatial/geoparquet/issues/123

This is incomplete (more parameters should be added) and still draft (the script should be cleaned-up, ensure to add this to the CI to check the generated files, validate with json schema, etc). But wanted to open a PR already to see where and how we want to go with this.

This is a script that writes a bunch of .parquet files, and then also saves the metadata as a separate json file (extracted from the .parquet files using the existing script scripts/update_example_schemas.py).

For 1.0.0 we should have a validator that:

1. Tests not just the metadata but looks at the data itself to make sure it matches the metadata
2. Is user-friendly, not requiring python. Ideally a web-page and/or an easily installable binary.

This could be building on the current validator in this repo, or could be a new project we reference, but we want to be sure something exists, so putting this issue in to track it.

One idea that @jorisvandenbossche suggested is that we should have a set of data that shows the range of the specification, that implementors can use to make sure they're handling right, and which could be the basis of 'integration' testing.

This would include geoparquet files that have a variety of projections, geometry types (including multiple geometry types as in #119), plus things like multiple geometry columns, different edges values, etc. It could also be good to have a set of 'bad' files that can also be tested against.

When [Geo]Parquet files/sources are used within systems that treat them as tables (like Spark, Trino/Presto, Athena, etc.), basic Parquet metadata is tracked in a "catalog" (e.g., a Hive-compatible catalog like AWS Glue Catalog). The engine being used for querying uses metadata to limit the parts of files (and files themselves) that are scanned, but they only expose the columnar content that's present, not the metadata. In some cases, metadata can be queried from the catalogs (e.g., from Athena, but the catalogs need additional work to support the metadata specified by GeoParquet (and this largely hasn't been done yet).

In the meantime, I'm curious if it makes sense to take the same metadata that's contained in the footer and externalize it into an alternate file (which could be serialized as Parquet, Avro, JSON, etc.). This would allow the query engines to register the metadata as a separate "table" (query-able as a standard source vs. requiring catalog support) and surface/take advantage of "table"-level information like CRS at query-time. At the moment, the CRS of a geometry column is something that needs to be determined out of band.

This is somewhat similar to #79, in that it doesn't look at GeoParquet sources as "files" ("tables" are often backed by many files), and could be seen as another reason to (de-)duplicate data from file footers into something that covers the whole set.

/cc @jorisvandenbossche and @kylebarron, since we talked a bit about this at FOSS4G.