Read and write rasters in parallel using Rasterio and Dask

Last update: Aug 30, 2022

Related tags

Overview

dask-rasterio

dask-rasterio provides some methods for reading and writing rasters in parallel using Rasterio and Dask arrays.

Usage

Read a multiband raster

>>> from dask_rasterio import read_raster

>>> array = read_raster('tests/data/RGB.byte.tif')
>>> array
dask.array<stack, shape=(3, 718, 791), dtype=uint8, chunksize=(1, 3, 791)>

>>> array.mean()
dask.array<mean_agg-aggregate, shape=(), dtype=float64, chunksize=()>
>>> array.mean().compute()
40.858976977533935

Read a single band from a raster

>>> from dask_rasterio import read_raster

>>> array = read_raster('tests/data/RGB.byte.tif', band=3)
>>> array
dask.array<raster, shape=(718, 791), dtype=uint8, chunksize=(3, 791)>

Write a singleband or multiband raster

>>> from dask_rasterio import read_raster, write_raster

>>> array = read_raster('tests/data/RGB.byte.tif')

>>> new_array = array & (array > 100)
>>> new_array
dask.array<and_, shape=(3, 718, 791), dtype=uint8, chunksize=(1, 3, 791)>

>>> prof = ... # reuse profile from tests/data/RGB.byte.tif...
>>> write_raster('processed_image.tif', new_array, **prof)

Chunk size

Both read_raster and write_raster accept a block_size argument that acts as a multiplier to the block size of rasters. The default value is 1, which means the dask array chunk size will be the same as the block size of the raster file. You will have to adjust this value depending on the specification of your machine (how much memory do you have, and the block size of the raster).

Install

Install with pip:

pip install dask-rasterio

Development

This project is managed by Poetry. If you do not have it installed, please refer to Poetry instructions.

Now, clone the repository and run poetry install. This will create a virtual environment and install all required packages there.

Run poetry run pytest to run all tests.

Run poetry build to build package on dist/.

Issue tracker

Please report any bugs and enhancement ideas using the GitHub issue tracker:

https://github.com/dymaxionlabs/dask-rasterio/issues

Feel free to also ask questions on our Gitter channel, or by email.

Help wanted

Any help in testing, development, documentation and other tasks is highly appreciated and useful to the project.

For more details, see the file CONTRIBUTING.md.

License

Source code is released under a BSD-2 license. Please refer to LICENSE.md for more information.

Download and process satellite imagery in Python using Sentinel Hub services.

Description The sentinelhub Python package allows users to make OGC (WMS and WCS) web requests to download and process satellite images within your Py

659 Dec 23, 2022

An API built to format given addresses using Python and Flask.

An API built to format given addresses using Python and Flask. About The API returns properly formatted data, i.e. removing duplicate fields, distingu

1 Feb 27, 2022

This is a simple python code to get IP address and its location using python

IP address & Location finder @DEV/ED : Pavan Ananth Sharma Dependencies: ip2geotools Note: use pip install ip2geotools to install this in your termin

2 Jul 5, 2022

A package built to support working with spatial data using open source python

EarthPy EarthPy makes it easier to plot and manipulate spatial data in Python. Why EarthPy? Python is a generic programming language designed to suppo

414 Dec 23, 2022

Using SQLAlchemy with spatial databases

GeoAlchemy GIS Support for SQLAlchemy. Introduction GeoAlchemy is an extension of SQLAlchemy. It provides support for Geospatial data types at the ORM

109 Dec 1, 2022

Solving the Traveling Salesman Problem using Self-Organizing Maps

Solving the Traveling Salesman Problem using Self-Organizing Maps This repository contains an implementation of a Self Organizing Map that can be used

3.1k Dec 31, 2022

Example of animated maps in matplotlib + geopandas using entire time series of congressional district maps from UCLA archive. rendered, interactive version below

5 May 18, 2022

Hapi is a Python library for building Conceptual Distributed Model using HBV96 lumped model & Muskingum routing method

Current build status All platforms: Current release info Name Downloads Version Platforms Hapi - Hydrological library for Python Hapi is an open-sourc

15 Dec 26, 2022

Daily social mapping project in November 2021. Maps made using PyGMT whenever possible.

20 Nov 24, 2022

Comments

the time to write a dask array in tif is too long?

I have a image which have 40000cols and 40000 rows ,while I use data = read_raster(filename, band=1), and the use witer_raster(out_filename, data), it need about one minutes, I want to know can it be quickly

opened by DeZhao-Zhang 2
Add a Gitter chat badge to README.md

dymaxionlabs/dask-rasterio now has a Chat Room on Gitter

@munshkr has just created a chat room. You can visit it here: https://gitter.im/dymaxionlabs/dask-rasterio.

This pull-request adds this badge to your README.md:

If my aim is a little off, please let me know.

Happy chatting.

PS: Click here if you would prefer not to receive automatic pull-requests from Gitter in future.

opened by gitter-badger 1
Does dask-rasterio support masked array?
I'm working with dask masked array, and was wondering what would be the translation of these lines?

import rasterio with rasterio.open(inputFile) as source: # this is a 3D numpy array, with dimensions [band, row, col] src_array = source.read(masked=True)

Thank you for your cool lib!
opened by Becheler 0

TypeError: self._hds cannot be converted to a Python object for pickling

Seems that rasterio's _hds object is no more serializable

distributed.protocol.pickle - INFO - Failed to serialize ("('filled-2f9fe0560be0502eda038fa941309294', 0, 0)", <dask_rasterio.write.RasterioDataset object at 0x7f8f9deac828>, (slice(0, 748, None), slice(0, 22415, None)), <unlocked _thread.lock object at 0x7f8f9cb2af58>, False). Exception: self._hds cannot be converted to a Python object for pickling
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/miniconda3/envs/jupyter/lib/python3.6/site-packages/distributed/protocol/pickle.py in dumps(x)
     37     try:
---> 38         result = pickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL)
     39         if len(result) < 1000:

~/miniconda3/envs/jupyter/lib/python3.6/site-packages/rasterio/_io.cpython-36m-x86_64-linux-gnu.so in rasterio._io.DatasetWriterBase.__reduce_cython__()

TypeError: self._hds cannot be converted to a Python object for pickling

opened by arkanoid87 3

Releases(0.2.1)

0.2.1(Jun 9, 2018)
Fixes a test assertion

Source code(tar.gz)
Source code(zip)
0.2.0(Jun 9, 2018)
Add band keyword argument to read_raster

Source code(tar.gz)
Source code(zip)
0.1.0(Jun 6, 2018)

Source code(tar.gz)
Source code(zip)

Owner

Dymaxion Labs

Creating new value from geospatial imagery with deep learning

GitHub Repository

Satellite imagery for dummies.

felicette Satellite imagery for dummies. What can you do with this tool? TL;DR: Generate JPEG earth imagery from coordinates/location name with public

1.8k Jan 03, 2023

Location field and widget for Django. It supports Google Maps, OpenStreetMap and Mapbox

django-location-field Let users pick locations using a map widget and store its latitude and longitude. Stable version: django-location-field==2.1.0 D

481 Dec 29, 2022

Tool to suck data from ArcGIS Server and spit it into PostgreSQL

chupaESRI About ChupaESRI is a Python module/command line tool to extract features from ArcGIS Server map services. Name? Think "chupacabra" or "Chupa

34 Dec 04, 2022

🌐 Local tile server for viewing geospatial raster files with ipyleaflet or folium

🌐 Local Tile Server for Geospatial Rasters Need to visualize a rather large (gigabytes) raster you have locally? This is for you. A Flask application

192 Jan 04, 2023

Asynchronous Client for the worlds fastest in-memory geo-database Tile38

This is an asynchonous Python client for Tile38 that allows for fast and easy interaction with the worlds fastest in-memory geodatabase Tile38.

53 Dec 29, 2022

Read images to numpy arrays

mahotas-imread: Read Image Files IO with images and numpy arrays. Mahotas-imread is a simple module with a small number of functions: imread Reads an

67 Jan 07, 2023

Obtain a GNSS position fix from an 11-millisecond raw GNSS signal snapshot

Obtain a GNSS position fix from an 11-millisecond raw GNSS signal snapshot without any prior knowledge about the position of the receiver and only coarse knowledge about the time.

2 Nov 17, 2022

Blender addons to make the bridge between Blender and geographic data

Blender GIS Blender minimal version : 2.8 Mac users warning : currently the addon does not work on Mac with Blender 2.80 to 2.82. Please do not report

5.9k Jan 02, 2023

Use Mapbox GL JS to visualize data in a Python Jupyter notebook

Location Data Visualization library for Jupyter Notebooks Library documentation at https://mapbox-mapboxgl-jupyter.readthedocs-hosted.com/en/latest/.

620 Dec 15, 2022

Wraps GEOS geometry functions in numpy ufuncs.

PyGEOS PyGEOS is a C/Python library with vectorized geometry functions. The geometry operations are done in the open-source geometry library GEOS. PyG

362 Dec 23, 2022

A ninja python package that unifies the Google Earth Engine ecosystem.

A Python package that unifies the Google Earth Engine ecosystem. EarthEngine.jl | rgee | rgee+ | eemont GitHub: https://github.com/r-earthengine/ee_ex

47 Dec 27, 2022

Search and download Copernicus Sentinel satellite images

sentinelsat Sentinelsat makes searching, downloading and retrieving the metadata of Sentinel satellite images from the Copernicus Open Access Hub easy

837 Dec 28, 2022

gpdvega is a bridge between GeoPandas and Altair that allows to seamlessly chart geospatial data

gpdvega gpdvega is a bridge between GeoPandas a geospatial extension of Pandas and the declarative statistical visualization library Altair, which all

49 Jul 25, 2022

leafmap - A Python package for geospatial analysis and interactive mapping in a Jupyter environment.

A Python package for geospatial analysis and interactive mapping with minimal coding in a Jupyter environment

1.4k Jan 02, 2023

Tool to display your current position and angle above your radar

🛠 Tool to display your current position and angle above your radar. As a response to the CS:GO Update on 1.2.2022, which makes cl_showpos a cheat-pro

6 Jan 04, 2023

Create Siege configuration files from Cloud Optimized GeoTIFF.

cogeo-siege Documentation: Source Code: https://github.com/developmentseed/cogeo-siege Description Create siege configuration files from Cloud Optimiz

3 Dec 01, 2022

Global topography (referenced to sea-level) in a 10 arcminute resolution grid

Earth - Topography grid at 10 arc-minute resolution Global 10 arc-minute resolution grids of topography (ETOPO1 ice-surface) referenced to mean sea-le

1 Jan 20, 2022

Raster-based Spatial Analysis for Python

🌍 xarray-spatial: Raster-Based Spatial Analysis in Python 📍 Fast, Accurate Python library for Raster Operations ⚡ Extensible with Numba ⏩ Scalable w

649 Jan 01, 2023

Ingest and query genomic intervals from multiple BED files

Ingest and query genomic intervals from multiple BED files.

4 May 29, 2021

QLUSTER is a relative orbit design tool for formation flying satellite missions and space rendezvous scenarios

QLUSTER is a relative orbit design tool for formation flying satellite missions and space rendezvous scenarios, that I wrote in Python 3 for my own research and visualisation. It is currently unfinis

9 Aug 23, 2022