This PR contains work on refactoring to achieve separation of concerns between compression/decompression, storage, synchronization and array/chunk data management. Primarily motivated by #21.

Still TODO:

• [x] Improve test coverage
• [x] Rework persistance/storage documentation to generalise beyond directory store
• [x] Improve docstrings, add examples where appropriate
• [x] Do some benchmarking/profiling
• [x] Fall back to pure python install

I think there are some places where zarr would benefit immensely from some async capabilities when reading and writing data. I will try to illustrate this with the simplest example I can.

Let's consider a zarr array stored in a public S3 bucket, which we can read with fsspec's HTTPFileSystem interface (no special S3 API needed, just regular http calls).

import fsspec
url_base = 'https://mur-sst.s3.us-west-2.amazonaws.com/zarr/time'
mapper = fsspec.get_mapper(url_base)
za = zarr.open(mapper)
za.info

Note that this is a highly sub-optimal choice of chunks. The 1D array of shape (6443,) is stored in chunks of only (5,) items, resulting in over 1000 tiny chunks. Reading this data takes forever, over 5 minutes

%prun tdata = za[:]

         20312192 function calls (20310903 primitive calls) in 342.624 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1289  139.941    0.109  140.077    0.109 {built-in method _openssl.SSL_do_handshake}
     2578   99.914    0.039   99.914    0.039 {built-in method _openssl.SSL_read}
     1289   68.375    0.053   68.375    0.053 {method 'connect' of '_socket.socket' objects}
     1289    9.252    0.007    9.252    0.007 {built-in method _openssl.SSL_CTX_load_verify_locations}
     1289    7.857    0.006    7.868    0.006 {built-in method _socket.getaddrinfo}
     1289    1.619    0.001    1.828    0.001 connectionpool.py:455(close)
   930658    0.980    0.000    2.103    0.000 os.py:674(__getitem__)
...

I believe fsspec is introducing some major overhead by not reusing a connectionpool. But regardless, zarr is iterating synchronously over each chunk to load the data:

https://github.com/zarr-developers/zarr-python/blob/994f2449b84be544c9dfac3e23a15be3f5478b71/zarr/core.py#L1023-L1028

As a lower bound on how fast this approach could be, we bypass zarr and fsspec and just fetch all the chunks with requests:

import requests
s = requests.Session()

def get_chunk_http(n):
    r = s.get(url_base + f'/{n}')
    r.raise_for_status()
    return r.content

%prun all_data = [get_chunk_http(n) for n in range(za.shape[0] // za.chunks[0])] 

         12550435 function calls (12549147 primitive calls) in 98.508 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2576   87.798    0.034   87.798    0.034 {built-in method _openssl.SSL_read}
       13    1.436    0.110    1.437    0.111 {built-in method _openssl.SSL_do_handshake}
   929936    1.042    0.000    2.224    0.000 os.py:674(__getitem__)

As expected, reusing a connection pool sped things up, but it still takes 100 s to read the array.

Finally, we can try the same thing with asyncio

import asyncio
import aiohttp
import time

async def get_chunk_http_async(n, session):
    url = url_base + f'/{n}'
    async with session.get(url) as r:
        r.raise_for_status()
        data = await r.read()
    return data

async with aiohttp.ClientSession() as session:
    tic = time.time()
    all_data = await asyncio.gather(*[get_chunk_http_async(n, session)
                                    for n in range(za.shape[0] // za.chunks[0])])
    print(f"{time.time() - tic} seconds")

# > 1.7969944477081299 seconds

This is a MAJOR speedup!

I am aware that using dask could possibly help me here. But I don't have big data here, and I don't want to use dask. I want zarr to support asyncio natively.

I am quite new to async programming and have no idea how hard / complicated it would be to do this. But based on this experiment, I am quite sure there are major performance benefits to be had, particularly when using zarr with remote storage protocols.

Thoughts?

cc @cgentemann

This PR is for an early prototype of sharding support, as described in the corresponding issue #877. It serves mainly to discuss the overall implementation approach for sharding. This PR is not (yet) meant to be merged.

This prototype

• allows to specify shards as the number of chunks that should be contained in a shard (e.g. using arr.zeros((20, 3), chunks=(3, 3), shards=(2, 2), …)). One shard corresponds to one storage key, but can contain multiple chunks:
• ensures that this setting is persisted in the .zarray config and loaded when opening an array again, adding two entries:
  • "shard_format": "indexed" specifies the binary format of the shards and allows to extend sharding with other formats later
  • "shards": [2, 2] specifies how many chunks are contained in a shard,
• adds a IndexedShardedStore class that is used to wrap the chunk-store when sharding is enabled. This store handles the grouping of multiple chunks to one shard and transparently reads and writes them via the inner store in a binary format which is specified below. The original store API does not need to be adapted, it just stores shards instead of chunks, which are translated back to chunks by the IndexedShardedStore.
• adds a small script chunking_test.py for demonstration purposes, this is not meant to be merged but servers to illustrate the changes.

The currently implemented file format is still up for discussion. It implements "Format 2" @jbms describes in https://github.com/zarr-developers/zarr-python/pull/876#issuecomment-973462279.

Chunks are written successively in a shard (unused space between them is allowed), followed by an index referencing them. The index holding an offset, length pair of little-endian uint64 per chunk, the chunks-order in the index is row-major (C) order, e.g. for (2, 2) chunks per shard an index would look like:

| chunk (0, 0)    | chunk (0, 1)    | chunk (1, 0)    | chunk (1, 1)    |
| offset | length | offset | length | offset | length | offset | length |
| uint64 | uint64 | uint64 | uint64 | uint64 | uint64 | uint64 | uint64 |

Empty chunks are denoted by setting both offset and length to 2^64 - 1. All the index always has the full shape of all possible chunks per shard, even if they are outside of the array size.

For the default order of the actual chunk-content in a shard I'd propose to use Morton order, but this can easily be changed and customized, since any order can be read.

If the overall direction of this PR is pursued, the following steps (and possibly more) are missing:

• Functionality
  • [ ] Use a default write-order (Morton) of chunks in a shard and allow customization
  • [ ] Support deletion in the ShardedStore
  • [ ] Group chunk-wise operations in Array where possible (e.g. in digest & _resize_nosync)
  • [ ] Consider locking mechanisms to guard against concurrency issues within a shard
  • [ ] Allow partial reads and writes when the wrapped store supports them
  • [ ] Add support for prefixes before the chunk-dimensions in the storage key, e.g. for arrays that are contained in a group
  • [ ] Add warnings for inefficient reads/writes (might be configured)
  • [ ] Maybe use the new partial read method on the Store also for the current PartialReadBuffer usage (to detect if this is possible and reading via it)
• Tests
  • [ ] Add unit tests and/or doctests in docstrings
  • [ ] Test coverage is 100% (Codecov passes)
• Documentation
  • [ ] also document optional optimization possibilities on the Store or BaseStore class, such as getitems or partial reads
  • [ ] Add docstrings and API docs for any new/modified user-facing classes and functions
  • [ ] New/modified features documented in docs/tutorial.rst
  • [ ] Changes documented in docs/release.rst

changed 2021-12-07: added file format description and updated TODOs

A simple possible way of scanning all the metadata keys ('.zgroup'...) in a dataset and copying them into a single key, so that on systems where there is a substantial overhead to reading small files, everything can be grabbed in a single read. This is important in the context of xarray, which traverses all groups during opening the dataset, to find the various sub-groups and arrays.

The test shows how you could use the generated key. We could contemplate automatically looking for the metadata key when opening.

REF: https://github.com/pangeo-data/pangeo/issues/309

TODO:

• [x] Add unit tests and/or doctests in docstrings
• [x] Unit tests and doctests pass locally under Python 3.6 (e.g., run tox -e py36 or pytest -v --doctest-modules zarr)
• [x] Unit tests pass locally under Python 2.7 (e.g., run tox -e py27 or pytest -v zarr)
• [x] PEP8 checks pass (e.g., run tox -e py36 or flake8 --max-line-length=100 zarr)
• [x] Add docstrings and API docs for any new/modified user-facing classes and functions
• [x] New/modified features documented in docs/tutorial.rst
• [x] Doctests in tutorial pass (e.g., run tox -e py36 or python -m doctest -o NORMALIZE_WHITESPACE -o ELLIPSIS docs/tutorial.rst)
• [x] Changes documented in docs/release.rst
• [x] Docs build locally (e.g., run tox -e docs)
• [x] AppVeyor and Travis CI passes
• [ ] Test coverage to 100% (Coveralls passes)

First, apologies for submitting an un-solicited pull request. I know that is against the contributor guidelines. I thought this idea would be easier to discuss with a concrete implementation to look at.

In my highly opinionated view, the killer feature of zarr is its ability to efficiently store array data in cloud storage. Currently, the recommended way to do this is via outside packages (e.g. s3fs, gcsfs), which provide a MutableMapping that zarr can store things in.

In this PR, I have implemented an experimental google cloud storage class directly within zarr.

Why did I do this? Because, in the pangeo project, we are now making heavy use of the xarray -> zarr -> gcsfs -> cloud storage stack. I have come to the conclusion that a tighter coupling between zarr and gcs, via the google.cloud.storage API, may prove advantageous.

In addition to performance benefits and easier debugging, I think there are social advantages to having cloud storage as a first-class part of zarr. Lots of people want to store arrays in the cloud, and if zarr can provide this capability more natively, it could increase adoption.

Thoughts?

These tests require GCP credentials and the google cloud storage package. It is possible add encrypted credentials to travis, but I haven't done that yet. Tests are (mostly) working locally for me.

TODO:

• [x] Add unit tests and/or doctests in docstrings
• [ ] Unit tests and doctests pass locally under Python 3.6 (e.g., run tox -e py36 or pytest -v --doctest-modules zarr)
• [ ] Unit tests pass locally under Python 2.7 (e.g., run tox -e py27 or pytest -v zarr)
• [ ] PEP8 checks pass (e.g., run tox -e py36 or flake8 --max-line-length=100 zarr)
• [ ] Add docstrings and API docs for any new/modified user-facing classes and functions
• [ ] New/modified features documented in docs/tutorial.rst
• [ ] Doctests in tutorial pass (e.g., run tox -e py36 or python -m doctest -o NORMALIZE_WHITESPACE -o ELLIPSIS docs/tutorial.rst)
• [ ] Changes documented in docs/release.rst
• [ ] Docs build locally (e.g., run tox -e docs)
• [ ] AppVeyor and Travis CI passes
• [ ] Test coverage to 100% (Coveralls passes)

This adds support to read and write from and to N5 containers. The N5Store handling the conversion between the zarr and N5 format will automatically be selected whenever the path for a container ends in .n5 (similar to how the ZipStore is used for files ending in .zip).

The conversion is done mostly transparently, with one exception being the N5ChunkWrapper: This is a Codec with id n5_wrapper that will automatically be wrapped around the requested compressor. For example, if you create an array with a zlib compressor, in fact the n5_wrapper codec will be used that delegates to the zlib codec internally. The additional codec was necessary to introduce N5's chunk headers and ensure big endian storage.

In a related note, gzip compressed N5 arrays can at the moment not be read, since numcodecs treats zlib and gzip as synonyms, which they aren't (their compression headers differ). PR https://github.com/zarr-developers/numcodecs/pull/87 solves this issue.

See also https://github.com/zarr-developers/zarr/issues/231

TODO:

• [x] Add unit tests and/or doctests in docstrings
• [x] Unit tests and doctests pass locally under Python 3.6 (e.g., run tox -e py36 or pytest -v --doctest-modules zarr)
• [x] Unit tests pass locally under Python 2.7 (e.g., run tox -e py27 or pytest -v zarr)
• [x] PEP8 checks pass (e.g., run tox -e py36 or flake8 --max-line-length=100 zarr)
• [x] Add docstrings and API docs for any new/modified user-facing classes and functions
• [x] New/modified features documented in docs/tutorial.rst
• [x] Doctests in tutorial pass (e.g., run tox -e py36 or python -m doctest -o NORMALIZE_WHITESPACE -o ELLIPSIS docs/tutorial.rst)
• [x] Changes documented in docs/release.rst
• [x] Docs build locally (e.g., run tox -e docs)
• [x] AppVeyor and Travis CI passes
• [x] Test coverage to 100% (Coveralls passes)

I don't really understand how to use the new dimension_separator keyword, in particular:

1. Creating a DirectoryStore(dimension_separator="/") does not have the effect I would expect (see code and problem description below).
2. Why does zarr still have the NestedDirectoryStore? Shouldn't it be the same as DirectoryStore(dimension_separator="/")? Hence I would assume that NestedDirectoryStore could either be removed or (if to be kept for legacy purposes) should just map to DirectoryStore(dimension_seperator="/").

Minimal, reproducible code sample, a copy-pastable example if possible

import zarr
store = zarr.DirectoryStore("test.zarr", dimension_separator="/")
g = zarr.open(store, mode="a")
ds = g.create_dataset("test", shape=(10, 10, 10))
ds[:] = 1

Problem description

Now, I would assume that the chunks are nested, but I get:

$ ls test.zarr/test
0.0.0

but to, to my confusion, also this:

$ cat test.zarr/test/.zarray
...
"dimension_separator": "/",
...

If I use NestedDirectoryStore instead, the chunks are nested as expected.

Version and installation information

Please provide the following:

• Value of zarr.__version__: 2.8.3
• Value of numcodecs.__version__: 0.7.3
• Version of Python interpreter: 3.8.6
• Operating system: Linux
• How Zarr was installed: using conda

Addresses #657

This matches NumPy behaviour in that basic, boolean, and vectorized integer (fancy) indexing are all accessible from __{get,set}item__. Users still have access to all the indexing methods if they want to be sure to use only basic indexing (integer + slices).

I'm not 100% sure about the approach, but it seemed much easier to use a try/except than to try to detect all the cases when fancy indexing should be used. Happy to hear some guidance about how best to arrange that.

I still need to update docstrings + docs, will do that now — thanks for the checklist below. 😂

TODO:

• [x] Add unit tests and/or doctests in docstrings
• [x] Add docstrings and API docs for any new/modified user-facing classes and functions
• [x] New/modified features documented in docs/tutorial.rst
• [x] Changes documented in docs/release.rst
• [x] GitHub Actions have all passed
• [x] Test coverage is 100% (Codecov passes)

• Zarr has a lot of redundant files in the root directory. The information contained in these files can be easily moved to pyproject.toml.
• Latest Python PEPs encourage users to get rid of setup.py and switch to pyproject.toml
• We should not be using setup.cfg until it is a necessity - https://github.com/pypa/setuptools/issues/3214
• We should not be using setuptools as a frontend (python setup.py install) - this is not maintained (confirmed by setuptools developers, but I cannot find the exact issue number at this moment)
• Zarr should switch to pre-commit.ci and remove the pre-commit workflow

I have tried to perform a 1:1 port. No extra information was added and all the existing metadata has been moved to pyproject.toml.

TODO:

• [ ] Add unit tests and/or doctests in docstrings
• [ ] Add docstrings and API docs for any new/modified user-facing classes and functions
• [ ] New/modified features documented in docs/tutorial.rst
• [ ] Changes documented in docs/release.rst
• [ ] GitHub Actions have all passed
• [ ] Test coverage is 100% (Codecov passes)

Fixes #540 Ref https://github.com/zarr-developers/zarr-python/pull/373#issuecomment-592722584 (@rabernat )

Introduces a short Store implementation for generic fsspec url+options. Allows both lowercasing of keys and choice between '.' and '/'-based ("nested") keys.

For testing, have hijacked the TestNestedDirectoryStore tests just as an example - this is not how things will remain.

[Description of PR]

TODO:

• [ ] Add unit tests and/or doctests in docstrings
• [ ] Add docstrings and API docs for any new/modified user-facing classes and functions
• [ ] New/modified features documented in docs/tutorial.rst
• [ ] Changes documented in docs/release.rst
• [ ] Docs build locally (e.g., run tox -e docs)
• [ ] AppVeyor and Travis CI passes
• [ ] Test coverage is 100% (Coveralls passes)

[Description of PR]

TODO:

• [ ] Add unit tests and/or doctests in docstrings
• [ ] Add docstrings and API docs for any new/modified user-facing classes and functions
• [ ] New/modified features documented in docs/tutorial.rst
• [x] Changes documented in docs/release.rst
• [x] GitHub Actions have all passed
• [x] Test coverage is 100% (Codecov passes)

Fixes #1308

Currently, zarr-python errors when reading a json file with non-ascii characters encoded with utf-8, however, Zarr.jl writes json files that include non-ascii characters using utf-8 encoding. This PR will enable zarr attributes written in Zarr.jl to be read by zarr-python.

TODO:

• [x] Add unit tests and/or doctests in docstrings
• [x] Add docstrings and API docs for any new/modified user-facing classes and functions
• [x] New/modified features documented in docs/tutorial.rst
• [x] Changes documented in docs/release.rst
• [ ] GitHub Actions have all passed
• [ ] Test coverage is 100% (Codecov passes)

Bumps numpy from 1.24.0 to 1.24.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps actions/setup-python from 4.3.0 to 4.4.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Closes https://github.com/zarr-developers/zarr-python/issues/1306

cc @ravwojdyla @djhoese

TODO:

• [x] Add unit tests and/or doctests in docstrings
• [ ] ~Add docstrings and API docs for any new/modified user-facing classes and functions~
• [ ] ~New/modified features documented in docs/tutorial.rst~
• [ ] Changes documented in docs/release.rst
• [x] GitHub Actions have all passed
• [x] Test coverage is 100% (Codecov passes)

Zarr version

2.13.4.dev68

Numcodecs version

0.11.0

Python Version

3.10.6

Operating System

Linux

Installation

With pip, using the instructions here https://zarr.readthedocs.io/en/stable/contributing.html

Description

I expect zarr to be able to read attribute files that contain non-ASCII characters, because JSON files use utf-8 encoding. However, currently, there is a check to error if the JSON file contains any non-ASCII characters.

Steps to reproduce

import zarr
import tempfile
tempdir = tempfile.mkdtemp()
f = open(tempdir + '/.zgroup','w', encoding='utf-8')
f.write('{"zarr_format":2}')
f.close()
f = open(tempdir + '/.zattrs','w', encoding='utf-8')
f.write('{"foo": "た"}')
f.close()
z = zarr.open(tempdir, mode='r')
z.attrs['foo']

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/github/zarr-python/zarr/attrs.py", line 74, in __getitem__
    return self.asdict()[item]
  File "/root/github/zarr-python/zarr/attrs.py", line 55, in asdict
    d = self._get_nosync()
  File "/root/github/zarr-python/zarr/attrs.py", line 48, in _get_nosync
    d = self.store._metadata_class.parse_metadata(data)
  File "/root/github/zarr-python/zarr/meta.py", line 104, in parse_metadata
    meta = json_loads(s)
  File "/root/github/zarr-python/zarr/util.py", line 56, in json_loads
    return json.loads(ensure_text(s, 'ascii'))
  File "/root/pyenv/zarr-dev/lib/python3.10/site-packages/numcodecs/compat.py", line 181, in ensure_text
    s = codecs.decode(s, encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 9: ordinal not in range(128)

Additional output

No response