With the most recent release, I am getting an error on consolidate. Something odd seems to be happening on the second dimension (based upon the error message).

Traceback (most recent call last):
  File "./bug.py", line 38, in <module>
    main()
  File "./bug.py", line 34, in main
    tiledb.consolidate(name)
  File "tiledb/libtiledb.pyx", line 4420, in tiledb.libtiledb.consolidate
  File "tiledb/libtiledb.pyx", line 387, in tiledb.libtiledb._raise_ctx_err
  File "tiledb/libtiledb.pyx", line 372, in tiledb.libtiledb._raise_tiledb_error
tiledb.libtiledb.TileDBError: [TileDB::Query] Error: Subarray out of bounds. subarray: [0, 27999, 0, 20699] domain: [0, 27999, 0, 20645]

To reproduce:

import tiledb
import numpy as np


def make_array(name, shape):

    filters = tiledb.FilterList([
        tiledb.ZstdFilter(),
    ])
    attrs = [
        tiledb.Attr(dtype=np.float32, filters=filters)
    ]
    domain = tiledb.Domain(tiledb.Dim(name="obs", domain=(0, shape[0] - 1), tile=min(shape[0], 200), dtype=np.uint32),
                           tiledb.Dim(name="var", domain=(0, shape[1] - 1), tile=min(shape[1], 100), dtype=np.uint32))

    schema = tiledb.ArraySchema(domain=domain, sparse=False, attrs=attrs,
                                cell_order='row-major', tile_order='row-major')
    tiledb.DenseArray.create(name, schema)


def main():

    shape = (28000, 20646)
    name = "X"
    make_array(name, shape)

    stride = int(np.power(10, np.around(np.log10(1e8 / shape[1]))))
    with tiledb.DenseArray(name, mode='w') as X:
        for row in range(0, shape[0], stride):
            lim = min(row+stride, shape[0])
            print(row, lim)
            X[row:lim, :] = np.random.rand(lim-row, shape[1])

        tiledb.consolidate(name)


if __name__ == '__main__':
    main()

This happens on travis:

import tiledb

python/3.8.6/site-packages/tiledb/__init__.py:38: in <module>
    ctypes.CDLL(lib_name)
python/3.8.6/lib/python3.8/ctypes/__init__.py:373: in __init__
    self._handle = _dlopen(self._name, mode)
E   OSError: libtiledb.so: cannot open shared object file: No such file or directory

/edit: yup, 0.8.1 specific. If I specify tiledb!=0.8.1 it installs 0.8 and my tests run

if sparse == False and (not index_dims or "index_col" not in kwargs): NameError: name 'kwargs' is not defined

kwargs was replaced with tiledb_args

Hi.

I have following scenario, in which I open a tiledb that contains a Dense Array. I apply QueryConditions on the tiledb obj and it complains that QueryConditions may only be applied to sparse arrays.

import numpy as np
import pandas as pd
import tiledb as td

# create the df

columns = ["index", "seqnames", "start", "end"]
records = np.array(
    [
        [0, "chr1", 0, 5],
        [1, "chr1", 6, 10],
        [2, "chr2", 11, 15],
        [3, "chr2", 16, 20],
        [4, "chr3", 21, 25],
    ]
)

df = pd.DataFrame(records, columns=columns)

# write to tiledb

td.from_pandas("./test_pandas", df)


# open tiledb with query

with td.open("./test_pandas", "r") as a:
    qc = td.QueryCondition("seqnames == 'chr1'")
    q = a.query(attr_cond=qc)

The traceback:

---------------------------------------------------------------------------
TileDBError                               Traceback (most recent call last)
test.py in <cell line: 28>()
     [168](file:///test.py?line=167) with td.open("./test_pandas", "r") as a:
     [169](file:///test.py?line=168)     qc = td.QueryCondition("seqnames == 'chr1'")
---> [170](file:///test.py?line=169)     q = a.query(attr_cond=qc)

File tiledb/libtiledb.pyx:4369, in tiledb.libtiledb.DenseArrayImpl.query()

File tiledb/libtiledb.pyx:4098, in tiledb.libtiledb.Query.__init__()

TileDBError: QueryConditions may only be applied to sparse arrays

The documentation implies that this would work with dense arrays. Is there a work around for this?

Code can reproduce:

import numpy as np
import tiledb

dense_array_name = 'test_1d'

ctx = tiledb.Ctx()

dom = tiledb.Domain(tiledb.Dim(ctx=ctx, domain=(0, 11), tile=12, dtype=np.int64), ctx=ctx)
schema = tiledb.ArraySchema(ctx=ctx, domain=dom, sparse=False,
                            attrs=[tiledb.Attr(ctx=ctx, dtype=np.int64)])

tiledb.DenseArray.create(dense_array_name, schema)

with tiledb.DenseArray(dense_array_name, ctx=ctx, mode='w') as A:
    A[:2] = np.arange(2)
    A[2:4] = np.arange(2, 4)
    A[4:6] = np.arange(4, 6)
    A[6:8] = np.arange(6, 8)
    A[8:10] = np.arange(8, 10)
    A[10:12] = np.arange(10, 12)
    
tiledb.consolidate(uri=dense_array_name)

I have just updated the tiledb-py version to 0.4.1 via conda. Problem still. My laptop is Macbook pro with Mac OS High Sierra. Python version is 3.7.3.

If a QueryCondition has leading white space, the QueryCondition parser will throw an error. It does not seem sensitive to intermediate or trailing white space.

Ideally, it would ignore all whitespace including NL, allowing multi-line strings, eg, in Python:

qc = tiledb.QueryCondition("""
         a == 'a'
      and
        b == 'b'
    """)

Example failure running TileDb-Py 0.17.5:

In [34]: tiledb.QueryCondition("a == 'a'")
Out[34]: QueryCondition(expression="a == 'a'")

In [35]: tiledb.QueryCondition(" a == 'a'")
---------------------------------------------------------------------------
IndentationError                          Traceback (most recent call last)
File ~/projects/soma-scratch/venv/lib/python3.10/site-packages/tiledb/query_condition.py:115, in QueryCondition.__post_init__(self)
    114 try:
--> 115     self.tree = ast.parse(self.expression, mode="eval")
    116 except:

File /usr/lib/python3.10/ast.py:50, in parse(source, filename, mode, type_comments, feature_version)
     49 # Else it should be an int giving the minor version for 3.x.
---> 50 return compile(source, filename, mode, flags,
     51                _feature_version=feature_version)

IndentationError: unexpected indent (<unknown>, line 1)

During handling of the above exception, another exception occurred:

TileDBError                               Traceback (most recent call last)
Cell In [35], line 1
----> 1 tiledb.QueryCondition(" a == 'a'")

File <string>:5, in __init__(self, expression, ctx)

File ~/projects/soma-scratch/venv/lib/python3.10/site-packages/tiledb/query_condition.py:117, in QueryCondition.__post_init__(self)
    115     self.tree = ast.parse(self.expression, mode="eval")
    116 except:
--> 117     raise tiledb.TileDBError(
    118         "Could not parse the given QueryCondition statement: "
    119         f"{self.expression}"
    120     )
    122 if not self.tree:
    123     raise tiledb.TileDBError(
    124         "The query condition statement could not be parsed properly. "
    125         "(Is this an empty expression?)"
    126     )

TileDBError: Could not parse the given QueryCondition statement:  a == 'a'

It seems that TileDB deadlocks when writing concurrently to an array in S3. It doesn't lock up with ThreadPoolExecutor, or when writing to a file:// array.

TileDB-Py 0.9.3 TileDB 2.3.2

from concurrent.futures.process import ProcessPoolExecutor

import numpy as np
from tiledb import *

array_name = 's3://...'
ctx = Ctx(...)

s = ArraySchema(domain=Domain(Dim('a', dtype=np.int8, domain=(0, 126))), attrs=[Attr('x', dtype=np.int32)], sparse=True)

SparseArray.create(array_name, schema=s, ctx=ctx)


def write(i):
    with SparseArray(array_name, mode='w', ctx=ctx) as A:
        A[i] = i  # Processes hang here


with ProcessPoolExecutor() as pool:
    for i in range(50):
        pool.submit(write, i)


with SparseArray(array_name, ctx=ctx) as A:
    print(A[:])

Hello,

I’m trying to port a Python program from an HDFS back end to S3. Running some simple tests I’m getting an uncaught exception from TileDB. As the s3 back ends, I'm generally using minio for development, through I've reproduced the same issue using a Ceph object store. Here’s the condensed example:

[email protected]:/tdmq-dist$ python3
Python 3.6.9 (default, Oct  8 2020, 12:12:24)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tiledb
>>> tiledb.__version__
'0.7.0
>>> service_info = {
...         'version' : '0.1',
...         'tiledb' : {
...             'storage.root' : 's3://firstbucket/',
...             'config': {
...                 "vfs.s3.aws_access_key_id": "tdm-user",
...                 "vfs.s3.aws_secret_access_key": "tdm-user-s3",
...                 "vfs.s3.endpoint_override": "minio:9000",
...                 "vfs.s3.scheme": "http",
...                 "vfs.s3.region": "",
...                 "vfs.s3.verify_ssl": "false",
...                 "vfs.s3.use_virtual_addressing": "false",
...                 "vfs.s3.use_multipart_upload": "false",
...                 "vfs.s3.logging_level": 'TRACE'
...                 }
...             }
...         }
>>> def clean_s3(tdmq_s3_service_info):
...     import tiledb
...     config = tiledb.Config(params=tdmq_s3_service_info['tiledb']['config'])
...     bucket = tdmq_s3_service_info['tiledb']['storage.root']
...     assert bucket.startswith('s3://')
...     ctx = tiledb.Ctx(config=config)
...     vfs = tiledb.VFS(ctx=ctx)
...     if vfs.is_bucket(bucket):
...         vfs.empty_bucket(bucket)
...     else:
...         vfs.create_bucket(bucket)
...     return tdmq_s3_service_info

The first one or two times I call clean_s3(service_info) it works fine.

>>> clean_s3(service_info)
log4j:WARN File option not set for appender [FSLOGGER].
log4j:WARN Are you using FileAppender instead of ConsoleAppender?
{'version': '0.1', 'tiledb': {'storage.root': 's3://firstbucket/', 'config': {'vfs.s3.aws_access_key_id': 'tdm-user', 'vfs.s3.aws_secret_access_key': 'tdm-user-s3', 'vfs.s3.endpoint_override': 'minio:9000', 'vfs.s3.scheme': 'http', 'vfs.s3.region': '', 'vfs.s3.verify_ssl': 'false', 'vfs.s3.use_virtual_addressing': 'false', 'vfs.s3.use_multipart_upload': 'false', 'vfs.s3.logging_level': 'TRACE'}}}
>>> clean_s3(tdmq_s3_service_info)
{'version': '0.1', 'tiledb': {'storage.root': 's3://firstbucket/', 'config': {'vfs.s3.aws_access_key_id': 'tdm-user', 'vfs.s3.aws_secret_access_key': 'tdm-user-s3', 'vfs.s3.endpoint_override': 'minio:9000', 'vfs.s3.scheme': 'http', 'vfs.s3.region': '', 'vfs.s3.verify_ssl': 'false', 'vfs.s3.use_virtual_addressing': 'false', 'vfs.s3.use_multipart_upload': 'false', 'vfs.s3.logging_level': 'TRACE'}}}

Then something breaks. Further calls to the function result in an uncaught exception in TileDB:

>>> clean_s3(s3_service_info)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 8, in clean_s3
  File "tiledb/libtiledb.pyx", line 5511, in tiledb.libtiledb.VFS.is_bucket
  File "tiledb/libtiledb.pyx", line 481, in tiledb.libtiledb._raise_ctx_err
  File "tiledb/libtiledb.pyx", line 466, in tiledb.libtiledb._raise_tiledb_error
tiledb.libtiledb.TileDBError: Error: Internal TileDB uncaught exception; basic_string::compare: __pos (which is 18446744073709551615) > this->size() (which is 4)
>>> clean_s3(service_info)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 8, in clean_s3
  File "tiledb/libtiledb.pyx", line 5511, in tiledb.libtiledb.VFS.is_bucket
  File "tiledb/libtiledb.pyx", line 481, in tiledb.libtiledb._raise_ctx_err
  File "tiledb/libtiledb.pyx", line 466, in tiledb.libtiledb._raise_tiledb_error
tiledb.libtiledb.TileDBError: Error: Internal TileDB uncaught exception; basic_string::compare: __pos (which is 18446744073709551615) > this->size() (which is 4)

Once this happens I can't seem to do any s3 operation. Here are some other information I have collected:

• Reinitializing the tiledb context has no effect.
• Trying to reload the module has no effect (i.e., importlib.reload(tiledb)).
• I can reproduce this with both minio and Ceph.
• Generally, things break after the first couple of times I've called the function, generally in quick succession. Sometimes it happens on the first call.
• I've tried distancing subsequent calls to clean_s3 by as much as 15 seconds, but the thing still breaks.
• I'm running in Docker, in a custom ubuntu-based image. TileDB-Py is installed via pip.
• I have condensed this example from a more complicated scenario and another example I put together (which I posted in the forum. In those cases, the exception was being generated by tiledb.object_type and by tiledb.DenseArray.create.

Given the value of _pos, I guess it's got something to do with an unsigned type being used where a signed one is expected -- or maybe not. Let me know I can be of help.

tiledb version 0.5.6

attempting to write zero-length unicode array triggers assertion

test case:

attrs = [tiledb.Attr(name="foo", dtype=np.unicode)]
domain = tiledb.Domain(tiledb.Dim(domain=(0, 1000), tile=1000))
schema = tiledb.ArraySchema(domain=domain, sparse=False, attrs=attrs)
tiledb.DenseArray.create("foo", schema)

with tiledb.DenseArray("foo", mode="w") as A:
    # change empty string to non-empty and things work
    val = np.array(['' for i in range(0, 1001)], dtype=np.unicode)
    A[:] = val

result:

# python foo.py
Traceback (most recent call last):
  File "foo.py", line 61, in <module>
    A[:] = np.zeros((1000,), dtype=np.unicode)
  File "tiledb/libtiledb.pyx", line 4175, in tiledb.libtiledb.DenseArrayImpl.__setitem__
  File "tiledb/libtiledb.pyx", line 274, in tiledb.libtiledb._write_array
  File "tiledb/np2buf.pyx", line 108, in tiledb.libtiledb.array_to_buffer
AssertionError

Hello there,

I'm working with a TileDB containing to Domains with negative values. When querying values I get:

tiledb.libtiledb.TileDBError: [TileDB::Query] Error: Subarray lower bound is larger than upper bound. subarray: [86.75, 86.5, 26.75, 26.75, 157498, 157498] domain: [-89.75, 89.75, -179.75, 179.75, 157498, 157863]

My query looks as follows:

data = self.data.query(attrs=["some_data"], coords=True)[-3.25,26.75,157498.4375]

Also the error message does not display the subarray I define in my query.

I ran this on TileDB-py 0.3.0 with TileDB 1.4.0.

It seems like the slicing operation from an SparseArray or a DenseArray would only support queries such as:

a[1, 30:40]

However, if I have a list of values to query (say 1, 4, 10), it would fail:

a[[1, 4, 10], 30:40]

The only way I can think of doing this is to iterate through the list and do a slice like:

a[1, 30:40] a[4, 30:40] a[10, 30:40]

Is there a more efficient to do this? I'd imagine doing this in python would be pretty slow. If I have more dimensions it can get more complicated. For instance, if I know the exact index for the things I want to query say for a 3d array:

a[[1, 4, 10], [20, 30, 40], 30:40]

Then I have to get the cartesian production of [1, 4, 10] and [20, 30, 40] and slice through the rest.

In zarr for instance, there's a special function to do this (https://zarr.readthedocs.io/en/stable/tutorial.html#orthogonal-indexing). Do you see this is a viable option to implement in TileDB?

Replace "private" constructor parameters (_capsule, _lt_obv) with CtxMixin classmethods (from_capsule, from_pybind11).Follow-up to https://github.com/TileDB-Inc/TileDB-Py/pull/1548.