This line is not PEP8-compliant, as app is an unused variable. But, looking at the documentation, I am hesitant to delete it as it creates an application instance if it does not exist. Can we just delete this? (cc @ctrueden, the author of said line).

Unfortunately, this code is not tested :frowning:

Hi,

I write very frequently scripts for Imagej in python using the Jython interpreter but starting to consider using directly python. For that I am considering these options:

• imglyb (your library)
• pyimagej [https://pypi.org/project/pyimagej/]
• https://nbviewer.jupyter.org/github/imagej/tutorials/blob/master/notebooks/ImageJ-Tutorials-and-Demo.ipynb

From the pure scripting approach, is there any significant differences among those?

The built-in scyjava array converters will eventually be written to use python memoryview, allowing for zero-copy wrapping of primitive Java arrays. It seems sensible that we would use this logic in imglyb to convert compatible Imgs in the Java -> python direction.

Goals of this PR:

• [x] Refactor imglyb/ directory into src/imglyb/ directory. See this article for the benefits
• [x] Minimize setup.py in favor of setup.cfg. This seems to be the preferred approach nowadays.
• [x] Add a linting strategy (using black)
• [x] Add a code coverage strategy
• [x] Allow these features to block merge through Github Actions

Closes #15

When wrapping a numpy array to an ImgLib2 image, the dimension order is reversed. For example, a 3D numpy array dimensioned [3, 5, 7] would become a 3D RandomAccessibleInterval dimensioned [7, 5, 3]. This PR attempts to fix that inconsistency such that dimensions stay consistent across the two worlds.

A numpy array has two different orders:

C is column-contiguous order in memory (last index varies the fastest). C order means that operating row-rise on the array will be slightly quicker. FORTRAN-contiguous order in memory (first index varies the fastest). F order means that column-wise operations will be faster.

ImgLib2 generally prefers F order. In particular, the Views.flatIterable method returns an IterableInterval in F order, and the IntervalIndexer methods perform F-ordered rasterization.

On the Python side: if you do not specify an order, then numpy arrays default to C order.

@hanslovsky Is this discrepancy between ImgLib2's general preference and NumPy's default preference the reason for inverting the axes? Or is there another reason?

Since NumPy supports both C and F, my strong vote is to discontinue this dimension flipping behavior, in favor of consistently keeping everything the same.

@hanslovsky If you agree, I can polish up this PR. Certainly, I would add unit tests to prove that all works as expected in the various cases.

Alternately, if breaking existing behavior makes you nervous, we could add a flip_dimensions flag to the to_imglib method, defaulting to True. I would still change pyimagej to pass False for this flag though, because I think it's good to reduce the number of things our less technical users need to know about and work around.

Hi,

It appears I am not able to use the imglyb.to_numpy(rai) method

import imglyb
import scyjava

scyjava.start_jvm()

Views = imglyb.util.Views

ArrayImgs = scyjava.jimport("net.imglib2.img.array.ArrayImgs")

img = ArrayImgs.floats(32,32,32)
print(img.dimensionsAsLongArray())

arr = imglyb.to_numpy(img)

yields

[32, 32, 32]
Traceback (most recent call last):
  File "C:\Users\cameron.arshadi\Desktop\repos\imglyb\examples\wrap_arraylike.py", line 13, in <module>
    arr = imglyb.to_numpy(img)
          ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\cameron.arshadi\Miniconda3\envs\imglyb-test\Lib\site-packages\imglyb\__init__.py", line 25, in to_numpy
    return _ImgLibReferenceGuard(source)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\cameron.arshadi\Miniconda3\envs\imglyb-test\Lib\site-packages\imglyb\imglib_ndarray.py", line 67, in __new__
    address = get_address(rai)
              ^^^^^^^^^^^^^^^^
  File "C:\Users\cameron.arshadi\Miniconda3\envs\imglyb-test\Lib\site-packages\imglyb\imglib_ndarray.py", line 50, in get_address
    access = jpype.JObject(class_name_full, rai).update(None)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\cameron.arshadi\Miniconda3\envs\imglyb-test\Lib\site-packages\jpype\_jobject.py", line 59, in __new__
    return _JObjectFactory(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\cameron.arshadi\Miniconda3\envs\imglyb-test\Lib\site-packages\jpype\_jobject.py", line 113, in _JObjectFactory
    raise TypeError("Invalid type conversion to %s requested." % tp)
TypeError: Invalid type conversion to ArrayImg [32x32x32] requested.

I used a fresh conda environment on WIndows 10

conda create -n imglyb-test -c conda-forge python=3.9 imglyb
conda activate imglyb-test

Downgrading jpype to 1.4.0 and 1.3.0 did not help

Output of conda list

# packages in environment at C:\Users\cameron.arshadi\Miniconda3\envs\imglyb-test:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h8ffe710_4    conda-forge
ca-certificates           2022.12.7            h5b45459_0    conda-forge
imglyb                    2.0.1              pyh8a188c0_0    conda-forge
intel-openmp              2022.2.1         h57928b3_19741    conda-forge
jgo                       1.0.4              pyhd8ed1ab_0    conda-forge
jpype1                    1.3.0            py39h2e07f2f_2    conda-forge
libblas                   3.9.0              16_win64_mkl    conda-forge
libcblas                  3.9.0              16_win64_mkl    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
libhwloc                  2.8.0                h039e092_1    conda-forge
libiconv                  1.17                 h8ffe710_0    conda-forge
liblapack                 3.9.0              16_win64_mkl    conda-forge
libsqlite                 3.40.0               hcfcfb64_0    conda-forge
libxml2                   2.10.3               hc3477c8_0    conda-forge
libzlib                   1.2.13               hcfcfb64_4    conda-forge
maven                     3.8.6                h57928b3_0    conda-forge
mkl                       2022.1.0           h6a75c08_874    conda-forge
numpy                     1.23.5           py39hbccbffa_0    conda-forge
openjdk                   17.0.3               h57928b3_4    conda-forge
openssl                   3.0.7                hcfcfb64_1    conda-forge
packaging                 22.0               pyhd8ed1ab_0    conda-forge
pip                       22.3.1             pyhd8ed1ab_0    conda-forge
psutil                    5.9.4            py39ha55989b_0    conda-forge
pthreads-win32            2.9.1                hfa6e2cd_3    conda-forge
python                    3.9.15          h4de0772_0_cpython    conda-forge
python_abi                3.9                      3_cp39    conda-forge
scyjava                   1.8.1              pyhd8ed1ab_0    conda-forge
setuptools                65.5.1             pyhd8ed1ab_0    conda-forge
symlink-exe-runtime       1.0                  hcfcfb64_0    conda-forge
tbb                       2021.7.0             h91493d7_1    conda-forge
tk                        8.6.12               h8ffe710_0    conda-forge
tzdata                    2022g                h191b570_0    conda-forge
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
vc                        14.3                 h3d8a991_9    conda-forge
vs2015_runtime            14.32.31332          h1d6e394_9    conda-forge
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h8d14728_0    conda-forge

This could, possibly, be a separate package, e.g. imglyb_bdv or imglyb_vistools. It is not always necessary (or desired) to have the BDV dependencies on the classpath with imglyb.

Alternatively, this could be controlled through imglyb_config.

cc @ctrueden

The setup.py file lists jnius as a requirement. Jnius has been renamed to pyjnius, and so this can send conda installation warnings when installing imglyb, which can be confusing for new users.

This change just renames the requirement to pyjnius to reduce confusion.

While there is generally no consensus on how python projects should be packaged, there seem to be some good benefits to the src package structure and the Python Packaging Authority now promotes this structure.

https://github.com/pypa/packaging.python.org/issues/320 describes how they came to the determination they did.

We should consider adding it to imglyb.

HOME env not defined on windows

See also: https://forum.image.sc/t/analysis-with-imagej-and-visualization-in-the-jupyter-notebook/11052/27?u=hanslovsky

I added support for converting dask arrays to imglib2 CachedCellImg but it does not seem to work correctly:

I spent a lot of time trying to fix that and my personal interest in dask is very low, so I will not continue working on this. Contributions always welcome.

Relevant code: https://github.com/imglib/imglyb/blob/a47f6793e3b269f48c3ee397ed542388b9e1cbda/test.py https://github.com/imglib/imglyb/blob/a47f6793e3b269f48c3ee397ed542388b9e1cbda/imglyb/cell.py https://github.com/imglib/imglib2-imglyb/blob/401951b23146fee0e1bc4dd4a4f9b9302df7b8d4/src/main/java/net/imglib2/python/Helpers.java#L111-L146

Using JNI's NewDirectByteBuffer function, one can wrap an existing memory address and length into an off-heap ByteBuffer. Now that ImgLib2 supports ByteBuffer-backed data, we could make use of this to have zero-copy access to NumPy arrays from Java without the Unsafe hacks currently used. This would hopefully make the imglib2-unsafe library obsolete.

~~We need to determine the best way to call that JNI function. It is used in the JPype project in a few places; perhaps there is an existing JPype call we can use, without needing to resort to our own Cython code or use of JNA...~~ Edit: It should be as simple as calling jpype.nio.convertToDirectBuffer(narr) on the numpy array and feeding the resultant DirectByteBuffer to ImgLib2. After verifying this indeed does not copy the data, of course.

This PR supports the conversion from numpy arrays with dtype np.bool8 to RandomAccessibleInterval<NativeBoolType>s, and adds a regression test to ensure this conversion is possible. (It actually adds more regression tests than that).

This PR is the minimal possible change required to implement this functionality.

Conversion the other way is tricky, as we have discussed in #13, and the PRs that this work leans on are not enough to implement that conversion. The main issue lies in creating an Image of some BooleanType that can be converted. I am unable to create an UnsafeImg of any BooleanType, and although I can create an ArrayImg<BitType>, I cannot convert that using imglyb.to_numpy. Therefore, I leave that work for another PR.

This PR requires the work of imglib/imglib2-unsafe#8 and imglib/imglib2-imglyb#10 to make their way into releases before this will work. I have tested locally with snapshots of each.

imglyb does not support converting bool type images to numpy. This means that users are unable to convert masks/thresholded images into python numpy/xarray objects.

Here is a minimal example:

import imagej
import scyjava as sj

# initialize
ij = imagej.init()

# load blobs
img = ij.io().open('blobs.tif')

# threshold image w/ ops
CreateNamespace = sj.jimport('net.imagej.ops.create.CreateNamespace')

img_invert = ij.op().namespace(CreateNamespace).img(img)
ij.op().image().invert(img_invert, img)
threshold = ij.op().threshold().otsu(img_invert)

# get threshold from java
py_threshold = ij.py.from_java(threshold)

Returns the traceback:

File "/home/edward/Documents/repos/loci/imglyb/imglyb/util.py", line 149, in _to_imglib
    raise NotImplementedError("Cannot convert dtype to ImgLib2 type yet: {}".format(source.dtype))
NotImplementedError: Cannot convert dtype to ImgLib2 type yet: bool

Similar to what is done with wrapping as cached cell images (#7).

I am not 100% sure about this one. The user still needs to make sure that the reference store stays within scope as long as necessary. Instead, callers could do the same thing for the actual ndarray, and a reference store may be obsolete here.