Read and write TIFF files

Related tags

File & Path Utilitiespythonimageimagejnumpytiffformat-readerlife-sciences-imageome-tiffwhole-slide-imagingzarr
Overview

Read and write TIFF files

Tifffile is a Python library to

  1. store numpy arrays in TIFF (Tagged Image File Format) files, and
  2. read image and metadata from TIFF-like files used in bioimaging.

Image and metadata can be read from TIFF, BigTIFF, OME-TIFF, STK, LSM, SGI, NIHImage, ImageJ, MicroManager, FluoView, ScanImage, SEQ, GEL, SVS, SCN, SIS, BIF, ZIF (Zoomable Image File Format), QPTIFF (QPI), NDPI, and GeoTIFF files.

Image data can be read as numpy arrays or zarr arrays/groups from strips, tiles, pages (IFDs), SubIFDs, higher order series, and pyramidal levels.

Numpy arrays can be written to TIFF, BigTIFF, OME-TIFF, and ImageJ hyperstack compatible files in multi-page, volumetric, pyramidal, memory-mappable, tiled, predicted, or compressed form.

A subset of the TIFF specification is supported, mainly 8, 16, 32 and 64-bit integer, 16, 32 and 64-bit float, grayscale and multi-sample images. Specifically, CCITT and OJPEG compression, chroma subsampling without JPEG compression, color space transformations, samples with differing types, or IPTC, ICC, and XMP metadata are not implemented.

TIFF, the Tagged Image File Format, was created by the Aldus Corporation and Adobe Systems Incorporated. BigTIFF allows for files larger than 4 GB. STK, LSM, FluoView, SGI, SEQ, GEL, QPTIFF, NDPI, SCN, SVS, ZIF, BIF, and OME-TIFF, are custom extensions defined by Molecular Devices (Universal Imaging Corporation), Carl Zeiss MicroImaging, Olympus, Silicon Graphics International, Media Cybernetics, Molecular Dynamics, PerkinElmer, Hamamatsu, Leica, ObjectivePathology, Roche Digital Pathology, and the Open Microscopy Environment consortium, respectively.

For command line usage run python -m tifffile --help

Author: Christoph Gohlke
Organization: Laboratory for Fluorescence Dynamics, University of California, Irvine
License: BSD 3-Clause
Version: 2021.11.2

Requirements

This release has been tested with the following requirements and dependencies (other versions may work):

Revisions

2021.11.2
Pass 4731 tests. Lazy-load non-essential tag values (breaking). Warn when reading from closed file. Support ImageJ 'prop' metadata type (#103). Support writing indexed ImageJ format. Fix multi-threaded access of multi-page Zarr stores with chunkmode 2. Raise error if truncate is used with compression, packints, or tile. Read STK metadata without UIC2tag. Improve log and warning messages (WIP). Improve string representation of large tag values.
2021.10.12
Revert renaming of 'file' parameter in FileSequence.asarray (breaking). Deprecate 'file' parameter in FileSequence.asarray.
2021.10.10
Disallow letters as indices in FileSequence; use categories (breaking). Do not warn of missing files in FileSequence; use files_missing property. Support predictors in ZarrTiffStore.write_fsspec. Add option to specify zarr group name in write_fsspec. Add option to specify categories for FileSequence patterns (#76). Add option to specify chunk shape and dtype for ZarrFileSequenceStore. Add option to tile ZarrFileSequenceStore and FileSequence.asarray. Add option to pass additional zattrs to Zarr stores. Detect Roche BIF files.
2021.8.30
Fix horizontal differencing with non-native byte order. Fix multi-threaded access of memory-mappable, multi-page Zarr stores (#67).
2021.8.8
Fix tag offset and valueoffset for NDPI > 4 GB (#96).
2021.7.30
Deprecate first parameter to TiffTag.overwrite (no longer required). TiffTag init API change (breaking). Detect Ventana BIF series and warn that tiles are not stitched. Enable reading PreviewImage from RAW formats (#93, #94). Work around numpy.ndarray.tofile is very slow for non-contiguous arrays. Fix issues with PackBits compression (requires imagecodecs 2021.7.30).
2021.7.2
Decode complex integer images found in SAR GeoTIFF. Support reading NDPI with JPEG-XR compression. Deprecate TiffWriter RGB auto-detection, except for RGB24/48 and RGBA32/64.
2021.6.14
Set stacklevel for deprecation warnings (#89). Fix svs_description_metadata for SVS with double header (#88, breaking). Fix reading JPEG compressed CMYK images. Support ALT_JPEG and JPEG_2000_LOSSY compression found in Bio-Formats. Log warning if TiffWriter auto-detects RGB mode (specify photometric).
2021.6.6
Fix TIFF.COMPESSOR typo (#85). Round resolution numbers that do not fit in 64-bit rationals (#81). Add support for JPEG XL compression. Add numcodecs compatible TIFF codec. Rename ZarrFileStore to ZarrFileSequenceStore (breaking). Add method to export fsspec ReferenceFileSystem from ZarrFileStore. Fix fsspec ReferenceFileSystem v1 for multifile series. Fix creating OME-TIFF with micron character in OME-XML.
2021.4.8
Fix reading OJPEG with wrong photometric or samplesperpixel tags (#75). Fix fsspec ReferenceFileSystem v1 and JPEG compression. Use TiffTagRegistry for NDPI_TAGS, EXIF_TAGS, GPS_TAGS, IOP_TAGS constants. Make TIFF.GEO_KEYS an Enum (breaking).
2021.3.31
Use JPEG restart markers as tile offsets in NDPI. Support version 1 and more codecs in fsspec ReferenceFileSystem (untested).
2021.3.17
Fix regression reading multi-file OME-TIFF with missing files (#72). Fix fsspec ReferenceFileSystem with non-native byte order (#56).
2021.3.16
TIFF is no longer a defended trademark. Add method to export fsspec ReferenceFileSystem from ZarrTiffStore (#56).
2021.3.5
Preliminary support for EER format (#68). Do not warn about unknown compression (#68).
2021.3.4
Fix reading multi-file, multi-series OME-TIFF (#67). Detect ScanImage 2021 files (#46). Shape new version ScanImage series according to metadata (breaking). Remove Description key from TiffFile.scanimage_metadata dict (breaking). Also return ScanImage version from read_scanimage_metadata (breaking). Fix docstrings.
2021.2.26
Squeeze axes of LSM series by default (breaking). Add option to preserve single dimensions when reading from series (WIP). Do not allow appending to OME-TIFF files. Fix reading STK files without name attribute in metadata. Make TIFF constants multi-thread safe and pickleable (#64). Add detection of NDTiffStorage MajorVersion to read_micromanager_metadata. Support ScanImage v4 files in read_scanimage_metadata.
2021.2.1
Fix multi-threaded access of ZarrTiffStores using same TiffFile instance. Use fallback zlib and lzma codecs with imagecodecs lite builds. Open Olympus and Panasonic RAW files for parsing, albeit not supported. Support X2 and X4 differencing found in DNG. Support reading JPEG_LOSSY compression found in DNG.
2021.1.14
Try ImageJ series if OME series fails (#54) Add option to use pages as chunks in ZarrFileStore (experimental). Fix reading from file objects with no readinto function.
2021.1.11
Fix test errors on PyPy. Fix decoding bitorder with imagecodecs >= 2021.1.11.
2021.1.8
Decode float24 using imagecodecs >= 2021.1.8. Consolidate reading of segments if possible.
2020.12.8
Fix corrupted ImageDescription in multi shaped series if buffer too small. Fix libtiff warning that ImageDescription contains null byte in value. Fix reading invalid files using JPEG compression with palette colorspace.
2020.12.4
Fix reading some JPEG compressed CFA images. Make index of SubIFDs a tuple. Pass through FileSequence.imread arguments in imread. Do not apply regex flags to FileSequence axes patterns (breaking).
2020.11.26
Add option to pass axes metadata to ImageJ writer. Pad incomplete tiles passed to TiffWriter.write (#38). Split TiffTag constructor (breaking). Change TiffTag.dtype to TIFF.DATATYPES (breaking). Add TiffTag.overwrite method. Add script to change ImageDescription in files. Add TiffWriter.overwrite_description method (WIP).
2020.11.18
Support writing SEPARATED color space (#37). Use imagecodecs.deflate codec if available. Fix SCN and NDPI series with Z dimensions. Add TiffReader alias for TiffFile. TiffPage.is_volumetric returns True if ImageDepth > 1. Zarr store getitem returns numpy arrays instead of bytes.
2020.10.1
Formally deprecate unused TiffFile parameters (scikit-image #4996).
2020.9.30
Allow to pass additional arguments to compression codecs. Deprecate TiffWriter.save method (use TiffWriter.write). Deprecate TiffWriter.save compress parameter (use compression). Remove multifile parameter from TiffFile (breaking). Pass all is_flag arguments from imread to TiffFile. Do not byte-swap JPEG2000, WEBP, PNG, JPEGXR segments in TiffPage.decode.
2020.9.29
Fix reading files produced by ScanImage > 2015 (#29).
2020.9.28
Derive ZarrStore from MutableMapping. Support zero shape ZarrTiffStore. Fix ZarrFileStore with non-TIFF files. Fix ZarrFileStore with missing files. Cache one chunk in ZarrFileStore. Keep track of already opened files in FileCache. Change parse_filenames function to return zero-based indices. Remove reopen parameter from asarray (breaking). Rename FileSequence.fromfile to imread (breaking).
2020.9.22
Add experimental zarr storage interface (WIP). Remove unused first dimension from TiffPage.shaped (breaking). Move reading of STK planes to series interface (breaking). Always use virtual frames for ScanImage files. Use DimensionOrder to determine axes order in OmeXml. Enable writing striped volumetric images. Keep complete dataoffsets and databytecounts for TiffFrames. Return full size tiles from Tiffpage.segments. Rename TiffPage.is_sgi property to is_volumetric (breaking). Rename TiffPageSeries.is_pyramid to is_pyramidal (breaking). Fix TypeError when passing jpegtables to non-JPEG decode method (#25).
2020.9.3
Do not write contiguous series by default (breaking). Allow to write to SubIFDs (WIP). Fix writing F-contiguous numpy arrays (#24).
2020.8.25
Do not convert EPICS timeStamp to datetime object. Read incompletely written Micro-Manager image file stack header (#23). Remove tag 51123 values from TiffFile.micromanager_metadata (breaking).
2020.8.13
Use tifffile metadata over OME and ImageJ for TiffFile.series (breaking). Fix writing iterable of pages with compression (#20). Expand error checking of TiffWriter data, dtype, shape, and tile arguments.
2020.7.24
Parse nested OmeXml metadata argument (WIP). Do not lazy load TiffFrame JPEGTables. Fix conditionally skipping some tests.
2020.7.22
Do not auto-enable OME-TIFF if description is passed to TiffWriter.save. Raise error writing empty bilevel or tiled images. Allow to write tiled bilevel images. Allow to write multi-page TIFF from iterable of single page images (WIP). Add function to validate OME-XML. Correct Philips slide width and length.
2020.7.17
Initial support for writing OME-TIFF (WIP). Return samples as separate dimension in OME series (breaking). Fix modulo dimensions for multiple OME series. Fix some test errors on big endian systems (#18). Fix BytesWarning. Allow to pass TIFF.PREDICTOR values to TiffWriter.save.
2020.7.4
Deprecate support for Python 3.6 (NEP 29). Move pyramidal subresolution series to TiffPageSeries.levels (breaking). Add parser for SVS, SCN, NDPI, and QPI pyramidal series. Read single-file OME-TIFF pyramids. Read NDPI files > 4 GB (#15). Include SubIFDs in generic series. Preliminary support for writing packed integer arrays (#11, WIP). Read more LSM info subrecords. Fix missing ReferenceBlackWhite tag for YCbCr photometrics. Fix reading lossless JPEG compressed DNG files.
2020.6.3
...

Refer to the CHANGES file for older revisions.

Notes

The API is not stable yet and might change between revisions.

Tested on little-endian platforms only.

Python 32-bit versions are deprecated. Python <= 3.7 are no longer supported.

Tifffile relies on the imagecodecs package for encoding and decoding LZW, JPEG, and other compressed image segments.

Several TIFF-like formats do not strictly adhere to the TIFF6 specification, some of which allow file or data sizes to exceed the 4 GB limit:

  • BigTIFF is identified by version number 43 and uses different file header, IFD, and tag structures with 64-bit offsets. It adds more data types. Tifffile can read and write BigTIFF files.
  • ImageJ hyperstacks store all image data, which may exceed 4 GB, contiguously after the first IFD. Files > 4 GB contain one IFD only. The size (shape and dtype) of the up to 6-dimensional image data can be determined from the ImageDescription tag of the first IFD, which is Latin-1 encoded. Tifffile can read and write ImageJ hyperstacks.
  • OME-TIFF stores up to 8-dimensional data in one or multiple TIFF of BigTIFF files. The 8-bit UTF-8 encoded OME-XML metadata found in the ImageDescription tag of the first IFD defines the position of TIFF IFDs in the high dimensional data. Tifffile can read OME-TIFF files, except when the OME-XML metadata are stored in a separate file. Tifffile can write numpy arrays to single-file OME-TIFF.
  • LSM stores all IFDs below 4 GB but wraps around 32-bit StripOffsets. The StripOffsets of each series and position require separate unwrapping. The StripByteCounts tag contains the number of bytes for the uncompressed data. Tifffile can read large LSM files.
  • STK (MetaMorph Stack) contains additional image planes stored contiguously after the image data of the first page. The total number of planes is equal to the counts of the UIC2tag. Tifffile can read STK files.
  • NDPI uses some 64-bit offsets in the file header, IFD, and tag structures. Tag values/offsets can be corrected using high bits stored after IFD structures. Tifffile can read NDPI files > 4 GB. JPEG compressed segments with dimensions >65530 or missing restart markers are not decodable with libjpeg. Tifffile works around this limitation by separately decoding the MCUs between restart markers. BitsPerSample, SamplesPerPixel, and PhotometricInterpretation tags may contain wrong values, which can be corrected using the value of tag 65441.
  • Philips TIFF slides store wrong ImageWidth and ImageLength tag values for tiled pages. The values can be corrected using the DICOM_PIXEL_SPACING attributes of the XML formatted description of the first page. Tifffile can read Philips slides.
  • Ventana/Roche BIF slides store tiles and metadata in a BigTIFF container. Tiles may overlap and require stitching based on the TileJointInfo elements in the XMP tag. Volumetric scans are stored using the ImageDepth extension. Tifffile can read BIF and decode individual tiles, but does not perform stitching.
  • ScanImage optionally allows corrupted non-BigTIFF files > 2 GB. The values of StripOffsets and StripByteCounts can be recovered using the constant differences of the offsets of IFD and tag values throughout the file. Tifffile can read such files if the image data are stored contiguously in each page.
  • GeoTIFF sparse files allow strip or tile offsets and byte counts to be 0. Such segments are implicitly set to 0 or the NODATA value on reading. Tifffile can read GeoTIFF sparse files.

Other libraries for reading scientific TIFF files from Python:

Some libraries are using tifffile to write OME-TIFF files:

Other tools for inspecting and manipulating TIFF files:

References

Examples

Write a numpy array to a single-page RGB TIFF file:

>>> data = numpy.random.randint(0, 255, (256, 256, 3), 'uint8')
>>> imwrite('temp.tif', data, photometric='rgb')

Read the image from the TIFF file as numpy array:

>>> image = imread('temp.tif')
>>> image.shape
(256, 256, 3)

Write a 3D numpy array to a multi-page, 16-bit grayscale TIFF file:

>>> data = numpy.random.randint(0, 2**12, (64, 301, 219), 'uint16')
>>> imwrite('temp.tif', data, photometric='minisblack')

Read the whole image stack from the TIFF file as numpy array:

>>> image_stack = imread('temp.tif')
>>> image_stack.shape
(64, 301, 219)
>>> image_stack.dtype
dtype('uint16')

Read the image from the first page in the TIFF file as numpy array:

>>> image = imread('temp.tif', key=0)
>>> image.shape
(301, 219)

Read images from a selected range of pages:

>>> images = imread('temp.tif', key=range(4, 40, 2))
>>> images.shape
(18, 301, 219)

Iterate over all pages in the TIFF file and successively read images:

>>> with TiffFile('temp.tif') as tif:
...     for page in tif.pages:
...         image = page.asarray()

Get information about the image stack in the TIFF file without reading the image data:

>>> tif = TiffFile('temp.tif')
>>> len(tif.pages)  # number of pages in the file
64
>>> page = tif.pages[0]  # get shape and dtype of the image in the first page
>>> page.shape
(301, 219)
>>> page.dtype
dtype('uint16')
>>> page.axes
'YX'
>>> series = tif.series[0]  # get shape and dtype of the first image series
>>> series.shape
(64, 301, 219)
>>> series.dtype
dtype('uint16')
>>> series.axes
'QYX'
>>> tif.close()

Inspect the "XResolution" tag from the first page in the TIFF file:

>>> with TiffFile('temp.tif') as tif:
...     tag = tif.pages[0].tags['XResolution']
>>> tag.value
(1, 1)
>>> tag.name
'XResolution'
>>> tag.code
282
>>> tag.count
1
>>> tag.dtype

Iterate over all tags in the TIFF file:

>>> with TiffFile('temp.tif') as tif:
...     for page in tif.pages:
...         for tag in page.tags:
...             tag_name, tag_value = tag.name, tag.value

Overwrite the value of an existing tag, e.g. XResolution:

>>> with TiffFile('temp.tif', mode='r+b') as tif:
...     _ = tif.pages[0].tags['XResolution'].overwrite((96000, 1000))

Write a floating-point ndarray and metadata using BigTIFF format, tiling, compression, and planar storage:

>>> data = numpy.random.rand(2, 5, 3, 301, 219).astype('float32')
>>> imwrite('temp.tif', data, bigtiff=True, photometric='minisblack',
...         compression='zlib', planarconfig='separate', tile=(32, 32),
...         metadata={'axes': 'TZCYX'})

Write a 10 fps time series of volumes with xyz voxel size 2.6755x2.6755x3.9474 micron^3 to an ImageJ hyperstack formatted TIFF file:

>>> volume = numpy.random.randn(6, 57, 256, 256).astype('float32')
>>> imwrite('temp.tif', volume, imagej=True, resolution=(1./2.6755, 1./2.6755),
...         metadata={'spacing': 3.947368, 'unit': 'um', 'finterval': 1/10,
...                   'axes': 'TZYX'})

Read the volume and metadata from the ImageJ file:

>>> with TiffFile('temp.tif') as tif:
...     volume = tif.asarray()
...     axes = tif.series[0].axes
...     imagej_metadata = tif.imagej_metadata
>>> volume.shape
(6, 57, 256, 256)
>>> axes
'TZYX'
>>> imagej_metadata['slices']
57
>>> imagej_metadata['frames']
6

Create a TIFF file containing an empty image and write to the memory-mapped numpy array:

>>> memmap_image = memmap(
...     'temp.tif', shape=(256, 256, 3), dtype='float32', photometric='rgb'
... )
>>> type(memmap_image)

   
>>> memmap_image[
   255, 
   255, 
   1] 
   = 
   1.0
>>> memmap_image.flush()
>>> 
   del memmap_image

Memory-map and read contiguous image data in the TIFF file:

>>> memmap_image = memmap('temp.tif')
>>> memmap_image.shape
(256, 256, 3)
>>> memmap_image[255, 255, 1]
1.0
>>> del memmap_image

Write two numpy arrays to a multi-series TIFF file:

>>> series0 = numpy.random.randint(0, 255, (32, 32, 3), 'uint8')
>>> series1 = numpy.random.randint(0, 1023, (4, 256, 256), 'uint16')
>>> with TiffWriter('temp.tif') as tif:
...     tif.write(series0, photometric='rgb')
...     tif.write(series1, photometric='minisblack')

Read the second image series from the TIFF file:

>>> series1 = imread('temp.tif', series=1)
>>> series1.shape
(4, 256, 256)

Successively write the frames of one contiguous series to a TIFF file:

>>> data = numpy.random.randint(0, 255, (30, 301, 219), 'uint8')
>>> with TiffWriter('temp.tif') as tif:
...     for frame in data:
...         tif.write(frame, contiguous=True)

Append an image series to the existing TIFF file:

>>> data = numpy.random.randint(0, 255, (301, 219, 3), 'uint8')
>>> imwrite('temp.tif', data, photometric='rgb', append=True)

Create a TIFF file from a generator of tiles:

>>> data = numpy.random.randint(0, 2**12, (31, 33, 3), 'uint16')
>>> def tiles(data, tileshape):
...     for y in range(0, data.shape[0], tileshape[0]):
...         for x in range(0, data.shape[1], tileshape[1]):
...             yield data[y : y + tileshape[0], x : x + tileshape[1]]
>>> imwrite('temp.tif', tiles(data, (16, 16)), tile=(16, 16),
...         shape=data.shape, dtype=data.dtype, photometric='rgb')

Write two numpy arrays to a multi-series OME-TIFF file:

>>> series0 = numpy.random.randint(0, 255, (32, 32, 3), 'uint8')
>>> series1 = numpy.random.randint(0, 1023, (4, 256, 256), 'uint16')
>>> with TiffWriter('temp.ome.tif') as tif:
...     tif.write(series0, photometric='rgb')
...     tif.write(series1, photometric='minisblack',
...               metadata={'axes': 'ZYX', 'SignificantBits': 10,
...                         'Plane': {'PositionZ': [0.0, 1.0, 2.0, 3.0]}})

Write a tiled, multi-resolution, pyramidal, OME-TIFF file using JPEG compression. Sub-resolution images are written to SubIFDs:

>>> data = numpy.arange(1024*1024*3, dtype='uint8').reshape((1024, 1024, 3))
>>> with TiffWriter('temp.ome.tif', bigtiff=True) as tif:
...     options = dict(tile=(256, 256), photometric='rgb', compression='jpeg')
...     tif.write(data, subifds=2, **options)
...     # save pyramid levels to the two subifds
...     # in production use resampling to generate sub-resolutions
...     tif.write(data[::2, ::2], subfiletype=1, **options)
...     tif.write(data[::4, ::4], subfiletype=1, **options)

Access the image levels in the pyramidal OME-TIFF file:

>>> baseimage = imread('temp.ome.tif')
>>> second_level = imread('temp.ome.tif', series=0, level=1)
>>> with TiffFile('temp.ome.tif') as tif:
...     baseimage = tif.series[0].asarray()
...     second_level = tif.series[0].levels[1].asarray()

Iterate over and decode single JPEG compressed tiles in the TIFF file:

>>> with TiffFile('temp.ome.tif') as tif:
...     fh = tif.filehandle
...     for page in tif.pages:
...         for index, (offset, bytecount) in enumerate(
...             zip(page.dataoffsets, page.databytecounts)
...         ):
...             _ = fh.seek(offset)
...             data = fh.read(bytecount)
...             tile, indices, shape = page.decode(
...                 data, index, jpegtables=page.jpegtables
...             )

Use zarr to read parts of the tiled, pyramidal images in the TIFF file:

>>> import zarr
>>> store = imread('temp.ome.tif', aszarr=True)
>>> z = zarr.open(store, mode='r')
>>> z

   
>>> z[
   0]  
   # base layer

   
    
>>> z[
    0][
    256:
    512, 
    512:
    768].shape  
    # read a tile from the base layer
(256, 256, 3)
>>> store.close()

Read images from a sequence of TIFF files as numpy array:

>>> imwrite('temp_C001T001.tif', numpy.random.rand(64, 64))
>>> imwrite('temp_C001T002.tif', numpy.random.rand(64, 64))
>>> image_sequence = imread(['temp_C001T001.tif', 'temp_C001T002.tif'])
>>> image_sequence.shape
(2, 64, 64)
>>> image_sequence.dtype
dtype('float64')

Read an image stack from a series of TIFF files with a file name pattern as numpy or zarr arrays:

>>> image_sequence = TiffSequence('temp_C0*.tif', pattern=r'_(C)(\d+)(T)(\d+)')
>>> image_sequence.shape
(1, 2)
>>> image_sequence.axes
'CT'
>>> data = image_sequence.asarray()
>>> data.shape
(1, 2, 64, 64)
>>> with image_sequence.aszarr() as store:
...     zarr.open(store, mode='r')

   
>>> image_sequence.close()

Write the zarr store to a fsspec ReferenceFileSystem in JSON format:

>>> with image_sequence.aszarr() as store:
...     store.write_fsspec('temp.json', url='file://')

Open the fsspec ReferenceFileSystem as a zarr array:

>>> import fsspec
>>> import tifffile.numcodecs
>>> tifffile.numcodecs.register_codec()
>>> mapper = fsspec.get_mapper(
...     'reference://', fo='temp.json', target_protocol='file')
>>> zarr.open(mapper, mode='r')
Comments
  • Creating a virtual zarr datacube from COG s3 objects

    Creating a virtual zarr datacube from COG s3 objects

    @cgohlke, as mentioned at https://github.com/fsspec/kerchunk/issues/78#issuecomment-1088696226, I'd like to try reading a time stack of geotiffs via the Zarr library.

    As as test, I've got 2 COGs on S3-API-accessible object storage that I'd like to create a virtual Zarr dataset for using the tifffile library.

    I can get it going if the COGS are local, but I'm a little unsure how to handle the s3 objects.

    I'm trying to do:

    

from tifffile import TiffSequence
import zarr
import fsspec

fs = fsspec.filesystem('s3', anon=True, client_kwargs={'endpoint_url': 'https://mghp.osn.xsede.org'})

flist = fs.ls('/rsignellbucket1/lcmap/cog')

s3_flist = [f's3://{f}' for f in flist]

# try with s3 urls:
image_sequence = TiffSequence(s3_flist, pattern=r'.*_(\d+)_V12_LCPRI.tif')
with image_sequence.aszarr() as store:
    dz = zarr.open(store, mode='r')

# try with filelike objects:
file_like_object_list = [fs.open(f) for f in flist]
image_sequence = TiffSequence(file_like_object_list, pattern=r'.*_(\d+)_V12_LCPRI.tif')

    but neither one works. These buckets are public access (no egress!) so this code should be reproducible.

    Here's the Jupyter Notebook demonstrating the local success (except I'd like to use the native 2048x2048 chunking in the COG instead of the entire file!) and remote s3 failures.

    What is the best way to approach this?

    enhancement question 
    opened by rsignell-usgs 51
  • [Proposal]: Dump ReferenceFileSystem spec for ZarrTiffStore that can be read natively as zarr

    [Proposal]: Dump ReferenceFileSystem spec for ZarrTiffStore that can be read natively as zarr

    Thank you so much for your work on this project. I just came across experimental aszarr and ZarrTiffStore and am so excited! I'd written some one-off stores wrapping tifffile to read different pyramidal images as zarr (for napari), but having this in tifffile is incredible!

    I'm curious if you've seen the proposed JSON specification for describing a ReferenceFileSystem? Asking naively, and a bit selfishly, would it be possible to detect whether a ZarrTiffStore can be natively read by zarr and "export" one of these references?

    I work on web-based tools for visualizing OME-TIFF / Zarr data, and it would be really useful to quickly create these references.

    Here is an example viewing a multiscale tiff on the web using zarr.js, and this is the python script I wrote with the newest version of tifffile to generate the reference. I wonder if there is some way to generalize this script, but don't have the familiarly with the underlying formats to know if this is a silly idea.

    I notice that the TiffZarrStore handles all compression, so I know at least you need to detect whether the chunk compression is supported in Zarr.

    enhancement 
    opened by manzt 21
  • Fix fei_metadata for HELIOS FIB-SEM tiff images

    Fix fei_metadata for HELIOS FIB-SEM tiff images

    I have FIBSEM images where the metadata is not being parsed correctly by tifffile.

    Based on how the other metadata functions work (in particular, the sis_metadata function) I believe this line was intended to combine the two dictionary results: https://github.com/cgohlke/tifffile/blob/375d97f62df6482142b51f1b38a49bdd24d18a60/tifffile/tifffile.py#L5814

    However in practice, python will only ever return the values from the first dictionary (the FEI_SFEG metadata).

    When I open a HELIOS tiff image, I get an empty dictionary back instead of the real metadata.

    from tifffile import TiffFile
tif = TiffFile(filename)
tif.fei_metadata  # unexpectedly returns an empty dictionary, instead of the real metadata values

    This pull request modifies the code to combine the python dictionaries using the dictionary .update() method, using the same pattern as the existing tifffile function sis_metadata.

    enhancement 
    opened by GenevieveBuckley 11
  • Support Sony ARW lossless compression decoding

    Support Sony ARW lossless compression decoding

    See SubIFD0 of the Sony A1 sample here.

    This is very similar to the already supported DNG lossless JPEG (same compression "7"), with the small difference in tile arrangement. The ARW SubIFD0 is currently successfully decompressed (by lj92 codec I presume), but the image output is slightly jumbled because of difference in channel interleaving.

    In DNG, a 256x256 CFA tile is compressed as 128x256x2ch JPEG SOF3 tile, and I think line de-interleaved like this:

    1111 2222 1111 2222

    In ARW, this is 512x512 CFA tile to 256x256x4ch JPEG SOF3 tile which I guess should be de-interleaved like

    1212 3434 1212 3434

    (or potentially channels 2 and 3 swapped)

    opened by kmilos 11
  • Files produced in ScanImage 2020.1 cannot be parsed.

    Files produced in ScanImage 2020.1 cannot be parsed.

    We at Vidrio have received a ticket pertaining parsing scanimage tifs with tifffile. Just passing this on:

    In the newest scanimage, metadata from the images has changed and is no longer parseable by https://pypi.org/project/tifffile/

    My pipeline relies on this package. It seems the error is when the package tries to check if the metadata is a scanimage image in a not very robust way. I've fixed this for myself but this may affect other users.

    def is_scanimage(self):
    """Page contains ScanImage metadata."""
    return (
        self.description[:12] == 'state.config'
        or self.software[:22] == 'SI.LINE_FORMAT_VERSION'
        or 'scanimage.SI' in self.description[-256:]
    enhancement help wanted 
    opened by throwaway-scan 11
  • Opening corrupt Aperio SVS file causes Out-of-Memory error

    Opening corrupt Aperio SVS file causes Out-of-Memory error

    Hi @cgohlke

    I encountered a corrupted AperioSVS file that causes a machine with 64GB of ram to run out-of-memory when trying to instantiate the TiffFile instance. Creating the instance causes a dramatic increase in process memory usage in the TiffTag loading part of tifffile, which just causes the server to kill the python process. Ideally I would expect the code to just crash early, so that cleanup can happen in python.

    Sadly I can't share the file, so here's some additional information:

    The filesize is 5958078846 bytes and it's recognized as TIFF.CLASSIC_LE tifffile detects tagno=3126 tags, of which almost all raise an unknown tag data type error. Then for tag tagno=2327 it calls TiffTag.from_file with offset=3256275384 and header=b'\x01\x00\x02\x00\x00?\x00\xeb5\x05\x92\x1b'

    from_file

    Which due to the large count causes a ~4.000.000 character string to be unpacked later in this code path and due to some string ops causes a massive increase in Memory consumption.

    I was wondering if there are upper bounds on the amount of data a TiffTag can contain, and this codepath could error early.

    Cheers, Andreas 😃

    bug 
    opened by ap-- 10
  • NotImplementedError: lzw_encode

    NotImplementedError: lzw_encode

    Hi there, I'm trying to write a multi page tiff file, like this

    with tifffile.TiffWriter("nyname.tif", append=True) as P: P.write(**this_page_dict)

    where the dict is:

    {
'extratags': [(274, 3, 1, 1), (339, 3, 1, 1), (285, 2, 25, b'myTitle')], 
'bitspersample': 16,
'compression': 5, 
'photometric': 1, 
'rowsperstrip': 8, 
'planarconfig': 1, 
'resolution': (1677721600, 16777216), 
'data': array(..., dtype=uint16), 
'dtype': dtype('uint16')
}

    It's giving me a NotImplemented error, since I'm using compression=5 (LZW).

    Traceback (most recent call last): File "...filename.py", line 122, in P.write(**this_page_dict) File "...\venv\lib\site-packages\tifffile\tifffile.py", line 2342, in write strip = compress(strip) File "imagecodecs_imcd.pyx", line 856, in imagecodecs._imcd.lzw_encode NotImplementedError: lzw_encode

    Any pointer as of why and maybe how to "solve" / work around this?

    enhancement 
    opened by davideboschetto 10
  • imagej_metadata failed with ValueError

    imagej_metadata failed with ValueError

    When opening a TiffFile the method imagej_metadata raises this exception:

    imagej_metadata failed with ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

    Line: 15365 @ tifffile.py:

    if not bytecounts:

    Maybe replace with something like:

    if not bytecounts.size > 0:

    bug 
    opened by samhitech 9
  • How to read TIFF files from MicroManager with memory-mapping

    How to read TIFF files from MicroManager with memory-mapping

    Hi, first of all thanks for this great Python package.

    A question: I would like to seek and read within a set of large (4.3GB) OME-TIFF files, which were stored with MicroManager 2. Is it possible to memory-map and read these files with tifffile? I was only able to find the ability to write to memory-mapped files.

    Thanks and best regards, Michael

    question 
    opened by michaelmell 9
  • Support patching NDPI files

    Support patching NDPI files

    Hi Christoph,

    When trying to use the overwrite()-method on a tag in a ndpi-file this code raises an exception:

    if tiff.version == 42 and tiff.offsetsize == 8:
    # TODO: support patching NDPI > 4 GB files
    raise NotImplementedError('cannot patch NDPI > 4 GB files')

    Is this of concern even if the file is < 4 GB? Or if the size of the new value is not larger than the old? What is needed to implement patching ndpi files?

    Regards

    enhancement 
    opened by erikogabrielsson 8
  • Preserving singlet dimensions when loading ImageJ hyperstack tifs

    Preserving singlet dimensions when loading ImageJ hyperstack tifs

    Noticed a change in behavior between 2019.5.30 -> 2019.6.18 and wanted to make sure it was intended. Hopefully I'm just missing a new way of preserving dimensions of size 1 when loading ImageJ hyperstack tifs.

    # works on tifffile==2019.5.30
# assert fails on tifffile==2019.6.18
import numpy as np
import tifffile
a = np.zeros((10, 5, 1, 256, 256), dtype='uint16')
md = {'frames': 10, 'slices': 5, 'channels': 1}
tifffile.imsave('test.tif', a, imagej=True, metadata=md)
b = tifffile.imread('test.tif')
assert b.shape == a.shape, f'Loaded array has shape {b.shape}, not {a.shape}'

    In the new version, it appears that dimensions of size 1 are not included in the final stack shape, even if they are specified in the imagej metadata:

    https://github.com/cgohlke/tifffile/blob/6991c4ba1a1d1936cfad1bb249d1c74de08d449b/tifffile/tifffile.py#L2630-L2648

    In the older version, the channel dim would be included if it was specified in the metadata:

    https://github.com/cgohlke/tifffile/blob/ffbdf61b9c7cc627d17260d2cfd25c053001b0fc/tifffile/tifffile.py#L2526-L2529

    question 
    opened by nhthayer 8
  • add support to `memmap` to take in a list of tiff files with compatible shapes

    add support to `memmap` to take in a list of tiff files with compatible shapes

    It might be useful to add support to memmap to take in a list of tif files of compatible shapes and return a numpy.memmap with a shape representing concatenation along one of the axes. I could create a wrapper class for my own use cases but I thought I'd post here because this type of functionality might be useful to other users. I could help implement it (no timeline though) if you think it's worth having.

    enhancement 
    opened by kushalkolar 4
  • Numeric predictor settings are lost when writing

    Numeric predictor settings are lost when writing

    https://github.com/cgohlke/tifffile/blob/master/tifffile/tifffile.py#L2063 takes the provided predictor value and converts it to a bool, which is then converted back into a numeric type later based on the datatype ( https://github.com/cgohlke/tifffile/blob/master/tifffile/tifffile.py#L2119 )

    The problem with this is that multiple predictor options are available for each data type - for example, 34894=floatingpointx2 - these cannot be used currently when writing.

    I'm working on a proof of concept patch to fix this, I'll push what I have so far tonight for comments.

    bug 
    opened by Entropy512 3
Releases(v2022.10.10)

  • v2022.10.10(Oct 11, 2022)

    Tifffile is a Python library to

    1. store NumPy arrays in TIFF (Tagged Image File Format) files, and
    2. read image and metadata from TIFF-like files used in bioimaging.

    Image and metadata can be read from TIFF, BigTIFF, OME-TIFF, STK, LSM, SGI, NIHImage, ImageJ, MicroManager, FluoView, ScanImage, SEQ, GEL, SVS, SCN, SIS, BIF, ZIF (Zoomable Image File Format), QPTIFF (QPI), NDPI, and GeoTIFF files.

    Image data can be read as NumPy arrays or Zarr arrays/groups from strips, tiles, pages (IFDs), SubIFDs, higher order series, and pyramidal levels.

    Image data can be written to TIFF, BigTIFF, OME-TIFF, and ImageJ hyperstack compatible files in multi-page, volumetric, pyramidal, memory-mappable, tiled, predicted, or compressed form.

    Tifffile can also be used to inspect TIFF structures, read image data from multi-dimensional file sequences, write fsspec ReferenceFileSystem for TIFF files and image file sequences, patch TIFF tag values, and parse many proprietary metadata formats.

    Source code(tar.gz)
    Source code(zip)
    tifffile-2022.10.10-py3-none-any.whl(205.38 KB)
    tifffile-2022.10.10.tar.gz(317.55 KB)

  • v2022.8.12(Aug 12, 2022)

    Tifffile is a Python library to

    1. store NumPy arrays in TIFF (Tagged Image File Format) files, and
    2. read image and metadata from TIFF-like files used in bioimaging.

    Image and metadata can be read from TIFF, BigTIFF, OME-TIFF, STK, LSM, SGI, NIHImage, ImageJ, MicroManager, FluoView, ScanImage, SEQ, GEL, SVS, SCN, SIS, BIF, ZIF (Zoomable Image File Format), QPTIFF (QPI), NDPI, and GeoTIFF files.

    Image data can be read as NumPy arrays or Zarr arrays/groups from strips, tiles, pages (IFDs), SubIFDs, higher order series, and pyramidal levels.

    Image data can be written to TIFF, BigTIFF, OME-TIFF, and ImageJ hyperstack compatible files in multi-page, volumetric, pyramidal, memory-mappable, tiled, predicted, or compressed form.

    Tifffile can also be used to inspect TIFF structures, read image data from multi-dimensional file sequences, write fsspec ReferenceFileSystem for TIFF files and image file sequences, patch TIFF tag values, and parse many proprietary metadata formats.

    Source code(tar.gz)
    Source code(zip)
    tifffile-2022.8.12-py3-none-any.whl(203.65 KB)
    tifffile-2022.8.12.tar.gz(313.49 KB)

  • v2022.8.8(Aug 8, 2022)

    Tifffile is a Python library to

    1. store NumPy arrays in TIFF (Tagged Image File Format) files, and
    2. read image and metadata from TIFF-like files used in bioimaging.

    Image and metadata can be read from TIFF, BigTIFF, OME-TIFF, STK, LSM, SGI, NIHImage, ImageJ, MicroManager, FluoView, ScanImage, SEQ, GEL, SVS, SCN, SIS, BIF, ZIF (Zoomable Image File Format), QPTIFF (QPI), NDPI, and GeoTIFF files.

    Image data can be read as NumPy arrays or Zarr arrays/groups from strips, tiles, pages (IFDs), SubIFDs, higher order series, and pyramidal levels.

    Image data can be written to TIFF, BigTIFF, OME-TIFF, and ImageJ hyperstack compatible files in multi-page, volumetric, pyramidal, memory-mappable, tiled, predicted, or compressed form.

    Tifffile can also be used to inspect TIFF structures, read image data from multi-dimensional file sequences, write fsspec ReferenceFileSystem for TIFF files and image file sequences, patch TIFF tag values, and parse many proprietary metadata formats.

    Source code(tar.gz)
    Source code(zip)
    tifffile-2022.8.8-py3-none-any.whl(203.23 KB)
    tifffile-2022.8.8.tar.gz(312.32 KB)

  • v2022.7.31(Aug 1, 2022)

    Tifffile is a Python library to

    1. store NumPy arrays in TIFF (Tagged Image File Format) files, and
    2. read image and metadata from TIFF-like files used in bioimaging.

    Image and metadata can be read from TIFF, BigTIFF, OME-TIFF, STK, LSM, SGI, NIHImage, ImageJ, MicroManager, FluoView, ScanImage, SEQ, GEL, SVS, SCN, SIS, BIF, ZIF (Zoomable Image File Format), QPTIFF (QPI), NDPI, and GeoTIFF files.

    Image data can be read as NumPy arrays or Zarr arrays/groups from strips, tiles, pages (IFDs), SubIFDs, higher order series, and pyramidal levels.

    Image data can be written to TIFF, BigTIFF, OME-TIFF, and ImageJ hyperstack compatible files in multi-page, volumetric, pyramidal, memory-mappable, tiled, predicted, or compressed form.

    Tifffile can also be used to inspect TIFF structures, read image data from multi-dimensional file sequences, write fsspec ReferenceFileSystem for TIFF files and image file sequences, patch TIFF tag values, and parse many proprietary metadata formats.

    Source code(tar.gz)
    Source code(zip)
    tifffile-2022.7.31-py3-none-any.whl(202.55 KB)
    tifffile-2022.7.31.tar.gz(310.66 KB)

  • v2022.7.28(Jul 29, 2022)

    Tifffile is a Python library to

    1. store NumPy arrays in TIFF (Tagged Image File Format) files, and
    2. read image and metadata from TIFF-like files used in bioimaging.

    Image and metadata can be read from TIFF, BigTIFF, OME-TIFF, STK, LSM, SGI, NIHImage, ImageJ, MicroManager, FluoView, ScanImage, SEQ, GEL, SVS, SCN, SIS, BIF, ZIF (Zoomable Image File Format), QPTIFF (QPI), NDPI, and GeoTIFF files.

    Image data can be read as NumPy arrays or Zarr arrays/groups from strips, tiles, pages (IFDs), SubIFDs, higher order series, and pyramidal levels.

    Image data can be written to TIFF, BigTIFF, OME-TIFF, and ImageJ hyperstack compatible files in multi-page, volumetric, pyramidal, memory-mappable, tiled, predicted, or compressed form.

    Tifffile can also be used to inspect TIFF structures, read image data from multi-dimensional file sequences, write fsspec ReferenceFileSystem for TIFF files and image file sequences, patch TIFF tag values, and parse many proprietary metadata formats.

    Source code(tar.gz)
    Source code(zip)
    tifffile-2022.7.28-py3-none-any.whl(202.17 KB)
    tifffile-2022.7.28.tar.gz(310.06 KB)

  • v2022.5.4(Jul 4, 2022)

Owner
Christoph Gohlke
Christoph Gohlke
GitHub Repository
Singer is an open source standard for moving data between databases, web APIs, files, queues, and just about anything else you can think of.

Singer is an open source standard for moving data between databases, web APIs, files, queues, and just about anything else you can think of. Th

Singer 1.1k Jan 05, 2023
CSV-Handler written in Python3

CSVHandler This code allows you to work intelligently with CSV files. A file in CSV syntax is converted into several lists, which are combined in a to

Max Tischberger 1 Jan 13, 2022
Dragon Age: Origins toolset to extract/build .erf files, patch language-specific .dlg files, and view the contents of files in the ERF or GFF format

DAOTools This is a set of tools for Dragon Age: Origins modding. It can patch the text lines of .dlg files, extract and build an .erf file, and view t

8 Dec 06, 2022
FUSE filesystem Python scripts for Nintendo console files

ninfs (formerly fuse-3ds) is a FUSE program to extract data from Nintendo game consoles. It works by presenting a virtual filesystem with the contents of your games, NAND, or SD card contents, and yo

Ian Burgwin 343 Jan 02, 2023
Kartothek - a Python library to manage large amounts of tabular data in a blob store

Kartothek - a Python library to manage (create, read, update, delete) large amounts of tabular data in a blob store

15 Dec 25, 2022
MetaMove is written in Python3 and aims at easing batch renaming operations based on file meta data.

MetaMove MetaMove is written in Python3 and aims at easing batch renaming operations based on file meta data. MetaMove abuses eval combined with f-str

Jan Philippi 2 Dec 28, 2021
Creates folders into a directory to categorize files in that directory by file extensions and move all things from sub-directories to current directory.

Categorize and Uncategorize Your Folders Table of Content TL;DR just take me to how to install. What are Extension Categorizer and Folder Dumper Insta

Furkan Baytekin 1 Oct 17, 2021
A tiny Python library for writing multi-channel TIFF stacks.

xtiff A tiny Python library for writing multi-channel TIFF stacks. The aim of this library is to provide an easy way to write multi-channel image stac

23 Dec 27, 2022
A python script to convert an ucompressed Gnucash XML file to a text file for Ledger and hledger.

README 1 gnucash2ledger gnucash2ledger is a Python script based on the Github Gist by nonducor (nonducor/gcash2ledger.py). This Python script will tak

Thomas Freeman 0 Jan 28, 2022
Transforme rapidamente seu arquivo CSV (de qualquer tamanho) para SQL de forma rápida.

Transformador de CSV para SQL Transforme rapidamente seu arquivo CSV (de qualquer tamanho) para SQL de forma rápida, e com isso insira seus dados usan

William Rodrigues 4 Oct 17, 2022
Python's Filesystem abstraction layer

PyFilesystem2 Python's Filesystem abstraction layer. Documentation Wiki API Documentation GitHub Repository Blog Introduction Think of PyFilesystem's

pyFilesystem 1.8k Jan 02, 2023
A JupyterLab extension that allows opening files and directories with external desktop applications.

A JupyterLab extension that allows opening files and directories with external desktop applications.

martinRenou 0 Oct 14, 2021
File-manager - A basic file manager, written in Python

File Manager A basic file manager, written in Python. Installation Install Pytho

Samuel Ko 1 Feb 05, 2022
A python script generate password files in plain text

KeePass (or any desktop pw manager?) Helper WARNING: This script will generate password files in plain text. ITS NOT SECURE. I needed help remembering

Eric Thomas 1 Nov 21, 2021
Python's Filesystem abstraction layer

PyFilesystem2 Python's Filesystem abstraction layer. Documentation Wiki API Documentation GitHub Repository Blog Introduction Think of PyFilesystem's

pyFilesystem 1.8k Jan 02, 2023
Uncompress DEFLATE streams in pure Python

stream-inflate Uncompress DEFLATE streams in pure Python. Installation pip install stream-inflate Usage from stream_inflate import stream_inflate impo

Michal Charemza 7 Oct 13, 2022
Test app for importing contact information in CSV files.

Contact Import TestApp Test app for importing contact information in CSV files. Explore the docs » · Report Bug · Request Feature Table of Contents Ab

1 Feb 06, 2022
A simple bulk file renamer, written in python.

Python File Editor A simple bulk file renamer, written in python. There are two functions, the bulk rename and the bulk file extention change. Bulk Fi

Sam Bloomfield 2 Dec 22, 2021
Pti-file-format - Reverse engineering the Polyend Tracker instrument file format

pti-file-format Reverse engineering the Polyend Tracker instrument file format.

Jaap Roes 14 Dec 30, 2022
Annotate your Python requirements.txt file with summaries of each package.

Summarize Requirements 🐍 📜 Annotate your Python requirements.txt file with a short summary of each package. This tool: takes a Python requirements.t

Zeke Sikelianos 8 Apr 22, 2022