gala.imio: Image IO

gala.imio.apply_segmentation_map(superpixels, sp_to_body_map)

Return a segmentation from superpixels and a superpixel to body map.

Parameters:

superpixels : numpy ndarray, arbitrary shape, int type

A superpixel (or supervoxel) map (aka label field).

sp_to_body_map : numpy ndarray, shape (NUM_SUPERPIXELS, 2), int type

An array of (superpixel, body) map pairs.

Returns:

segmentation : numpy ndarray, same shape as ‘superpixels’, int type

The segmentation induced by the superpixels and map.

gala.imio.compute_sp_to_body_map(sps, bodies)

Return unique (sp, body) pairs from a superpixel map and segmentation.

Parameters:

sps : numpy ndarray, arbitrary shape

The superpixel (supervoxel) map.

bodies : numpy ndarray, same shape as sps

The corresponding segmentation.

Returns:

sp_to_body : numpy ndarray, shape (NUM_SPS, 2)

Notes

No checks are made for sane inputs. This means that incorrect input, such as non-matching shapes, or superpixels mapping to more than one segment, will result in undefined behavior downstream with no warning.

gala.imio.extract_segments(seg, ids)

Get a uint8 volume containing only the specified segment ids.

Parameters:

seg : array of int

The input segmentation.

ids : list of int, maximum length 255

A list of segments to extract from seg.

Returns:

segs : array of uint8

A volume with 1, 2, …, len(ids) labels where the required segments were, and 0 elsewhere.

Notes

This function is designed to output volumes to VTK format for viewing in ITK-SNAP

Examples

>>> segments = array([[45, 45, 51, 51],
...                   [45, 83, 83, 51]])
>>> extract_segments(segments, [83, 45])
array([[2, 2, 0, 0],
       [2, 1, 1, 0]], dtype=uint8)

gala.imio.pil_to_numpy(img)

Convert an Image object to a numpy array.

Parameters:

img : Image object (from the Python Imaging Library)

Returns:

ar : numpy ndarray

The corresponding numpy array (same shape as the image)

gala.imio.raveler_body_annotations(orphans, non_traversing=None)

Return a Raveler body annotation dictionary of orphan segments.

Orphans are labeled as body annotations with not sure status and a string indicating orphan in the comments field.

Non-traversing segments have only one contact with the surface of the volume, and are labeled does not traverse in the comments.

Parameters:

orphans : iterable of int

The ID numbers corresponding to orphan segments.

non_traversing : iterable of int (optional, default None)

The ID numbers of segments having only one exit point in the volume.

Returns:

body_annotations : dict

A dictionary containing entries for ‘data’ and ‘metadata’ as specified in the Raveler body annotations format [1, 2].

References

[1] https://wiki.janelia.org/wiki/display/flyem/body+annotation+file+format and: [2] https://wiki.janelia.org/wiki/display/flyem/generic+file+format

gala.imio.raveler_output_shortcut(svs, seg, gray, outdir, sps_out=None)

Compute the Raveler format and write to directory, all at once.

Parameters:

svs : np.ndarray, int, shape (M, N, P)

The supervoxel map.

seg : np.ndarray, int, shape (M, N, P)

The segmentation map. It is assumed that no supervoxel crosses any segment boundary.

gray : np.ndarray, uint8, shape (M, N, P)

The grayscale EM images corresponding to the above segmentations.

outdir : string

The export directory for the Raveler volume.

sps_out : np.ndarray, int, shape (M, N, P) (optional)

The precomputed serial section 2D superpixel map. Output will be much faster if this is provided.

Returns:

sps_out : np.ndarray, int, shape (M, N, P)

The computed serial section 2D superpixel map. Keep this when making multiple calls to raveler_output_shortcut with the same supervoxel map.

gala.imio.raveler_rgba_to_int(im, ignore_alpha=True)

Convert a volume using Raveler’s RGBA encoding to int. [1]

Parameters:

im : np.ndarray, shape (M, N, P, 4)

The image stack to be converted.

ignore_alpha : bool, optional

By default, the alpha channel does not encode anything. However, if we ever need 32 bits, it would be used. This function supports that with ignore_alpha=False. (default is True.)

Returns:

im_int : np.ndarray, shape (M, N, P)

The label volume.

References

[1] https://wiki.janelia.org/wiki/display/flyem/Proofreading+data+and+formats

gala.imio.raveler_serial_section_map(nd_map, min_size=0, do_conn_comp=False, globally_unique_ids=True)

Produce serial_section_map and label one corner of each plane as 0.

Raveler chokes when there are no pixels with label 0 on a plane, so this function produces the serial section map as normal but then adds a 0 to the [0, 0] corner of each plane, IF the volume doesn’t already have 0 pixels.

Notes

See serial_section_map for more info.

gala.imio.raveler_to_labeled_volume(rav_export_dir, get_glia=False, use_watershed=False, probability_map=None, crop=None)

Import a raveler export stack into a labeled segmented volume.

Parameters:

rav_export_dir : string

The directory containing the Raveler stack.

get_glia : bool (optional, default False)

Return the segment numbers corresponding to glia, if available.

use_watershed : bool (optional, default False)

Fill in 0-labeled voxels using watershed.

probability_map : np.ndarray, same shape as volume to be read (optional)

If use_watershed is True, use probability_map as the landscape. If this is not provided, it uses a flat landscape.

crop : tuple of int (optional, default None)

A 6-tuple of [xmin, xmax, ymin, ymax, zmin, zmax].

Returns:

output_volume : np.ndarray, shape (Z, X, Y)

The segmentation in the Raveler volume.

glia : list of int (optional, only returned if get_glia is True)

The IDs in the segmentation corresponding to glial cells.

gala.imio.read_cremi(fn, datasets=['/volumes/raw', '/volumes/labels/neuron_ids'])

Read volume formatted as described in CREMI data challenge [R11].

The format is HDF5, with:

• raw image data (uint8) in: /volumes/raw
• (optional) membrane prediction data (uint8, inverted) in: /volumes/membrane
• synaptic cleft annotations in: /volumes/labels/clefts
• neuron ids (uint64) in: /volumes/labels/neuron_ids
• (optional) fragment data (uint64) in: /volumes/labels/fragments

We currently ignore the synaptic cleft annotations, and return only the raw image and the neuron ids.

Parameters:

fn : string

The input filename.

Returns:

datasets : list of array

The arrays corresponding to the requested datasets.

References

gala.imio.read_h5_stack(fn, group='stack', crop=[None, None, None, None, None, None], **kwargs)

Read a volume in HDF5 format into numpy.ndarray.

Parameters:

fn : string

The filename of the input HDF5 file.

group : string, optional (default ‘stack’)

The group within the HDF5 file containing the dataset.

crop : list of int, optional (default ‘[None]*6’, no crop)

A crop to get of the volume of interest. Only available for 2D and 3D volumes.

Returns:

stack : numpy ndarray

The stack contained in fn, possibly cropped.

gala.imio.read_image_stack(fn, *args, **kwargs)

Read a 3D volume of images in image or .h5 format into a numpy.ndarray.

This function attempts to automatically determine input file types and wraps specific image-reading functions.

Parameters:

fn : filename (string)

A file path or glob pattern specifying one or more valid image files. The file format is automatically determined from this argument.

*args : filenames (string, optional)

More than one positional argument will be interpreted as a list of filenames pointing to all the 2D images in the stack.

**kwargs : keyword arguments (optional)

Arguments to be passed to the underlying functions. A ‘crop’ keyword argument is supported, as a list of length 6: [xmin, xmax, ymin, ymax, zmin, zmax]. Use ‘None’ for no crop in that coordinate.

Returns:

stack : 3-dimensional numpy ndarray

Notes

If reading in .h5 format, keyword arguments are passed through to read_h5_stack().

Automatic file type detection may be deprecated in the future.

gala.imio.read_mapped_segmentation(fn, sp_group='stack', sp_to_body_group='transforms')

Read a volume in mapped HDF5 format into a numpy.ndarray pair.

Parameters:

fn : string

The filename to open.

sp_group : string, optional (default ‘stack’)

The group within the HDF5 file where the superpixel map is stored.

sp_to_body_group : string, optional (default ‘transforms’)

The group within the HDF5 file where the superpixel to body map is stored.

Returns:

segmentation : numpy ndarray, same shape as ‘superpixels’, int type

The segmentation induced by the superpixels and map.

gala.imio.read_mapped_segmentation_raw(fn, sp_group='stack', sp_to_body_group='transforms')

Read a volume in mapped HDF5 format into a numpy.ndarray pair.

Parameters:

fn : string

The filename to open.

sp_group : string, optional (default ‘stack’)

The group within the HDF5 file where the superpixel map is stored.

sp_to_body_group : string, optional (default ‘transforms’)

The group within the HDF5 file where the superpixel to body map is stored.

Returns:

sp_map : numpy ndarray, arbitrary shape

The superpixel (or supervoxel) map.

sp_to_body_map : numpy ndarray, shape (NUM_SUPERPIXELS, 2)

The superpixel to body (segment) map, as (superpixel, body) pairs.

gala.imio.read_multi_page_tif(fn, crop=[None, None, None, None, None, None])

Read a multi-page tif file into a numpy array.

Parameters:

fn : string

The filename of the image file being read.

Returns:

ar : numpy ndarray

The image stack in array format.

Notes

Currently, only grayscale images are supported.

gala.imio.read_prediction_from_ilastik_batch(fn, **kwargs)

Read the prediction produced by Ilastik from batch processing.

Parameters:

fn : string

The filename to read from.

group : string (optional, default ‘/volume/prediction’)

Where to read from in the HDF5 file hierarchy.

single_channel : bool (optional, default True)

Read only the 0th channel (final dimension) from the volume.

Returns:

None

gala.imio.read_vtk(fin)

Read a numpy volume from a VTK structured points file.

Code adapted from Erik Vidholm’s readVTK.m Matlab implementation.

Parameters:

fin : string

The input filename.

Returns:

ar : numpy ndarray

The array contained in the file.

gala.imio.segs_to_raveler(sps, bodies, min_size=0, do_conn_comp=False, sps_out=None)

Return a Raveler tuple from 3D superpixel and body maps.

Parameters:

sps : numpy ndarray, shape (M, N, P)

The supervoxel map.

bodies : numpy ndarray, shape (M, N, P)

The body map. Superpixels should not map to more than one body.

min_size : int, optional (default: 0)

Superpixels smaller than this size on a particular plane are blacked out.

do_conn_comp : bool (default: False)

Whether to do a connected components operation on each plane. This is required if we want superpixels to be contiguous on each plane, since 3D-contiguous superpixels are not guaranteed to be contiguous along a slice.

sps_out : numpy ndarray, shape (M, N, P), optional (default: None)

A Raveler-compatible superpixel map, meaning that superpixels are unique to each plane along axis 0. (See superpixels in the return values.) If provided, this saves significant computation time.

Returns:

superpixels : numpy ndarray, shape (M, N, P)

The superpixel map. Non-zero superpixels are unique to each plane. That is, np.unique(superpixels[i]) and np.unique(superpixels[j]) have only 0 as their intersection.

sp_to_segment : numpy ndarray, shape (Q, 3)

The superpixel to segment map. Segments are unique to each plane. The first number on each line is the plane number.

segment_to_body : numpy ndarray, shape (R, 2)

The segment to body map.

gala.imio.serial_section_map(nd_map, min_size=0, do_conn_comp=False, globally_unique_ids=True)

Produce a plane-by-plane superpixel map with unique IDs.

Raveler requires sps to be unique and different on each plane. This function converts a fully 3D superpixel map to a serial-2D superpixel map compatible with Raveler.

Parameters:

nd_map : np.ndarray, int, shape (M, N, P)

The original superpixel map.

min_size : int (optional, default 0)

Remove superpixels smaller than this size (on each plane)

do_conn_comp : bool (optional, default False)

In some cases, a single supervoxel may result in two disconnected superpixels in 2D. Set to True to force these to have different IDs.

globally_unique_ids : bool (optional, default True)

If True, every plane has unique IDs, with plane n having IDs {i1, i2, …, in} and plane n+1 having IDs {in+1, in+2, …, in+ip}, and so on.

Returns:

relabeled_planes : np.ndarray, int, shape (M, N, P)

A volume equal to nd_map but with superpixels relabeled along axis 0. That is, the input volume is reinterpreted as M slices of shape (N, P).

gala.imio.ucm_to_raveler(ucm, sp_threshold=0.0, body_threshold=0.1, **kwargs)

Return Raveler map from a UCM.

Parameters:

ucm : numpy ndarray, shape (M, N, P)

An ultrametric contour map. This is a map of scored segment boundaries such that if A, B, and C are segments, then score(A, B) = score(B, C) >= score(A, C), for some permutation of A, B, and C. A hierarchical agglomeration process produces a UCM.

sp_threshold : float, optional (default: 0.0)

The value for which to threshold the UCM to obtain the superpixels.

body_threshold : float, optional (default: 0.1)

The value for which to threshold the UCM to obtain the segments/bodies. The condition body_threshold >= sp_threshold should hold in order to obtain sensible results.

**kwargs : dict, optional

Keyword arguments to be passed through to segs_to_raveler.

Returns:

superpixels : numpy ndarray, shape (M, N, P)

The superpixel map. Non-zero superpixels are unique to each plane. That is, np.unique(superpixels[i]) and np.unique(superpixels[j]) have only 0 as their intersection.

sp_to_segment : numpy ndarray, shape (Q, 3)

The superpixel to segment map. Segments are unique to each plane. The first number on each line is the plane number.

segment_to_body : numpy ndarray, shape (R, 2)

The segment to body map.

gala.imio.write_cremi(data_dict, fn, resolution=(40.0, 4.0, 4.0))

Write a volume formatted as described in CREMI data challenge [1]_.

Parameters:

data_dict : dictionary of string to arrays

The data dictionary mapping HDF groups to arrays.

fn : string

The filename to write to.

resolution : tuple of float, optional

The resolution along each axis of the datasets. Currently, this is the same for each dataset written.

gala.imio.write_h5_stack(npy_vol, fn, group='stack', compression=None, chunks=None, shuffle=None, attrs=None)

Write a numpy.ndarray 3D volume to an HDF5 file.

Parameters:

npy_vol : numpy ndarray

The array to be saved to HDF5.

fn : string

The output filename.

group : string, optional (default: ‘stack’)

The group within the HDF5 file to write to.

compression : {None, ‘gzip’, ‘szip’, ‘lzf’}, optional (default: None)

The compression to use, if any. Note that ‘lzf’ is only available through h5py, so implementations in other languages will not be able to read files created with this compression.

chunks : tuple, True, or None (default: None)

Whether to use chunking in the HDF5 dataset. Default is None. True lets h5py choose a chunk size automatically. Otherwise, use a tuple of int of the same length as npy_vol.ndim. From the h5py documentation: “In the real world, chunks of size 10kB - 300kB work best, especially for compression. Very small chunks lead to lots of overhead in the file, while very large chunks can result in inefficient I/O.”

shuffle : bool, optional

Shuffle the bytes on disk to improve compression efficiency.

attrs : dict, optional

A dictionary, keyed by string, of attributes to append to the dataset.

gala.imio.write_ilastik_batch_volume(im, fn)

Write a volume to an HDF5 file for Ilastik batch processing.

Parameters:

im : np.ndarray, shape (M, N[, P])

The image volume to be saved.

fn : string

The filename in which to save the volume.

Returns:

None

gala.imio.write_ilastik_project(images, labels, fn, label_names=None)

Write one or more image volumes and corresponding labels to Ilastik.

Parameters:

images : np.ndarray or list of np.ndarray, shapes (M_i, N_i[, P_i])

The grayscale images to be saved.

labels : np.ndarray or list of np.ndarray, same shapes as images

The label maps corresponding to the images.

fn : string

The filename to save the project in.

label_names : list of string (optional)

The names corresponding to each label in labels. (Not implemented!)

Returns:

None

Notes

Limitations:

Assumes the same labels are used for all images. Supports only grayscale images and volumes, and a maximum of 8 labels. Requires at least one unlabeled voxel in the label field.

gala.imio.write_image_stack(npy_vol, fn, **kwargs)

Write a numpy.ndarray 3D volume to a stack of images or an HDF5 file.

Parameters:

npy_vol : numpy ndarray

The volume to be written to disk.

fn : string

The filename to be written, or a format string when writing a 3D stack to a 2D format (e.g. a png image stack).

**kwargs : keyword arguments

Keyword arguments to be passed to wrapped functions. See corresponding docs for valid arguments.

Returns:

out : None

Examples

>>> import numpy as np
>>> from gala.imio import write_image_stack
>>> im = 255 * np.array([
... [[0, 1, 0], [1, 0, 1], [0, 1, 0]],
... [[1, 0, 1], [0, 1, 0], [1, 0, 1]]], dtype=uint8)
>>> im.shape
(2, 3, 3)
>>> write_image_stack(im, 'image-example-%02i.png', axis=0)
>>> import os
>>> fns = sorted(filter(lambda x: x.endswith('.png'), os.listdir('.')))
>>> fns # two 3x3 images
['image-example-00.png', 'image-example-01.png']
>>> os.remove(fns[0]); os.remove(fns[1]) # doctest cleanup

gala.imio.write_json(annot, fn='annotations-body.json', directory=None)

Write an annotation dictionary in Raveler format to a JSON file.

The annotation file format is described in: https://wiki.janelia.org/wiki/display/flyem/body+annotation+file+format and: https://wiki.janelia.org/wiki/display/flyem/generic+file+format

Parameters:

annot : dict

A body annotations dictionary (described in pages above).

fn : string (optional, default ‘annotations-body.json’)

The filename to which to write the file.

directory : string (optional, default None, or ‘.’)

A directory in which to write the file.

Returns:

None

gala.imio.write_mapped_segmentation(superpixel_map, sp_to_body_map, fn, sp_group='stack', sp_to_body_group='transforms')

Write a mapped segmentation to an HDF5 file.

Parameters:

superpixel_map : numpy ndarray, arbitrary shape

sp_to_body_map : numpy ndarray, shape (NUM_SPS, 2)

A many-to-one map of superpixels to bodies (segments), specified as rows of (superpixel, body) pairs.

fn : string

The output filename.

sp_group : string, optional (default ‘stack’)

the group within the HDF5 file to store the superpixel map.

sp_to_body_group : string, optional (default ‘transforms’)

the group within the HDF5 file to store the superpixel to body map.

Returns:

None

gala.imio.write_png_image_stack(npy_vol, fn, axis=-1, bitdepth=None)

Write a numpy.ndarray 3D volume to a stack of .png images.

Parameters:

npy_vol : numpy ndarray, shape (M, N, P)

The volume to be written to disk.

fn : format string

The file pattern to which to write the volume.

axis : int, optional (default = -1)

The axis along which output the images. If the input array has shape (M, N, P), and axis is 1, the function will write N images of shape (M, P) to disk. In keeping with Python convention, -1 specifies the last axis.

Returns:

None : None

No value is returned.

Notes

Only 8-bit and 16-bit single-channel images are currently supported.

gala.imio.write_to_raveler(sps, sp_to_segment, segment_to_body, directory, gray=None, raveler_dir='/usr/local/raveler-hdf', nproc_contours=16, body_annot=None)

Output a segmentation to Raveler format.

Parameters:

sps : np.ndarray, int, shape (nplanes, nx, ny)

The superpixel map. Superpixels can only occur on one plane.

sp_to_segment : np.ndarray, int, shape (nsps + nplanes, 3)

Superpixel-to-segment map as a 3 column list of (plane number, superpixel id, segment id). Segments must be unique to a plane, and each plane must contain the map {0: 0}

segment_to_body: np.ndarray, int, shape (nsegments, 2)

The segment to body map.

directory: string

The directory in which to write the stack. This directory and all necessary subdirectories will be created.

gray: np.ndarray, uint8 or uint16, shape (nplanes, nx, ny) (optional)

The grayscale images corresponding to the superpixel maps.

raveler dir: string (optional, default `/usr/local/raveler-hdf`)

Where Raveler is installed.

nproc_contours: int (optional, default 16)

How many processes to use when generating the Raveler contours.

body_annot: dict or np.ndarray (optional)

Either a dictionary to write to JSON in Raveler body annotation format, or a numpy ndarray of the segmentation from which to compute orphans and non traversing bodies (which then get written out as body annotations).

Returns:

None

Notes

Raveler is the EM segmentation proofreading tool developed in-house at Janelia for the FlyEM project.

gala.imio.write_vtk(ar, fn, spacing=[1.0, 1.0, 1.0])

Write 3D volume to VTK structured points format file.

Code adapted from Erik Vidholm’s writeVTK.m Matlab implementation.

Parameters:

ar : a numpy array, shape (M, N, P)

The array to be written to disk.

fn : string

The desired output filename.

spacing : iterable of float, optional (default: [1.0, 1.0, 1.0])

The voxel spacing in x, y, and z.

Returns:

None : None

This function does not have a return value.