Configure array chunking and compression

WarningContent review status

This page contains unreviewed content created with large language model (LLM) tools.

A layout profile selects a chunk shape, shard shape, and compression codec for each stored array according to its data type and axis labels. tensorbids defines the profile and registry interfaces; bitsio provides four profiles for its Zarr backends.

1 List the registered profiles

from tensorbids.resolver.layout_profiles import layout_profiles

for name, profile in sorted(layout_profiles().items()):
    print(name, [rule.name for rule in profile.rules], "base:", profile.base)
bitsio.archival ['int16-3d'] base: bitsio.unsharded
bitsio.default ['int16-3d', 'float32-3d', 'int8-3d', 'int16-time', 'int16-direction', 'other'] base: None
bitsio.fast ['int16-3d', 'float32-3d', 'int8-3d', 'int16-time', 'int16-direction'] base: bitsio.default
bitsio.unsharded ['int16-3d', 'float32-3d', 'int8-3d', 'int16-time', 'int16-direction', 'other'] base: None
Profile Picks Use it for
bitsio.default Approximately 64-voxel chunks with Zstandard-based compression; usually one shard per array General-purpose reads with a low object count
bitsio.unsharded Default chunking with one object per chunk Storage systems that handle many small objects efficiently
bitsio.archival Whole-volume int16 chunks compressed with zstd@19; otherwise inherits bitsio.unsharded Infrequently read data where compression ratio is the priority
bitsio.fast Lower compression levels with sharding Ingestion and write throughput

2 Create a dataset with one

ds, report = BITSZarrDataset.auto_create(source, "dest.bits", profile="bitsio.archival")

write_source() accepts the same profile argument.

3 Preview a layout before writing

from tensorbids.resolver.layout_profiles import resolve_layout

t1w = resolve_layout((256, 156, 256), "float32", ("i", "j", "k"), "bitsio.default")
print(t1w.rule, t1w.chunk_shape, t1w.codec, t1w.shard_shape((256, 156, 256), t1w.chunk_shape))
bold = resolve_layout((184, 64, 64, 30), "int16", ("time", "i", "j", "k"), "bitsio.archival")
print(bold.profile, bold.rule, bold.chunk_shape, bold.codec)
float32-3d (64, 64, 64) blosc-zstd@9:byteshuffle (256, 192, 256)
bitsio.unsharded int16-time (16, 32, 32, 30) blosc-zstd@9:byteshuffle

If no rule matches, resolution continues with the profile’s base profile. bitsio.archival defines a rule for three-dimensional int16 arrays. The four-dimensional BOLD array therefore falls back to bitsio.unsharded.

4 Fork a profile

from tensorbids.resolver.layout_profiles import LayoutRule

lab = layout_profiles()["bitsio.default"].fork(
    "lab.bold-frames",
    LayoutRule(
        "int16-time",
        "zstd@9",
        dtypes=frozenset({"int16"}),
        axis_labels=("time", "i", "j", "k"),
        grid=(1, 64, 64, 30),
    ),
)
frames = resolve_layout((184, 64, 64, 30), "int16", ("time", "i", "j", "k"), lab)
print(frames.profile, frames.rule, frames.chunk_shape, frames.codec)
lab.bold-frames int16-time (1, 64, 64, 30) zstd@9

fork() copies the original rules and base profile. A rule with an existing name replaces that rule; a newly named rule is evaluated before the copied rules.

5 Register it

In a process:

from tensorbids.resolver.layout_profiles import register_layout_profile

register_layout_profile(lab)
register_layout_profile(lab)
print("lab.bold-frames" in layout_profiles())
True

Registering an equivalent profile a second time is idempotent. Registering a different profile under the same name raises ValueError. Libraries should publish profiles through the tensorbids.layout_profiles entry-point group:

[project.entry-points."tensorbids.layout_profiles"]
"lab.bold-frames" = "lab.layouts:BOLD_FRAMES"

6 Override one write

chunk_shape= and codec= on auto_create or write_source take precedence over the profile for every array of that call:

ds, report = BITSZarrDataset.auto_create(source, "dest.bits", codec="zstd@19")

7 Constraints

  • Rules match data types and axis-label sequences, not BIDS suffixes. For example, a T1-weighted image and a three-dimensional mask with the same data type match the same rule.
  • Explicit chunk_shape and codec arguments override the profile but are applied to every image in that ingestion call.

8 See also