Storage backends and array layout

Content review status

This page contains unreviewed content created with large language model (LLM) tools.

bitsio stores numerical arrays as Zarr v3, auxiliary payloads in content-addressed storage, and relations and indexes as sharded Parquet. Layout profiles select chunking, sharding, and compression according to array data type and axis labels. Tensor content hashes are independent of those physical layout choices.

1 Implemented storage components

A tensor store maps an opaque key to an n-dimensional array. bitsio provides Zarr v3 implementations backed by TensorStore or zarr-python. Both backends read arrays written by the other when they support the selected codec. Storage roots may use a local directory, memory, S3, Google Cloud Storage, or Azure; the supported combinations depend on the selected backend and transport.

An auxiliary store maps a content hash to a smaller payload such as an affine matrix, image header, or resolved JSON sidecar. Equal canonical payloads share one key. Array-valued payloads use a fixed little-endian .npy representation for each payload kind; floating-point zero and NaN values are canonicalized before hashing. The store can use hash-partitioned objects or packfiles.

A table engine stores BIDS relations, user-defined relations, and the BIDS and provenance indexes as sharded Parquet.

A navigator is a typed operation graph defined by tensorbids. It references tensor and auxiliary keys and describes reconstruction steps such as reading, exact casting, affine dequantization, spatial selection, and file materialization. bitsio stores navigators through the tensorbids dataset filesystem and evaluates them with its reference resolver.

2 Tensor content hashes

A tensor content hash covers its data type string, including byte order, its shape, and its C-order values. It does not cover chunk boundaries, compressed bytes, or backend metadata:

import numpy as np
from bitsio import BITSZarrDataset, canonical_tensor_hash

ds = BITSZarrDataset.open("ds000114.bits", read_only=False)
values = np.arange(4 * 5 * 6, dtype="float32").reshape(4, 5, 6)
a = ds.write_array("sub-01/anat/sub-01_desc-a_T1w.nii", values, codec="zstd@19")
b = ds.write_array("sub-01/anat/sub-01_desc-b_T1w.nii", values, chunk_shape=(2, 5, 6), codec="blosc-zstd@5:byteshuffle")
print(ds.tensor.info(a).codec, "|", ds.tensor.info(b).codec, ds.tensor.info(b).chunks)
print(ds.tensor_hash(a) == ds.tensor_hash(b) == canonical_tensor_hash(values))
print(canonical_tensor_hash(values.astype(">f4")) == canonical_tensor_hash(values))

zstd@19 | blosc-zstd@5:byteshuffle (2, 5, 6)
True
False

The first result shows that the same values have the same hash under different codecs and chunk shapes. The second shows that byte order is part of tensor identity. A storage-only change can therefore be recorded as a relayout event without claiming that the tensor values changed. Store keys are opaque; applications should not parse their current textual representation.

3 Layout

An array plan fixes an array’s shape, data type, chunk shape, shard shape, and codec. A layout profile is an ordered set of rules that matches data types and axis-label sequences. The first matching rule supplies the physical layout. If no local rule matches, selection continues in the base profile.

Profile	Base	Rules
`bitsio.default`	none	Approximately 64-voxel chunks; Zstandard-based compression; generally one shard per array
`bitsio.unsharded`	none	Default chunk geometry with one object per chunk
`bitsio.archival`	`bitsio.unsharded`	Whole-volume `int16` chunks compressed with `zstd@19`
`bitsio.fast`	`bitsio.default`	Lower compression levels with sharding

A rule does not inspect BIDS suffixes. Arrays with the same data type and axis labels therefore select the same rule even when their BIDS meanings differ. See Configure array chunking and compression for profile customization.

4 Read amplification

Read amplification is the ratio between data read or decoded and data requested by the application. The G1 conformance target combines low amplification with a small number of range requests. For a window of side \(W\), placed uniformly along an axis of length \(n\) and chunked with extent \(C\), the number of chunks touched is

\[ k(o) = \left\lfloor \frac{o + W - 1}{C} \right\rfloor - \left\lfloor \frac{o}{C} \right\rfloor + 1, \qquad o \in \{0, \dots, n - W\}, \]

For cubic chunks and windows, the expected decoded amplification is \(\prod_{\text{axes}} \mathbb{E}[k] \cdot C^3 / W^3\). The following calculation uses a \(64^3\) window in a (256, 156, 256) float32 image:

import numpy as np

W, shape = 64, (256, 156, 256)


def chunks_touched(n, C):
    o = np.arange(n - W + 1)
    return (o + W - 1) // C - o // C + 1


print("chunk  chunks/window  decoded/needed")
for C in (16, 32, 64, 128):
    k = np.prod([chunks_touched(n, C).mean() for n in shape])
    print(f"{C:>5}  {k:>13.2f}  {k * C**3 / W**3:>13.2f}x")

chunk  chunks/window  decoded/needed
   16         120.09           1.88x
   32          26.07           3.26x
   64           7.75           7.75x
  128           2.29          18.31x

The default profile stores these images in \(64^3\) chunks within one shard. The next cell reads the shard indexes for the nine tutorial images with this shape and estimates compressed bytes fetched for random windows:

import json
from pathlib import Path

fetched, stored = [], []
for meta in sorted(Path("ds000114.bits/bits/tensors/arrays").glob("*/zarr.json")):
    m = json.loads(meta.read_text())
    if m["data_type"] != "float32" or tuple(m["shape"]) != shape:
        continue
    (shard,) = [p for p in meta.parent.glob("c/**/*") if p.is_file()]
    grid = tuple(s // 64 for s in m["chunk_grid"]["configuration"]["chunk_shape"])
    raw = shard.read_bytes()
    table = np.frombuffer(raw[-(np.prod(grid) * 16 + 4) : -4], "<u8").reshape(*grid, 2)
    nbytes = np.where(table[..., 0] == 2**64 - 1, 0, table[..., 1]).astype(float)
    stored.append(nbytes.sum())
    rng = np.random.default_rng(0)  # the same windows in every shard
    for _ in range(1000):
        origin = [rng.integers(0, n - W + 1) for n in shape]
        fetched.append(nbytes[tuple(slice(o // 64, (o + W - 1) // 64 + 1) for o in origin)].sum())
print(len(stored), "shards;", f"{np.mean(fetched) / 2**20:.2f} MiB fetched per window of", f"{np.mean(stored) / 2**20:.1f} MiB stored")
print(f"a window's decoded values: {W**3 * 4 / 2**20:.0f} MiB; the whole volume: {np.prod(shape) * 4 / 2**20:.1f} MiB")

9 shards; 1.37 MiB fetched per window of 5.8 MiB stored
a window's decoded values: 1 MiB; the whole volume: 39.0 MiB

The example illustrates three design considerations:

Smaller chunks reduce excess decoding for arbitrary windows but increase the number of touched chunks and storage objects or ranges.
Chunk-aligned samples avoid boundary overlap and provide the most predictable I/O cost.
Sharding reduces object count and can co-locate chunk ranges, but the reader must first obtain the shard index. Readers cache that index after the first access.

Compression changes the byte-level result: the measured random window fetches only part of the compressed shard, whereas a gzip-compressed NIfTI generally requires decompression from the start of the stream.

5 Precision

A writer can reduce stored precision in two distinct ways:

Exact narrowing during ingestion analyzes the header, value range, or a uniform exact step. It changes storage only when reconstructed values remain exact, so the navigator’s fidelity contract does not weaken.
Affine quantization maps values to integer codes with a declared scale and offset. It introduces bounded error and produces a lossy contract.

write_array() stores the supplied array directly and does not author an image fidelity contract. See Control stored numerical precision.

6 Compared with a NIfTI file

A .nii.gz file is a single gzip stream, so a reader normally decompresses from the start of that stream. A chunked BITS tensor can read and decode only the chunks intersecting the requested region. Its content hash covers tensor type, shape, and values rather than compressed storage bytes.