Architecture and library responsibilities
This page contains unreviewed content created with large language model (LLM) tools.
BITS separates a dataset’s BIDS-visible files from the physical storage used to serve them. tensorbids defines the data model and backend contracts. bitsio implements the tensor store, auxiliary store, Parquet table engine, and reference resolver. bitsfuse supplies higher-level integrations such as a filesystem projection and MONAI adapters.
1 Motivation
A conventional BIDS dataset (Gorgolewski et al. 2016) represents an image as a file, commonly a gzip-compressed NIfTI, and derives metadata by applying BIDS sidecar inheritance to the file’s path. This representation is well suited to exchange and command-line tools. It is less efficient when an application repeatedly reads small spatial regions: gzip decompression begins at the start of the stream, and sidecar inheritance must be resolved before metadata can be used.
BITS retains the BIDS-visible paths and file semantics while allowing the underlying numerical values, metadata, and relations to use storage optimized for their access patterns. Chunked arrays support region reads; merged sidecars avoid repeated inheritance resolution; and a persistent relational index serves BIDS entity queries.
2 Processing model
The main objects have distinct roles:
- A source adapter scans an external dataset and produces a typed manifest.
- The dataset writer stores numerical arrays, auxiliary payloads, and relations. It also writes a navigator for each virtual file. A navigator is a typed operation graph that references stored data and describes how to reconstruct the file.
- The index records BIDS entities and other query fields for each navigator.
- A query returns navigator references. The resolver evaluates each selected navigator as reconstructed bytes, an image object, or a numerical array.
- Record sets join image references with relation columns and can translate lazy spatial selections into tensor-store region reads.
This separation allows array chunking, compression, sharding, and backend selection to change without changing the BIDS target path or reconstruction logic.
3 Library boundaries
| Library | Responsibility |
|---|---|
tensorbids |
BITS models and protocols, including navigators, storage and resolver interfaces, queries, record sets, workspaces, and provenance operations |
bitsio |
Zarr v3 tensor storage, content-addressed auxiliary storage, a sharded-Parquet table engine, container ingestion, and the NumPy-based reference resolver |
bitsfuse |
User-facing integrations over a BITS dataset, including filesystem projection, writable staging, and MONAI adapters |
Libraries that consume BITS should depend on tensorbids for protocol types when they do not require a specific backend. Depend on bitsio when the concrete storage implementation or reference resolver is required. Code that uses the MONAI adapters should depend on bitsfuse.
4 Fidelity and provenance
The resolver does not infer data quality from a file extension. Each navigator has a mechanically derived fidelity contract:
lossless: reconstructed bytes are preserved;semantically_lossless: interpreted values are preserved although the byte representation may differ;mask_lossless: values are preserved within a declared spatial support; andlossy: reconstruction introduces a declared loss, such as affine quantization.
Provenance is represented separately. The navigator explains how stored data becomes a virtual file; the append-only lineage journal records how each stored array was created, changed, or removed. This distinction lets clients inspect file fidelity and data origin independently.
5 Conformance requirements
The BITS specification names three dataset guardrails:
| Requirement | Practical implication for bitsio |
|---|---|
| G1: bounded requests and read amplification for training samples | Chunk and shard geometry must be selected for the application’s read regions. Chunk-aligned reads perform best; arbitrary windows trade request count against excess decoded data. |
| G2: sufficient data to reconstruct supported BIDS files | Navigators retain the arrays and auxiliary payloads required for reconstruction. NIfTI content is reconstructed, but source gzip framing is not retained. |
| G3: complete provenance and a trusted manifest state | Lineage journals and producer metadata are stored with the dataset, and freezing records content and history digests. |
These are conformance targets rather than performance guarantees for every layout. In particular, no single chunk geometry is optimal for both whole-file reads and arbitrary small windows. See Storage backends and array layout.
6 Relationship to established tools
NiBabel (Markiewicz et al. 2026) implements neuroimaging file formats and image objects. pybids (Yarkoni et al. 2019) indexes and queries BIDS file trees. MONAI (Cardoso et al. 2022) provides medical-imaging transforms and data pipelines. BITS does not replace these concerns: bitsio provides an alternative physical representation and resolver, while bitsfuse provides adapters to file- and MONAI-oriented workflows.