Persistent BIDS index and query semantics
This page contains unreviewed content created with large language model (LLM) tools.
tensorbids defines BIDS query semantics and index schemas. bitsio persists the indexes as Parquet and executes relational plans against them. Source adapters resolve BIDS paths and sidecar inheritance during ingestion. The indexes are derived data and must be rebuilt after writes.
1 The index
The BIDS index has one row per navigator that materializes a BIDS-visible file. Internal navigators that only produce stored tensors do not appear. Each row includes the target path, content hash, standard path components, BIDS entities, fidelity contract, output specification, referenced store keys, and schema-defined metadata columns.
from bitsio import BITSZarrDataset
ds = BITSZarrDataset.open("ds000114.bits", read_only=False)
table = ds.query(suffix="bold").table
print(table.height, "rows,", table.width, "columns")
print(table.select("subject", "session", "task", "contract_level", "RepetitionTime").head(2))4 rows, 71 columns
shape: (2, 5)
┌─────────┬─────────┬────────────────┬────────────────┬────────────────┐
│ subject ┆ session ┆ task ┆ contract_level ┆ RepetitionTime │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ str ┆ f64 │
╞═════════╪═════════╪════════════════╪════════════════╪════════════════╡
│ 01 ┆ retest ┆ fingerfootlips ┆ lossless ┆ null │
│ 01 ┆ test ┆ fingerfootlips ┆ lossless ┆ null │
└─────────┴─────────┴────────────────┴────────────────┴────────────────┘The RepetitionTime index column is null in the current implementation even though the resolved sidecars contain that field. Metadata-column population is not yet connected to the index builder. Read sidecar fields through ds.sidecar(ref); do not use these index columns for filtering until metadata population is implemented.
bitsio stores the BIDS index as subject-partitioned Parquet under bits/bids-index/. BITSDataset.index() rebuilds the BIDS and provenance indexes. auto_create() and a successful workspace commit invoke it, but individual write methods do not:
import numpy as np
class Volume:
def __init__(self, data):
self.data, self.affine, self.shape = data, np.eye(4), data.shape
target = "sub-01/ses-test/anat/sub-01_ses-test_desc-new_T1w.nii"
volume = Volume(np.zeros((4, 4, 4), "float32"))
ds.write_volume(volume, target_path=target, geometry=ds.default_geometry(volume, target))
print(len(ds.query(desc="new")))
ds.index()
print(len(ds.query(desc="new")))0
1The provenance index has the same freshness requirement. If it already exists, lineage() and lineage_table() read the persisted snapshot; a newly written key is absent until the next index() call. See Export data lineage as W3C PROV.
2 Entities and filters
Filters accept a BIDS entity’s long or short name as defined by the BIDS schema. A scalar requires equality; a list, tuple, or set accepts any member. Query.REQUIRED selects rows where the entity is present, while Query.OPTIONAL imposes no presence requirement.
from tensorbids.query import Query
print(len(ds.query(acquisition=Query.REQUIRED)), len(ds.query(task="fingerfootlips")), len(ds.query(task=["fingerfootlips"])))
print(len(ds.query(ses="retest")) == len(ds.query(session="retest")))
print(len(ds.query(modality="func")), len(ds.query(datatype="func")))0 4 4
True
4 4For pybids compatibility, modality is an alias for the datatype directory. It does not refer to the BIDS mod entity.
3 Sidecar inheritance
BIDS metadata inheritance applies matching sidecars from general to specific locations in the source tree. During ingestion, the BIDS source adapter resolves that chain for each image. A more specific value replaces an inherited value as a whole, including nested objects. The merged JSON document is stored as a content-addressed auxiliary payload referenced by the navigator’s sidecar_ref parameter.
Consequently, normal reads do not traverse a sidecar hierarchy. Changing the source dataset after ingestion has no effect on the BITS dataset; use set_sidecar() to replace stored metadata. The current validation layer checks whether a sidecar is present for suffixes that require one, but it does not validate every required field inside that sidecar.
4 Relations
Relations have declared keys, also called their grain. Participants and splits are keyed by subject; phenotype relations may add session; run-level relations such as events use the entities that identify their run. records() assembles model records from relations and image references, query_relation() filters one relation, and join() joins the BIDS index with relations. See Import tabular data.
5 Compared with pybids
pybids constructs a BIDSLayout index from a BIDS tree and resolves metadata according to the indexed file hierarchy (Yarkoni et al. 2019). A BITS dataset stores the index and resolved sidecars as part of the dataset. Opening is therefore independent of a source-tree scan, but writes become queryable only after an index rebuild. tensorbids provides a limited BIDSLayout compatibility interface over the stored index for get, get_file, get_subjects, get_metadata, and related calls.