Data lineage and provenance
This page contains unreviewed content created with large language model (LLM) tools.
BITS distinguishes reconstruction from data origin. A navigator defines how stored data becomes a BIDS-visible file. A lineage journal records how each stored array was created or changed. The provenance index, lineage table, and W3C PROV export are derived views of those journals.
1 Definitions
A navigator is a typed operation graph from stored keys to a virtual file. Its fidelity contract is inferred from the graph: byte-preserving lossless, value-preserving semantically_lossless, spatially bounded mask_lossless, or lossy.
A journal event records one operation affecting a store key. Journals are partitioned by subject. The event types are:
| Event | Records |
|---|---|
ingest |
A key receives stored data from an external source or a producer recipe |
relayout |
The same tensor values move to another physical layout |
remove |
A key is removed from the current dataset state |
condense |
A computed result is materialized in storage |
evert |
Stored data is retired in favor of recomputation from a producer recipe |
rebase |
A producer recipe is updated to reference a new source state |
derive |
A derived relation column is recorded |
Each event includes the preceding journal hash. Changing an earlier entry therefore invalidates all later hashes in that journal. Workspace commit appends the workspace entries to the parent’s journal; commit itself is not a lineage event.
A producer recipe is a navigator whose output is a derived store key. The provenance index folds journal events into the current state of each key. lineage(key) reads that state, while history(key) selects the journal entries associated with the key.
import polars as pl
from bitsio import BITSZarrDataset
ds = BITSZarrDataset.open("ds000114.bits", read_only=False)
table = ds.lineage_table()
(row,) = table.filter((pl.col("subject") == "01") & pl.col("target_path").str.contains("T1w")).to_dicts()
key = row["store_key"]
lineage = ds.lineage(key)
print(lineage.lineage.value, lineage.realization.value, lineage.from_index, lineage.lossy)
print([entry.kind.value for entry in ds.history(key).entries])
print(lineage.ingest_record.source_ref)root stored True False
['ingest']
sub-01/ses-test/anat/sub-01_ses-test_T1w.nii.gzWhen a provenance index exists, lineage reads use the last persisted snapshot. Call index() after writing; otherwise a new key may be absent from lineage() and lineage_table().
2 Contracts and loss
When a container write authors a lossy navigator, for example through affine quantization to int8, the ingest event records that contract and the key’s lineage reports lossy. write_array() writes a tensor without authoring a virtual image or image fidelity contract. See Control stored numerical precision.
3 The PROV mapping
ds.prov() maps the current lineage of selected keys to the W3C PROV data model (PROV-DM):
| PROV | From |
|---|---|
prov:Entity |
A store key and its content hash, origin, storage status, subject, layout, and loss status |
prov:Activity |
The latest journal event for that key |
wasGeneratedBy |
Association between the key entity and its latest activity |
prov:PrimarySource and wasDerivedFrom |
An external source recorded by a root ingest |
prov:Plan and p-plan:Step |
The layout profile and matching rule used for storage |
prov:SoftwareAgent and wasAssociatedWith |
The software name passed as agent |
If a write supplies no ingest source, its target path is used as the source reference. Pass ingest_record when the output was derived from another file or dataset object. See Export data lineage as W3C PROV.
4 Integrity properties
BITS guardrail G3 requires a dataset to retain the provenance information needed to interpret its current content and to identify a trusted manifest state. bitsio stores lineage journals and producer recipes in base datasets and workspaces. A normal discarded-workspace archive retains its journal and producer metadata. freeze() records content and history digests and rejects a dataset with no provenance state to pin.
5 Condense and evert
condense() materializes a computed navigator output as stored data; evert() removes stored data while retaining a recipe that can recompute it. These APIs are part of the tensorbids operation model. The bitsio reference resolver is limited to reconstruction operators, so these operations have limited practical use in a bitsio-only workflow and are not used by the tutorials.
6 Compared with BIDS
BIDS GeneratedBy records pipelines at the derivative-dataset level (Gorgolewski et al. 2016). BITS lineage operates at the stored-key level and records successive events. W3C PROV export projects those entries into entities, activities, sources, plans, and software agents. Applications can use both dataset-level BIDS provenance and key-level BITS lineage.