Export data lineage as W3C PROV

WarningContent review status

This page contains unreviewed content created with large language model (LLM) tools.

Read the current state of the lineage journal as a table, then export selected stored arrays using the W3C PROV data model.

1 The lineage table

from bitsio import BITSZarrDataset

ds = BITSZarrDataset.open("ds000114.bits", read_only=False)
table = ds.lineage_table()
print(table.height, "keys;", table.columns[:6])
print(table.group_by("lineage", "realization", "lossy").len().sort("len"))
14 keys; ['subject', 'store_key', 'target_path', 'content_hash', 'lineage', 'realization']
shape: (1, 4)
┌─────────┬─────────────┬───────┬─────┐
│ lineage ┆ realization ┆ lossy ┆ len │
│ ---     ┆ ---         ┆ ---   ┆ --- │
│ str     ┆ str         ┆ bool  ┆ u32 │
╞═════════╪═════════════╪═══════╪═════╡
│ root    ┆ stored      ┆ false ┆ 14  │
└─────────┴─────────────┴───────┴─────┘

The table contains one row per current stored array, including its subject, target path, content hash, origin, storage status, producer, source record, last event, fidelity contract, layout, and storage layer. It is a Polars data frame, so ordinary expressions can select lossy rows or workspace rows (source_tree == "overlay").

Call ds.index() after writes. If a provenance index exists, lineage_table() reads that persisted snapshot and does not include later writes until the next index build.

2 Record what a write was made from

Pass an IngestSource when a write was derived from another dataset object:

import numpy as np
from tensorbids.provenance.ingest_record import IngestSource

(source,) = ds.query(subject="01", suffix="T1w")
(t1w,) = ds.read(target_path=source.target_path, form="tensor")


class Volume:
    def __init__(self, data, affine):
        self.data, self.affine, self.shape = data, affine, data.shape


(image,) = ds.read(target_path=source.target_path, form="object")
target = "sub-01/ses-test/anat/sub-01_ses-test_desc-brain_T1w.nii"
volume = Volume((t1w * (t1w > t1w.mean())).astype("float32"), image.affine)
ds.write_volume(
    volume,
    target_path=target,
    geometry=ds.default_geometry(volume, target),
    ingest_record=IngestSource(source_ref=source.target_path),
)
ds.index()
(row,) = ds.lineage_table().filter(ds.lineage_table()["target_path"] == target).to_dicts()
key = row["store_key"]
print(row["lineage"], row["ingest_record"])
root {"source_ref":"sub-01/ses-test/anat/sub-01_ses-test_T1w.nii","consent":null}

Without an explicit ingest_record, the write records its own target path as the source reference. Supply the input path when that default would be misleading.

3 Export PROV

from pathlib import Path

doc = ds.prov(keys=[key], agent="bitsio")
print(sorted(doc.to_json()))
provn = doc.to_provn()
print("\n".join(line for line in provn.splitlines() if "wasDerivedFrom" in line))
Path("lineage.provn").write_text(provn)
Path("lineage.dot").write_text(doc.to_dot())
print(len(ds.prov().to_json()["entity"]), "entities in the whole dataset")
['activity', 'agent', 'entity', 'prefix', 'wasAssociatedWith', 'wasDerivedFrom', 'wasGeneratedBy']
  wasDerivedFrom(bits:key/k-9a3dd036e002902ec67eb34b455a7b84, bits:source/e3142252d72dc694, bits:activity/ingest/k-9a3dd036e002902ec67eb34b455a7b84/2026-10-25T01:30:14.001230+00:00, [prov:type='prov:PrimarySource'])
33 entities in the whole dataset

Without keys, ds.prov() describes all current keys. The agent argument identifies the software associated with each activity. to_json() returns PROV-JSON, to_provn() returns PROV-N, and to_dot() returns a Graphviz graph:

dot -Tsvg lineage.dot -o lineage.svg

4 A discarded workspace

A discarded workspace retains its journal. Open the archived attempt and call lineage_table() on that archive view. See Inspect and purge discarded workspaces.

5 Current limitations

  • Activity identifiers include the event time, so repeated operations that produce identical data still receive distinct activity identifiers.
  • lineage_table() and lineage(key) use the last persisted provenance index when one exists. Rebuild the index after writing.

6 See also