Copy-on-write workspaces
This page contains unreviewed content created with large language model (LLM) tools.
A workspace is a copy-on-write overlay defined by tensorbids and implemented for bitsio stores by the overlay workspace executor. Reads prefer data in the overlay and fall back to the parent. Commit merges the overlay; discard removes its data and can retain lineage metadata for audit.
1 Storage and state
The parent is the dataset or workspace from which a new workspace is opened. The overlay is the storage location that receives new or replaced objects. bitsio stores it under bits/overlays/<id>/. The identifier is derived from the workspace name and the parent’s content state, so the same name opened against the same state produces the same identifier.
The open marker records the workspace name and identifier, its parent, the open time, and two parent-state digests. The content fingerprint covers live tensor content, referenced auxiliary payloads, and relation versions. The history digest covers current lineage-journal heads. A create-once lease serializes commit, discard, and rebase operations.
from bitsio import BITSZarrDataset
ds = BITSZarrDataset.open("ds000114.bits", read_only=False)
workspace = ds.open_workspace("try")
(marker,) = ds.workspaces().markers
print(workspace.overlay_root.startswith("bits/overlays/exp-"), marker.name, marker.parent)
print(marker.base_fingerprint.startswith("blake3:"), marker.base_history_digest.startswith("blake3:"))True try (dataset)
True True2 Reads and writes
A workspace view from ds.at(name) searches overlay layers from most specific to least specific. Writes through a writable view go only to its overlay. Nested workspace chains are readable, but bitsio does not yet route writes through more than one overlay, so a nested workspace view is read-only.
While a workspace remains open, direct writes through its parent view raise WorkspaceOpenError. This protects the state recorded in the marker. A commit or discard verifies those digests before changing the parent or removing the overlay.
from tensorbids.exceptions import WorkspaceOpenError
try:
ds.index()
except WorkspaceOpenError as error:
print(type(error).__name__, "while", [m.name for m in ds.workspaces().markers], "is open")WorkspaceOpenError while ['try'] is open3 Commit
A successful commit merges tensors, auxiliary objects, navigators, templates, relations, and producer metadata into the parent. Workspace journal events are appended to the parent journals in order; existing parent events are not rewritten. The executor then rebuilds the parent’s indexes, applies removals and replacements, and removes the overlay.
With the default validate=True, commit verifies that the merged parent has the expected content fingerprint. If the parent’s recorded content or history state has changed, commit returns CommitRefusal without merging. A key changed differently in both parent and overlay also causes a refusal. If the workspace remains valid against the new parent, rebase() updates its pinned state before a retry. See Commit a workspace.
4 Discard
Discard removes overlay arrays, relations, and derived indexes. By default it copies the lineage journals and producer metadata to bits/overlays/_discarded/<id>-<attempt>/ and writes the workspace record last. The presence of the record therefore marks a complete archive. It includes the workspace identity, attempt number, lifecycle times, and parent-state digests.
from pathlib import Path
import numpy as np
exp = ds.at("try", read_only=False)
exp.write_array("sub-01/ses-test/anat/sub-01_ses-test_desc-try_mask.nii", np.ones((4, 4, 4), "uint8"))
outcome = exp.discard()
print(outcome.mode, outcome.archive_root.startswith("bits/overlays/_discarded/exp-"), outcome.archive_root.endswith("-1"))
print(sorted(p.name for p in (Path("ds000114.bits") / outcome.archive_root).iterdir()))
ds.open_workspace("try")
table = ds.at("try").lineage_table()
print(table.filter(table["source_tree"] == "overlay").height)physical True True
['_record.json', 'provenance']
0Archiving outside the active overlay allows the same workspace name and identifier to be opened again without inheriting discarded data. Each discard increments the attempt number. purge() permanently removes one archived attempt, while discard(purge=True) skips archival.
If the underlying byte store cannot delete objects, the executor performs a logical discard: it removes the active marker but must leave the overlay bytes and record in place. The identifier cannot be reused until those objects are purged through a deletion-capable store.
5 Concurrency model
Workspace terminology resembles version-control terminology, but the behavior is intentionally narrower. Commit does not perform a three-way content merge; it requires the parent state to match or returns a refusal. Rebase updates the recorded parent state but does not resolve conflicting array values. A discarded archive retains lineage and workspace metadata, not the deleted array content, so it cannot be checked out as a historical dataset state.