Use a copy-on-write workspace
This page contains unreviewed content created with large language model (LLM) tools.
This chapter creates a copy-on-write workspace over the dataset from part one and writes a quantized copy of one BOLD image to that workspace. The base dataset remains unchanged. The following chapters inspect the write’s lineage, export it as W3C PROV, and discard the workspace while retaining an audit record.
1 Before you start
Open the ds000114.bits dataset created in part one with workspace operations enabled:
from bitsio import BITSZarrDataset
ds = BITSZarrDataset.open("ds000114.bits", read_only=False)
print(len(ds.query(suffix="bold")), "BOLD runs")4 BOLD runs2 Open a workspace
A workspace is an isolated overlay over a dataset or another workspace. Open one named "int8-bold", then obtain a writable view of that overlay:
from tensorbids.exceptions import WorkspaceOpenError
WORKSPACE_NAME = "int8-bold"
ds.open_workspace(WORKSPACE_NAME)
exp = ds.at(WORKSPACE_NAME, read_only=False)
print([marker.name for marker in ds.workspaces().markers])
try:
ds.write_table("split/other", ds.split("folds"))
except WorkspaceOpenError as error:
print("refused:", type(error).__name__)['int8-bold']
refused: WorkspaceOpenErrords continues to address the base dataset, while exp reads the workspace overlay before falling back to the base. While a workspace is open, writes to its parent are rejected with WorkspaceOpenError. This restriction allows a later commit or discard to verify that the parent has not changed since the workspace was opened.
The workspace identifier is derived from its name and the recorded state of its parent. A workspace ends by committing its changes or discarding them.
3 Write a run as int8 codes
Read subject 01’s BOLD run and write a quantized version whose stored codes span the full int8 range:
from dataclasses import dataclass
import numpy as np
from tensorbids.authoring import AffineQuantization
from tensorbids.provenance.ingest_record import IngestSource
@dataclass
class Volume:
"""Values to write, with their affine; a 4-D run is time first."""
data: np.ndarray
affine: np.ndarray
@property
def shape(self):
return self.data.shape
(source,) = exp.query(subject="01", session="test", suffix="bold")
(image,) = exp.read(target_path=source.target_path, form="object")
bold = np.moveaxis(image.get_fdata(dtype="float32"), -1, 0)
volume = Volume(bold, image.affine)
rung = AffineQuantization.spanning(bold.min(), bold.max(), "int8")
target = "sub-01/ses-test/func/sub-01_ses-test_task-fingerfootlips_desc-int8_bold.nii"
exp.write_volume(
volume,
target_path=target,
geometry=exp.default_geometry(volume, target, dtype="int8"),
precision=rung,
ingest_record=IngestSource(source_ref=source.target_path),
)
exp.index()
print(f"scale {rung.scale:.4f}, offset {rung.offset:.4f}")
(written,) = exp.read(target_path=target, form="tensor")
print("largest error", float(np.abs(written - bold).max()), "<=", rung.scale / 2)
(nifti,) = exp.read(target_path=target, form="object")
print(nifti.shape == image.shape, np.allclose(nifti.affine, image.affine))scale 16.7647, offset 2145.8824
largest error 8.35302734375 <= 8.382352941176471
True TrueAffineQuantization.spanning() derives a linear scale and offset from the observed value range. Quantization introduces an error of at most half a scale step, so the navigator’s fidelity contract is lossy. IngestSource records the source BOLD path in the lineage of the new stored array.
write_volume() expects a four-dimensional volume in (time, i, j, k) order and restores NIfTI’s time-last order when materializing the file. Move the time axis explicitly before writing, as shown above; the API cannot infer an incorrectly ordered input array.
The BITS tensor store contains the int8 codes directly. The reconstructed NIfTI object exposes dequantized float32 values; it is not an int8 NIfTI using scl_slope and scl_inter. The record-set API can return the codes and their scale parameters separately.
4 Train on the codes
Select the stored port and attach the scale and offset instead of dequantizing during the read:
import torch
from tensorbids.query import BIDSReference
codes = exp.records(
BIDSReference(suffix="bold", desc="int8", port="stored", dequantize="attach")
)
data = torch.utils.data.DataLoader(codes.loader(batch_size=1, stack=True), batch_size=None)
batch = next(iter(data))
q = batch["bold"]
print(q.dtype, tuple(q.shape))
print(batch["bold.scale"].tolist() == [rung.scale], batch["bold.offset"].tolist() == [rung.offset])
values = q.double() * batch["bold.scale"].view(-1, 1, 1, 1, 1) + batch["bold.offset"].view(-1, 1, 1, 1, 1)
print(np.allclose(values[0].numpy(), written, rtol=1e-5))torch.int8 (1, 184, 64, 64, 30)
True True
TrueThe batch contains int8 codes plus one scale and offset per record. Apply the affine transformation on the target device or inside the model. The result matches the resolver’s dequantized float32 read to numerical precision.
Lazy spatial windows work with stored codes in the same way as in part one:
windows.py
def window(record):
"""A random 32 x 32 x 16 crop of every frame of the run's codes."""
return {"bold": record["bold"].random_window(i=32, j=32, k=16)}from windows import window
crops = codes.select(window, seed=0).loader(batch_size=1, stack=True)
batch = next(iter(torch.utils.data.DataLoader(crops, batch_size=None)))
print(batch["bold"].dtype, tuple(batch["bold"].shape), batch["bold.scale"].tolist() == [rung.scale])torch.int8 (1, 184, 32, 32, 16) TrueA window retains the scale and offset of its source image because the affine dequantization is independent for each value.
Record sets backed by a workspace view cannot currently be serialized to PyTorch worker processes. Use the default num_workers=0 for these loaders.
Quantization is part of the stored representation, not a MONAI transform. The loader returns the stored codes and their scaling parameters.