Ingest a source that is not organized as BIDS

WarningContent review status

This page contains unreviewed content created with large language model (LLM) tools.

A source adapter maps an external collection to BIDS target paths and metadata before bitsio writes it. The SourceAdapter contract and manifest types are defined by tensorbids; BITSZarrDataset.auto_create() consumes any conforming adapter. The complete example is available in examples/flat_folder.py.

1 Write the adapter

An adapter has a stable name and a scan(fs) method that returns a tensorbids.io.source.SourceManifest. This example handles a directory of <subject>.nii.gz T1-weighted images and a subjects.csv file:

flat_folder.py
from tensorbids.exceptions import UnrecognisedSource
from tensorbids.io.source import SkippedFile, SourceImage, SourceManifest


class FlatFolder:
    """A folder of <subject>.nii.gz T1w images, and subjects.csv."""

    name = "flat-folder"

    def scan(self, fs):
        found = {
            path.rsplit("/", 1)[-1]: info.size
            for path, info in fs.glob_info("*.nii.gz").items()
        }
        if not found:
            raise UnrecognisedSource(f"{fs.root}: no *.nii.gz images")
        images = tuple(
            SourceImage(
                path=path,
                target_path=f"sub-{s}/anat/sub-{s}_T1w.nii",
                format="nifti",
                sidecar={"Source": path},
            )
            for path in sorted(found)
            for s in [path.removesuffix(".nii.gz")]
        )
        return SourceManifest(
            adapter=self.name,
            description={"Name": "flat folder", "BIDSVersion": "1.10.0"},
            images=images,
            skipped=(SkippedFile("subjects.csv", "imported with write_table"),),
            sizes=found,
        )
  • Each SourceImage associates one source path with a valid BIDS target path and optional sidecar metadata.
  • sizes records the size of every consumed file, keyed by its path relative to the source root.
  • The manifest must account for each discovered file as imported, nested, or skipped with a reason.
  • Raise UnrecognisedSource when the source does not match the adapter’s expected layout.

2 Use it

Construct a flat source directory from the tutorial data:

import csv
import shutil
from pathlib import Path

from bitsio.example_data import fetch_ds000114

root = fetch_ds000114()
flat = Path("scans")
flat.mkdir()
with (root / "participants.tsv").open() as tsv:
    rows = list(csv.DictReader(tsv, delimiter="\t"))
for row in rows:
    s = row["participant_id"].removeprefix("sub-")
    shutil.copyfile(root / f"sub-{s}/ses-test/anat/sub-{s}_ses-test_T1w.nii.gz", flat / f"{s}.nii.gz")
(flat / "subjects.csv").write_text(
    "subject,hand\n" + "".join(f"{r['participant_id'][4:]},{r['dominant_hand']}\n" for r in rows)
)
print(sorted(p.name for p in flat.iterdir())[:3], "...")
['01.nii.gz', '02.nii.gz', '03.nii.gz'] ...
from bitsio import BITSZarrDataset
from flat_folder import FlatFolder

ds, report = BITSZarrDataset.auto_create(flat, "flat.bits", adapter=FlatFolder())
ds.write_table("participants", flat / "subjects.csv")
print(report)
print(sorted(ds.written_paths())[:2])
created: 10 images of 10 subjects (T1w 10)
written 10, skipped 1; tables: none
T1w float32: 10 images in (64, 64, 64) chunks, blosc-zstd@9:byteshuffle (bitsio.default/float32-3d)
source 92.4 MB
['sub-01/anat/sub-01_T1w.nii', 'sub-02/anat/sub-02_T1w.nii']

The adapter records subjects.csv as deliberately skipped, and the example imports it separately with write_table(). A production adapter may instead include supported source tables in its manifest when BIDS table semantics are appropriate.

3 Ship it in a library

Publish the adapter through the tensorbids.source_adapters entry-point group, then select it by name:

[project.entry-points."tensorbids.source_adapters"]
flat-folder = "lab.adapters:FlatFolder"
ds, report = BITSZarrDataset.auto_create("scans", "flat.bits", adapter="flat-folder")

4 Constraints

  • Each target_path must be a valid BIDS path for the declared image format.
  • glob_info() may return paths relative to the filesystem root or normalized full keys, depending on the filesystem adapter. Normalize manifest paths consistently and key sizes by paths relative to the source root.
  • Use stable adapter names and target-path mapping rules. Changing either can make resumed ingestion disagree with an existing destination.

5 See also