Create a BITS dataset

WarningContent review status

This page contains unreviewed content created with large language model (LLM) tools.

This chapter converts a ten-subject subset of OpenNeuro ds000114 from BIDS to BITS. Later chapters use the resulting dataset for queries, image reads, and batched model input.

1 Install and fetch the data

Create a virtual environment and install the development versions of the BITS packages:

python3 -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
git clone https://github.com/hypercoil/tensorbids.git
git clone https://github.com/hypercoil/niflash.git
git clone https://github.com/hypercoil/bitsio.git
python -m pip install -e ./tensorbids -e ./niflash
python -m pip install -e "./bitsio[tensorstore,niflash]" torch

tensorstore provides a Zarr v3 array backend, niflash provides native neuroimaging container support, and PyTorch supplies the data loader used later in the tutorial.

Download the pinned subset of a BIDS dataset (OpenNeuro ds000114):

from bitsio.example_data import fetch_ds000114

root = fetch_ds000114()
files = sorted(p.relative_to(root).as_posix() for p in root.rglob("*") if p.is_file())
print(len(files), "files")
print("\n".join(files))
18 files
dataset_description.json
participants.tsv
sub-01/ses-retest/func/sub-01_ses-retest_task-fingerfootlips_bold.nii.gz
sub-01/ses-test/anat/sub-01_ses-test_T1w.nii.gz
sub-01/ses-test/func/sub-01_ses-test_task-fingerfootlips_bold.nii.gz
sub-02/ses-retest/func/sub-02_ses-retest_task-fingerfootlips_bold.nii.gz
sub-02/ses-test/anat/sub-02_ses-test_T1w.nii.gz
sub-02/ses-test/func/sub-02_ses-test_task-fingerfootlips_bold.nii.gz
sub-03/ses-test/anat/sub-03_ses-test_T1w.nii.gz
sub-04/ses-test/anat/sub-04_ses-test_T1w.nii.gz
sub-05/ses-test/anat/sub-05_ses-test_T1w.nii.gz
sub-06/ses-test/anat/sub-06_ses-test_T1w.nii.gz
sub-07/ses-test/anat/sub-07_ses-test_T1w.nii.gz
sub-08/ses-test/anat/sub-08_ses-test_T1w.nii.gz
sub-09/ses-test/anat/sub-09_ses-test_T1w.nii.gz
sub-10/ses-test/anat/sub-10_ses-test_T1w.nii.gz
task-fingerfootlips_bold.json
task-fingerfootlips_events.tsv

The first call downloads approximately 190 MB to data_home() (~/.cache/bitsio unless BITSIO_DATA is set). Subsequent calls use the cached copy.

2 Create the dataset

auto_create scans the source dataset, writes image arrays and supporting metadata to their respective stores, imports supported BIDS tables, writes the navigators required to reconstruct each image, and builds the BIDS and provenance indexes.

from bitsio import BITSZarrDataset

ds, report = BITSZarrDataset.auto_create(root, "ds000114.bits")
print(report)
created: 14 images of 10 subjects (bold 4, T1w 10)
written 14, skipped 0; tables: events/fingerfootlips, participants
T1w float32: 10 images in (64, 64, 64) chunks, blosc-zstd@9:byteshuffle (bitsio.default/float32-3d)
bold int16: 4 images in (16, 32, 32, 30) chunks, blosc-zstd@9:byteshuffle (bitsio.default/int16-time)
source 189.9 MB

View details of the dataset creation report:

import json   # For pretty printing
print("Stored array layouts")
print("\n".join(str(stratum) for stratum in report.strata))
print(f"\nImage counts\n{json.dumps(report.images, indent=2)}")
print("\nImported tables")
print({table.name: table.write.roundtrip.to_string() for table in report.write.tables})
Stored array layouts
StratumLayout(suffix='T1w', dtype='float32', rule='bitsio.default/float32-3d', chunk_shape=(64, 64, 64), codec='blosc-zstd@9:byteshuffle', images=10)
StratumLayout(suffix='bold', dtype='int16', rule='bitsio.default/int16-time', chunk_shape=(16, 32, 32, 30), codec='blosc-zstd@9:byteshuffle', images=4)

Image counts
{
  "subject": {
    "01": 3,
    "02": 3,
    "03": 1,
    "04": 1,
    "05": 1,
    "06": 1,
    "07": 1,
    "08": 1,
    "09": 1,
    "10": 1
  },
  "session": {
    "retest": 2,
    "test": 12
  },
  "datatype": {
    "func": 4,
    "anat": 10
  },
  "suffix": {
    "bold": 4,
    "T1w": 10
  }
}

Imported tables
{'events/fingerfootlips': 'semantically_lossless', 'participants': 'semantically_lossless'}

Each StratumLayout groups images that share a BIDS suffix, stored data type, chunk geometry, and codec. The report also separates source image counts by BIDS entity and records the round-trip fidelity of each imported table.

The following guides describe the main ingest customizations:

auto_create can resume an interrupted ingestion. If the destination already contains an image with the expected content, the operation skips that image:

_, again = BITSZarrDataset.auto_create(root, "ds000114.bits")
print(again)
resumed: 14 images of 10 subjects (bold 4, T1w 10)
written 0, skipped 16; tables: none
T1w float32: 10 images in (64, 64, 64) chunks, blosc-zstd@9:byteshuffle (bitsio.default/float32-3d)
bold int16: 4 images in (16, 32, 32, 30) chunks, blosc-zstd@9:byteshuffle (bitsio.default/int16-time)
source 189.9 MB