Skip to content

Ingesting Data in BIDS Format using the CLI

Introduction

BIDS (Brain Imaging Data Structure) has become one of the standard ways for organizing and sharing neuroimaging data. If you have existing data that is in the BIDS format that you would like to migrate to Flywheel, you can use the Flywheel CLI. Because BIDS follows a standard structure, the Flywheel CLI is able to read the metadata and automatically place your files into the correct location in Flywheel.

This article explains how to use the Flywheel CLI to ingest BIDS data and includes a reference guide for the command.

Prefer Videos?

We also have a collection of webinars on using BIDS and Flywheel.

Limitation

Currently, the Flywheel CLI supports ingesting the MR modality. To ingest other modalities into Flywheel (ex. EEG or MEG), use the browser upload

Prerequisites

Before You Begin

Ensure you have:

CLI Setup:

  • Flywheel CLI installed on your computer
  • Valid API key and authenticated CLI - verify with fw status
  • Stable internet connection for data upload

Permissions:

You need the following project-level permissions in the target group/project:

  • Single File Upload/Create or Bulk File Upload - Required for CLI ingest commands
  • Create Container - Required if creating new subjects, sessions, or acquisitions

To check your permissions, navigate to the project in the web UI and check your role. Learn more about user roles and permissions.

Data Requirements:

  • Data must follow the BIDS specification
  • Required files in root directory:
    • dataset_description.json - Contains dataset metadata
    • README (optional but recommended)
    • participants.tsv (optional)
  • Subject folders must follow sub-<label> naming convention
  • Session folders must follow ses-<label> naming convention (if using sessions)
  • Validate your BIDS dataset before upload using the BIDS Validator

For Automated Processing (Optional):

To automatically process data as it arrives, set up gear rules before uploading:

  • Configure gear rules to run analysis automatically on new BIDS data
  • Test rules on a small dataset first to verify they work as expected

Example BIDS Directory Structure

From the BIDS specification documentation, here are two examples of a single session and longitudinal and multi-site study.

Single Session Example**

15d8a924597780.png

Longitudinal studies with multiple sessions (visits) example

15d8a92459b82f.png

Instructions

To ingest BIDS data into Flywheel:

  1. Open Terminal or Windows Command Prompt.
  2. Note the location of the parent folder containing the BIDS data you are ingesting.

  3. Find the group ID for the group where you want to ingest your data. To find the group ID in the Flywheel UI:

    1. Open Chrome, and sign in to your Flywheel account
    2. From the menu on the left, select Groups.
    3. Note the group ID.

    Access Control

    You must have read/write permission to ingest data to the group.

  4. In Terminal or Windows Command Prompt, enter the following command to ingest the folder:

    fw ingest bids <BIDS_Folder_Location> <group_ID>

    For example: fw ingest bids ~/Documents/Study1 Ophthalmology

    The Flywheel CLI displays the BIDS data it is processing and then shows if it is creating a new container in Flywheel or if it is adding the data to an existing container.

  5. The ingest is complete once the Flywheel CLI stops showing data. If the Flywheel CLI is unable to ingest any data, an error occurs.

  6. Sign in to Flywheel, and view your data.

  7. In the left-hand navigation pane, select Projects

  8. Your ingested data will be under a project that matches the name of the parent folder for your BIDS data.

If you used the --project option in the command, your data will be under the project you specified in the command.

The Flywheel UI offers the option to view data in the BIDS directory structure. See our BIDS Overview article for more information.

Example: Ingest a Single Session

If you do not want to ingest all of your BIDS data, you can ingest a single subject or session folder to Flywheel. The --session and --subject options only upload folders matching the name you give in the command.

Below is a directory in the BIDS format. In this example, you want to upload session 01 for subject Control01.

15d8a9245a2b37.png

  1. Open Terminal or Windows Command Prompt.
  2. Note the location of the parent folder containing the BIDS data you are ingesting.

  3. Find the group ID for the group where you want to ingest your data. To find the group ID in the Flywheel UI:

    1. Sign in to your Flywheel account
    2. From the menu on the left, select Groups.
    3. Note the group ID.

    Access Control

    You must have read/write permission to ingest data to the group.

  4. Use the --subject and --session options with the ingest bids command. Using the example directory above, the command to ingest ses-01 for subject control01 would look like:

    fw ingest bids --subject sub-Control01 --session ses-01 ~/Documents/Study1 Ophthalmology

  5. The ingest is complete once the Flywheel CLI stops showing data. If the Flywheel CLI is unable to ingest data, an error occurs.

  6. Sign in to Flywheel, and view your data.

    Tip

    The Flywheel UI offers the option to view data in the BIDS directory structure. See our BIDS Overview article for more information.

Verify Success

After the ingest completes, verify your BIDS data was imported correctly:

1. Check the CLI Output

Look for confirmation messages in the terminal showing successful import of:

  • Subjects (from sub-<label> folders)
  • Sessions (from ses-<label> folders, if applicable)
  • Data files organized by modality (anat, func, dwi, etc.)

2. View Data in Flywheel Web UI

  1. Sign in to your Flywheel site
  2. Navigate to your project: Groups → [Your Group] → [Your Project]
  3. Verify:
  4. Project name matches your BIDS dataset name (or specified --project label)
  5. Subjects are created with correct labels
  6. Sessions exist under each subject (if your dataset has sessions)
  7. Acquisitions are organized by BIDS modality and data type

3. Check BIDS Metadata

In the Flywheel web UI:

  1. Enable BIDS view by clicking the BIDS icon (if available)
  2. Verify BIDS metadata is preserved:
  3. Dataset description information
  4. Subject and session metadata
  5. File naming follows BIDS conventions
  6. Sidecar JSON files are associated with imaging data

4. Validate BIDS Structure

Check that the BIDS structure is maintained:

  • Files are organized by subject/session/datatype
  • Required BIDS files are present (dataset_description.json, etc.)
  • File naming conventions are preserved
  • Metadata relationships are intact

Next Steps

After successfully importing your BIDS data:

Usage

BIDS ingest will ingest a BIDS organized folder as a project within flywheel, under the provided group. Learn more about installing the Flywheel CLI.

fw ingest bids [optional flags] [folder] [group]

Positional Arguments

Required Argument Description
folder path to the folder with the BIDS data you would like to upload
group Flywheel group ID

Optional Arguments

Optional Argument Description
--project LABEL, -p LABEL Label of Flywheel project in which to ingest.
--subject SUBJECT Only upload data from single subject folder (e.g., sub-01).
--session SESSION Only upload data from single session folder (e.g., ses-01).

General

Optional Argument Description
-h, --help Show help message and exit.
-C PATH, --config-file Specify configuration options via config file.*
--no-config Do NOT load the default configuration file.
-y, --yes Assume the answer is yes to all prompts.
--ca-certs CA_CERTS Path to a local Certificate Authority certificate bundle file. This option may be required when using a private Certificate Authority.
--timezone TIMEZONE Set the effective local timezone to use when uploading data.
-q, --quiet Squelch log messages to the console.
-d, --debug Turn on debug logging.
-v, --verbose Get more detailed output.

* Learn more about how to create this file.

Advanced Options

For large-scale or specialized ingest workflows, the following advanced features are available:

Cluster Ingest

Speed up large imports by using Flywheel's ingest cluster to process data directly from cloud storage (S3, GCS, Azure).

When to use: Importing datasets larger than 100 GB or when faster upload speeds are needed.

Learn more about Cluster Ingest

Duplicate Detection

Automatically detect and handle duplicate files during ingest to avoid redundant uploads.

When to use: Re-importing data that may partially overlap with existing data, or ensuring data integrity.

Learn more about Duplicate Detection

Audit Logging

Generate detailed audit logs of all ingest operations for compliance and troubleshooting.

When to use: Regulated environments requiring detailed import tracking, or when troubleshooting failed imports.

Learn more about Audit Logging

S3 Credentials Setup

Configure credentials for the ingest cluster to access your S3 buckets.

When to use: Required for cluster ingest from private S3 buckets.

Learn more about S3 Credentials

Common Errors

Common Ingest Errors

For authentication, permissions, network issues, and other errors common to all ingest commands, see the Ingest Troubleshooting Guide.

BIDS Validation Errors

Cause: Dataset does not conform to BIDS specification (missing required files, incorrect naming, invalid structure).

Solution:

  • Verify dataset_description.json exists in the root directory
  • Check subject folders follow sub-<label> naming convention
  • Check session folders follow ses-<label> naming convention
  • Use a BIDS validator tool before importing: BIDS Validator
  • Review BIDS specification for requirements

"Group not found, creating..."

Cause: The specified group does not exist and is being automatically created.

Solution:

  • This is informational, not an error
  • Verify the group ID is spelled correctly
  • Check the group was created successfully in Flywheel after import

See Also

Related Commands:

BIDS Resources:

Troubleshooting: