Skip to content

Getting Data In

The first step for any new project in Flywheel is to import your imaging data, so you can organize your data, automate processing, and collaborate with other researchers.

Flywheel offers several different methods for importing imaging data:

  1. Ad hoc Upload: Upload individual files to a project, subject, session, or acquisition in Flywheel. This allows you to add resources for your project as well manually create new sessions.
  2. Bulk Import & Upload: Import enormous datasets from your local computer, cloud storage, or a file system attached to the Flywheel server.
  3. DICOM Connector (a.k.a. "Reaper"): Connect to an imaging machine to upload data directly to Flywheel.
  4. (Legacy) DICOM Upload: Drag and drop datasets under 200 MB directly to a Project.
  5. (Legacy) Bulk Ingest: Import large datasets through the Terminal or Command Prompt on your computer.

The best method to use depends on the type of data you have, the size of your dataset, and the resources available to you. This article gives an overview of each of the options for importing data as well as steps for determining which method is best for your type of data.

Summary

Below is a comparison of the import options. See the sections below for more information on each option.

Feature Ad hoc Upload Bulk Import & Upload DICOM Connector (Legacy) DICOM Upload (Legacy) Bulk Ingest
De-identification (option) No No (Coming Soon) Yes Yes Yes
Placement based on DICOM headers (option) N/A Yes Yes Yes Yes
Placement based on source file paths (option) N/A Yes N/A No Yes
Maximum data volume per upload N/A ~5 TB N/A 200 MB ~500 GB
Maximum individual file size Unlimited (24-hour timeout) Unlimited* (24-hour timeout) Unlimited (24-hour timeout) 200 MB Unlimited (24-hour timeout)

Maximum File Size with Bulk Import

Although the maximum individual file size for Bulk Import is designed to be virtually unlimited, it is dependent upon the resources available on your Flywheel site.

This limit can be increased as large as necessary by adding additional disk space to your Flywheel site.

By default, Flywheel sites are sized to allow individual files up to approximately 50 GB in size to be transferred using Bulk Import simply to avoid unnecessary hosting costs.

Contact Flywheel Support if you would like this limit increased for your site.

The Flywheel Core Web App

The Flywheel Core Web App is the web application where you can view your data. Use this option for a quick way to upload smaller datasets (about 200 MB per upload). There are several ways to upload data in the Flywheel Core Web App:

  • Ad hoc Upload: Allows you to upload any filetype to Flywheel. This option does not evaluate data or read header information, so you must manually classify and add any metadata.
  • Bulk Import/Upload: Allows you to load enormous datasets from your local computer, cloud storage, or a file system attached to the Flywheel server.
  • (Legacy) DICOM Upload: The DICOM uploader allows raw, uncompressed DICOM data to be added to projects. Flywheel reads the DICOM header information to fill out the appropriate metadata fields and automatically group sessions into Flywheel.

Ingest (CLI)

Bulk Import vs. Bulk Ingest

Bulk Import is a new capability that will eventually replace Bulk Ingest. Flywheel recommends switching away from Bulk Ingest and over to Bulk Import where possible for a better overall experience.

Flywheel's Command Line Interface (CLI) is an additional Flywheel program you download to use with your computer's Command Prompt (Windows) or Terminal app (MacOS, Linux). The CLI provides a reliable way to import large amounts of data via the ingest command; it has a variety of flexible methods to import the data without needing to change how it is stored your local drive.

The Flywheel ingest command is a great way to import large amounts of historical data in to Flywheel. For example, if you have been using XNAT in the past to store data, but now you want all of your imaging data in Flywheel, use the CLI to bulk move the data. Each command for importing is designed for different filetypes or directory structures.

Evaluate the data you are importing. Which of these scenarios best describe your data?

DICOM Files

If you have folders that include DICOM files, you can use the corresponding commands. The fw ingest dicom command can be used on directories that include other file types. The Flywheel CLI searches for the .dcm file extension and adds just those files to Flywheel.

ingest DICOM command

Following the BIDS Standard

The second ingest option relies on a standard directory structure to place data in the correct location in Flywheel. For example, the fw ingest bids command uses the industry standard directory structure for BIDS data.

To learn more about how to use these commands, see the following articles:

Directory Structure Matches the Flywheel Hierarchy

The folder command should be used if you are importing data that you previously downloaded from Flywheel, or if you are importing data that was shared from another Flywheel site. The folder command relies on your folder directory to match the hierarchy within Flywheel. The folder command does not evaluate data before placing them, so it just places the data in a Flywheel location based on where the data appears in the directory.

Below is an example of the required directory structure for the folder command:
Flywheel Hierarchy Example directory for folder import
Alt text Alt text

See this article for directions on how to use the folder command:

Learn more about the supported methods for loading de-id profiles if you use site, group, or project de-id profiles.

Directories Contain a Mix of File Formats

If your directories are a mix of file formats and do not follow a standard structure described above, you can use the template to set rules for how to import data.

Learn more about the supported methods for loading de-id profiles if you use site, group, or project de-id profiles.

The Flywheel Connector

Add-On

The Flywheel Connector is a separate installation from Flywheel Core and requires a separate purchase. Contact your Site Admin to see if your institution uses a Flywheel Connector.

When available, having a Flywheel Connector is the most effective way to get new data into Flywheel. The Flywheel connector (formerly known as the Flywheel Reaper) moves data from an imaging device directly to Flywheel.

The behavior of the Flywheel Connector is often customized to meet specific business requirements. Contact your Flywheel Site Admin for more information about how to set up your session to automatically upload data from your imaging device to a project in Flywheel.

See the following article for more information on the connector:

If your data was uploaded by a connector, review this article for tips on finding your data in Flywheel.

Additional Concerns

De-identifying Data

Check with your Site Admin to see what type of de-identification is configured for your Flywheel site.

Learn more about de-identifying data on Flywheel.

Invalid Characters for File Names

Certain characters are not allowed to be used in file names. Refer to Illegal Characters for more details.