Skip to content

How to Run a Bulk Import

Overview

Flywheel's Bulk Import provides flexible options for importing data:

  • Default behavior - Flywheel automatically detects file types and organizes data based on DICOM headers or file paths
  • Custom behavior - Override default processing with filters, mappings, and grouping rules

Understanding Import Behavior

Before starting an import, review:

For pattern syntax details:

Instructions

0. Prerequisite -- CLI Installation

If the new (BETA) CLI is not already installed, follow the new (BETA) CLI installation instructions to install it.

An API key will be needed to sign in to the new (BETA) CLI and run commands. This can be the exact same API key used for the Legacy CLI.

If a new API key is needed, follow the documentation for creating a user API key.

1. Prepare source data

The first step is to decide how the source data will be organized into Flywheel's hierarchy.

Review the following documentation to understand your options:

For pattern syntax and examples, see the Filtering and Mapping Guide.

2. (Optional) Register external storage

Importing from 'My Computer'

This step can be skipped if you are importing data from your local computer.

If you are importing data from an external storage device, that storage device must first be registered as an External Storage within Flywheel before the import can be started. This provides Flywheel with the necessary information about how to locate and access the storage device.

Follow the documentation explaining how to create a new "External Storage" registration in Flywheel.

3. Start the import

  1. Login using the new (BETA) CLI by running the following command:

    1
    fw-beta login

  2. The CLI will then prompt for the API key. Enter your API key when prompted.

  3. Start the import using the following command:

    1
    fw-beta import run -p <project> -s <storage>

    Where:

    • <project> is replaced with the Flywheel Hierarchy of the destination project (e.g., fw://demo/Alzheimers)
      • Make sure to wrap this value in double-quotation marks if it contains any spaces (e.g., "fw://flywheel/Brain Tumor Progression")
    • <storage> is replaced with either:
      • The full path to where the source data can be found on your local machine (e.g., /Users/jdoe/Desktop/dataset1), or
      • The ID of the External Storage containing the source data (e.g., 682e21f7487fc46c72718887).

  4. Once the import begins, the progress of the import will be displayed in the CLI output.

Tip

If you are importing from an External Storage device, then you can exit out of this progress display by pressing Ctrl+c or by simply closing the terminal window altogether. This has no effect on the export job itself -- the export will still continue to run.

However, if you are importing from your local computer, then the CLI process should not be terminated until all files are fully uploaded off of your computer.

Next Steps

References

  1. fw-beta admin storage
  2. fw-beta import run