Skip to content

Syncing Data to Your Local Machine using the CLI

Use the fw sync command for downloading large volumes of data.

The Flywheel CLI Sync capability allows you to sync Flywheel data, including the folder structure, from Flywheel to your computer and Amazon S3 or Google Cloud buckets.

This is the recommended method for downloading larger datasets.

Note

The fw sync command only supports one-directional syncing, which means you cannot sync from your computer or cloud storage bucket to Flywheel.

Similar to the rsync utility, the Flywheel folder structure and data will be recreated on the destination file system on the first sync.

On subsequent syncs, only the differences between the source and the destination are copied.

Prerequisites

Follow these instructions to download and install the Flywheel CLI.

Required Arguments

  1. Open Terminal or Windows Command Prompt.
  2. Determine the source path for your Flywheel project. It follows this structure: fw://[GroupID]/[Project Label].

    You can find this path in the Flywheel UI with the following steps:

    1. Sign in to Flywheel.
    2. Go to your project.
    3. At the top, copy the path:

    NewExample-sortpath.png

  3. Determine the destination path for your Flywheel project. The destination path can be a location on your local computer or an Amazon S3/Google Cloud Bucket.

To sync to your computer

  1. Determine the location where you want to sync the Flywheel project on your computer.
  2. Enter the following command:

    fw sync [optional flags] [source-path] [destination-path] 
    

    For example:

    fw sync --full-project fw://psychology/"Longitudinal Anxiety Study" /local/data/project1
    

To sync to an Amazon S3 or Google Cloud bucket

  1. Configure the credentials for your bucket. The Flywheel CLI uses these credentials to access data in the storage bucket, so you must configure them before running the sync command. The Flywheel CLI does not support passing credential parameters to it. Make sure that the authenticated user has read/write access to data in the bucket.

  2. Start with the following command:

    fw sync <optional flags> <SRC> <DEST>
    
  3. Replace the placeholders with the relevant info for your data and environment, and add any optional flags. Use the following format for the source:

    • S3: s3://bucket-name/key-name
    • Google Cloud: gs://BUCKET_NAME1/

For example:

fw sync fw://psychology/"Anxiety Study" s3://MyStudy/DataForUpload
4. Copy and paste your command into Terminal or Windows Command prompt, and hit enter. When you use the --full-project optional flag, the fw sync command creates the following hierarchy:

project_label
|-- project_label.flywheel.json
|-- ANALYSES
|   |-- analyses_label
|       |-- analyses_label.flywheel.json
|       |-- INPUT
|       |-- OUTPUT
|-- FILES
|   |-- filename.ext
|   |-- filename.ext.flywheel.io
|-- SUBJECTS
    |-- subject_label
        |-- subject_label.flywheel.json
        |-- ANALYSES
        |-- FILES
        |-- SESSIONS
            |-- session_label
                |-- session_label.flywheel.json
                |-- ANALYSES
                |-- FILES
                |-- ACQUISITIONS
                    |-- acquisition_label
                        |-- acquisition_label.flywheel.json
                        |-- FILES

View Files Before Syncing

To perform a test run to preview how the project will be synced, enter the following command in Terminal or Windows Command Prompt:

fw sync --dry-run [source-path] [destination-path]

Review the audit log for a preview of the sync.

Only Sync Certain Filetypes

You may want to sync only certain filetypes to your computer. For example, you know that you want to sync DICOM files because you plan to run analyses locally. You can configure the sync to only include those filetypes.

In Terminal or Windows Command Prompt, enter the following command:

fw sync --include dicom [source-path] [destination-path]

Use Tags to Export a Subset of Data

This allows you to include or exclude data for download based on a subject, session, acquisition, analyses, or file tag.

fw sync --include-container-tags '{"container": ["some-tag"]}' [source-path] [destination-path]

Where Container is the location of the tag, and the options are: subject, session, acquisition, analyses, and file. Flywheel will sync that container and all children.

For example, if you want to download data only from subjects with the cohort1 tag, you would format it as: --include-container-tags '{"subject": ["cohort1"]}'.

Note

To filter by container tags, you must have tagged the container you wish to download using our tag management system. It is not currently possible to filter based on other metadata such as container labels.

When added to the command:

fw sync --include-container-tags '{"subject": ["cohort1"]}' fw://radiology/Study1 ~/Documents/ExportedData

Note

To filter by the analyses tag, you must include the --analyses or --full-project flag.

Filtering on More Than One Tag

It is possible to filter on more than one tag. When adding multiple tags, Flywheel uses AND logic to filter the data. This means that all tags specified must be present to download the data.

On the same container
 `--include-container-tag '{"session": ["cohort1", "complete"]}'`

In this example, only sessions with BOTH the cohort1 and complete tag are downloaded.

On more than one type of container
--include-container-tag '{"subject": ["example", "cohort1"], "session":["review","complete"]}'

In this example, only sessions with both the review and complete tags that also belong to subjects tagged with example and cohort are downloaded.

Optional Arguments

Optional argument Description
-h, --help show help message and exit
--config-file CONFIG_FILE, -C CONFIG_FILE Specify configuration options via config file
--no-config Do NOT load the default configuration file
-y, --yes Assume the answer is yes to all prompts
--ca-certs CA_CERTS The file to use for SSL Certificate Validation
--timezone TIMEZONE Set the effective local timezone for imports
--debug Turn on debug logging
--quiet Squelch log messages to the console
-i T, --include T Sync only files with the specified types (eg.: -i dicom)
-e T, --exclude T Skip files with the specified types (eg.: -e nifti -e qa)
--include-container-tags T Sync only the containers with specified tags and everything under them (eg.: --include-container-tag  '{"subject": ["some-tag"]}')
--exclude-container-tags T Skip the containers with specified tags and everything under them (eg.: --exclude-container-tag '{"project":  ["some-tag"]}')
-include-mlset T Sync only the subjects with specified ML Set and everything under them (eg.: --include-mlset Training)
--exclude-mlset T Skip the subjects with specified ML Set and everything under them (eg.: --exclude-mlset Validation)
-a, --analyses Include analyses
-m, --metadata Include metadata
-x, --full-project Include analyses and metadata
-z, --no-unpack Keep zipped DICOMs intact (default: extract)
-l, --list-only Show folder tree on source instead of syncing
-v, --verbose Show individual files with --list-only
-n, --dry-run Show what sync would do without transferring files
-j N, --jobs N The number of concurrent jobs to run (default: 4)
--tmp-path TMP_PATH Set custom temp dir where the zips will be extracted to (default: system temp dir)
--delete Delete extra files from destination
--export-templates-file EXPORT_TEMPLATES_FILE Set export templates YAML file
--save-audit-logs SAVE_AUDIT_LOGS Save audit log to the specified path on the current machine