Syncing Data to Your Local Machine using the CLI

Use the fw sync command for downloading large volumes of data.

The Flywheel CLI Sync capability allows you to sync Flywheel data, including the folder structure, from Flywheel to your computer and Amazon S3 or Google Cloud buckets.

This is the recommended method for downloading larger datasets.

Similar to the rsync utility, the Flywheel folder structure and data will be recreated on the destination file system on the first sync.

On subsequent syncs, only the differences between the source and the destination are copied.

Prerequisites

Follow these instructions to download and install the Flywheel CLI.

Required Arguments

1. Open Terminal or Windows Command Prompt.

2. Determine the source path for your Flywheel project. It follows this structure: fw://[GroupID]/[Project Label].

   You can find this path in the Flywheel UI with the following steps:

   1. Sign in to Flywheel.
   2. Go to your project.
   3. At the top, copy the path:

   

3. Determine the destination path for your Flywheel project. The destination path can be a location on your local computer or an Amazon S3/Google Cloud Bucket.

To sync to your computer

1. Determine the location where you want to sync the Flywheel project on your computer.

2. Enter the following command:

   fw sync [optional flags] [source-path] [destination-path] 
   

   For example:

   fw sync --full-project fw://psychology/"Longitudinal Anxiety Study" /local/data/project1
   

To sync to an Amazon S3 or Google Cloud bucket

1. Configure the credentials for your bucket. The Flywheel CLI uses these credentials to access data in the storage bucket, so you must configure them before running the sync command. The Flywheel CLI does not support passing credential parameters to it. Make sure that the authenticated user has read/write access to data in the bucket.

   • AWS: See Amazon's documentation on how to use the configure command to set up your credentials.
     • Learn more about creating a shared credentials file  or using environmental variables to set up credentials.
   • Google Cloud: See Google's documentation on how to use the gcloud auth login command to set up your credentials or learn more about the other authentication options.

2. Start with the following command:

   fw sync <optional flags> <SRC> <DEST>
   

3. Replace the placeholders with the relevant info for your data and environment, and add any optional flags. Use the following format for the source:

   • S3: s3://bucket-name/key-name
   • Google Cloud: gs://BUCKET_NAME1/

For example:

fw sync fw://psychology/"Anxiety Study" s3://MyStudy/DataForUpload

project_label
|-- project_label.flywheel.json
|-- ANALYSES
|   |-- analyses_label
|       |-- analyses_label.flywheel.json
|       |-- INPUT
|       |-- OUTPUT
|-- FILES
|   |-- filename.ext
|   |-- filename.ext.flywheel.io
|-- SUBJECTS
    |-- subject_label
        |-- subject_label.flywheel.json
        |-- ANALYSES
        |-- FILES
        |-- SESSIONS
            |-- session_label
                |-- session_label.flywheel.json
                |-- ANALYSES
                |-- FILES
                |-- ACQUISITIONS
                    |-- acquisition_label
                        |-- acquisition_label.flywheel.json
                        |-- FILES

View Files Before Syncing

To perform a test run to preview how the project will be synced, enter the following command in Terminal or Windows Command Prompt:

fw sync --dry-run [source-path] [destination-path]

Review the audit log for a preview of the sync.

Only Sync Certain Filetypes

You may want to sync only certain filetypes to your computer. For example, you know that you want to sync DICOM files because you plan to run analyses locally. You can configure the sync to only include those filetypes.

In Terminal or Windows Command Prompt, enter the following command:

fw sync --include dicom [source-path] [destination-path]

Use Tags to Export a Subset of Data

This allows you to include or exclude data for download based on a subject, session, acquisition, analyses, or file tag.

fw sync --include-container-tags '{"container": ["some-tag"]}' [source-path] [destination-path]

Where Container is the location of the tag, and the options are: subject, session, acquisition, analyses, and file. Flywheel will sync that container and all children.

For example, if you want to download data only from subjects with the cohort1 tag, you would format it as: --include-container-tags '{"subject": ["cohort1"]}'.

When added to the command:

fw sync --include-container-tags '{"subject": ["cohort1"]}' fw://radiology/Study1 ~/Documents/ExportedData

Filtering on More Than One Tag

It is possible to filter on more than one tag. When adding multiple tags, Flywheel uses AND logic to filter the data. This means that all tags specified must be present to download the data.

On the same container

 `--include-container-tag '{"session": ["cohort1", "complete"]}'`

In this example, only sessions with BOTH the cohort1 and complete tag are downloaded.

On more than one type of container

--include-container-tag '{"subject": ["example", "cohort1"], "session":["review","complete"]}'

In this example, only sessions with both the review and complete tags that also belong to subjects tagged with example and cohort are downloaded.

Optional Arguments