Use the Sync Command for Downloading Data

Introduction

The Flywheel CLI sync capability allows you to sync Flywheel data, including the folder structure, from Flywheel to your computer and Amazon S3 or Google Cloud buckets. This is the recommended method for downloading larger datasets.

Similar to the rsync utility, the Flywheel folder structure and data will be recreated on the destination file system on the first sync. On the following syncs, only the differences between the source and the destination files are copied. Note: this command only supports one-directional syncing, which means you cannot sync from your computer or cloud storage bucket to Flywheel.

Instruction Steps

Before Beginning

Follow these instructions to download and install the Flywheel CLI.

1. Open Terminal or Windows Command Prompt.
2. Determine the source path for your Flywheel project. It follows this structure: fw://[GroupID]/[Project Label].

You can find this path in the Flywheel UI by going to:

1. Sign in to Flywheel.
2. Go to your project.
3. At the top, copy the path.

1. Determine the destination path for your Flywheel project. The destination path can be a location on your local computer or an Amazon S3/Google Cloud Bucket.

2. To sync to your computer:

3. Determine the folder location where you want to sync the Flywheel project on your computer.

4. Enter the following command

fw sync [optional flags] [source-path] [destination-path] 

For example

fw sync --full-project fw://psychology/"Longitudinal Anxiety Study" /local/data/project1

• To sync to an Amazon S3 or Google Cloud bucket:

• Configure the credentials for your bucket. The Flywheel CLI uses these credentials to access data in the storage bucket, so you must configure them before running the sync command. The Flywheel CLI does not support passing credential parameters to it. Make sure that the authenticated user has read/write access to data in the bucket.

• AWS: See Amazon's documentation on how to use the configure command to set up your credentials or learn more about creating a shared credentials file or using environmental variables to set up credentials

• Google Cloud: See Google's documentation on how to use the gcloud auth login command to set up your credentials or learn more about using the other authentication options.

• Start with the following command:

fw sync <optional flags> <SRC> 
<DEST>

1. Replace with the relevant info for your data and environment, and add any optional flags. Use the following format for the source:

2. S3: s3://bucket-name/key-name

3. Google Cloud: gs://BUCKET_NAME1/

For example:

fw sync fw://psychology/"Anxiety Study" s3://MyStudy/DataForUpload 

1. Copy and paste your command into Terminal or Windows Command prompt, and hit enter.. When you use the --full-project optional flag, the fw sync command creates the following hierarchy:

project_label
|-- project_label.flywheel.json
|-- ANALYSES
|   |-- analyses_label
|       |-- analyses_label.flywheel.json
|       |-- INPUT
|       |-- OUTPUT
|-- FILES
|   |-- filename.ext
|   |-- filename.ext.flywheel.io
|-- SUBJECTS
    |-- subject_label
        |-- subject_label.flywheel.json
        |-- ANALYSES
        |-- FILES
        |-- SESSIONS
            |-- session_label
                |-- session_label.flywheel.json
                |-- ANALYSES
                |-- FILES
                |-- ACQUISITIONS
                    |-- acquisition_label
                        |-- acquisition_label.flywheel.json
                        |-- FILES

View Files Before Syncing

To perform a test run to preview how the project will be synced, enter the following command from Terminal or Windows Command Prompt:

fw sync --dry-run [source-path] [destination-path]

Review the audit log for a preview of the sync

Only Sync Certain Filetypes

You may want to only sync certain filetypes to your computer. For example, you know that you want to sync DICOM files because you plan to run analyses locally. You configure the sync to only include those filetypes.

In Terminal or Windows Command Prompt, enter the following command:

fw sync --include dicom [source-path] [destination-path]

Use Tags to Export a Subset of Data

This allows you to include or exclude data for download based on a subject, session, acquisition, analyses, or file that has been tagged at the container level.

fw sync --include-container-tags '{"container": ["some-tag"]}' [source-path] [destination-path]

Container is the location of the tag, and the options are: subject, session, acquisition, analyses, and file. Flywheel will sync that container and all children.

Note

To filter by container tags, you must have tagged the container you wish to download using our tag management system. It is not currently possible to filter based on other metadata such as container labels.

For example, let's say you want to download data only from subjects with the cohort1 tag, you would format it as: --include-container-tags '{"subject": ["cohort1"]}'.

When added to the command:

fw sync --include-container-tags '{"subject": ["cohort1"]}' fw://radiology/Study1 ~/Documents/ExportedData

Note 2

To filter by analyses tag, you must include the --analyses or --full-project flag.

Filtering on More Than One Tag

It is possible to filter on more than one tag. When adding multiple tags, Flywheel uses AND logic to filter the data. This means that all tags specified must be present to download the data.

• Include more than one tag on the same container: --include-container-tag '{"session": ["cohort1", "complete"]}'

For example, only sessions with BOTH the cohort1 and complete tag are downloaded.

• To include more than one type of container: --include-container-tag '{"subject": ["example", "cohort1"], "session":["review","complete"]}'

For example, only sessions with both the review and complete tags that also belong to subjects tagged with example and cohort are downloaded

Optional Arguments