The fw sync Command

Use the fw sync command for downloading large volumes of data.

The Flywheel CLI Sync capability allows you to sync Flywheel data, including the folder structure, from Flywheel to your computer and Amazon S3 or Google Cloud buckets.

This is the recommended method for downloading larger datasets.

Similar to the rsync utility, the Flywheel folder structure and data will be recreated on the destination file system on the first sync.

On subsequent syncs, only the differences between the source and the destination are copied.

Prerequisites

Follow these instructions to download and install the Flywheel CLI.

Before You Begin

Ensure you have:

CLI Setup:

• Flywheel CLI installed on your computer
• Valid API key and authenticated CLI - verify with fw status
• Stable internet connection for data download

Permissions:

You need the following project-level permissions:

• Download File - Required to download files from the project
• View Metadata - Required to access project structure and metadata

To check your permissions, navigate to the project in the web UI and check your role. Learn more about user roles and permissions.

Destination Requirements:

For syncing to your local computer:

• Sufficient disk space for the project data
• Write permissions for the destination directory
• Consider using --dry-run first to preview download size

For syncing to cloud storage (S3/Google Cloud):

• Cloud credentials configured before running sync
  • AWS: Configure AWS credentials
  • Google Cloud: Configure GCloud authentication
• Read/write access to the destination bucket
• Verify bucket path is correct and accessible

Important Notes:

• fw sync is one-directional (Flywheel → destination only)
• Subsequent syncs only transfer changed files (similar to rsync)
• Use --dry-run flag to preview what will be synced without transferring files
• Use --list-only to see the folder tree without syncing

Instructions

1. Open Terminal or Windows Command Prompt.

2. Determine the source path for your Flywheel project. It follows this structure: fw://[GroupID]/[Project Label].

   You can find this path in the Flywheel UI with the following steps:

   1. Sign in to Flywheel.
   2. Go to your project.
   3. At the top, copy the path:

   

3. Determine the destination path for your Flywheel project. The destination path can be a location on your local computer or an Amazon S3/Google Cloud Bucket.

To sync to your computer

1. Determine the location where you want to sync the Flywheel project on your computer.

2. Enter the following command:

   fw sync [optional flags] [source-path] [destination-path]

   For example:

   fw sync --full-project fw://psychology/"Longitudinal Anxiety Study" /local/data/project1

To sync to an Amazon S3 or Google Cloud bucket

1. Configure the credentials for your bucket. The Flywheel CLI uses these credentials to access data in the storage bucket, so you must configure them before running the sync command. The Flywheel CLI does not support passing credential parameters to it. Make sure that the authenticated user has read/write access to data in the bucket.

   • AWS: See Amazon's documentation on how to use the configure command to set up your credentials.
     • Learn more about creating a shared credentials file or using environmental variables to set up credentials.
   • Google Cloud: See Google's documentation on how to use the gcloud auth login command to set up your credentials or learn more about the other authentication options.

2. Start with the following command:

   fw sync <optional flags> <SRC> <DEST>

3. Replace the placeholders with the relevant info for your data and environment, and add any optional flags. Use the following format for the source:

   • S3: s3://bucket-name/key-name
   • Google Cloud: gs://BUCKET_NAME1/

   For example:

   fw sync fw://psychology/"Anxiety Study" s3://MyStudy/DataForUpload

4. Copy and paste your command into Terminal or Windows Command prompt, and hit enter. When you use the --full-project optional flag, the fw sync command creates the following hierarchy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

project_label
|-- project_label.flywheel.json
|-- ANALYSES
|   |-- analyses_label
|       |-- analyses_label.flywheel.json
|       |-- INPUT
|       |-- OUTPUT
|-- FILES
|   |-- filename.ext
|   |-- filename.ext.flywheel.io
|-- SUBJECTS
    |-- subject_label
        |-- subject_label.flywheel.json
        |-- ANALYSES
        |-- FILES
        |-- SESSIONS
            |-- session_label
                |-- session_label.flywheel.json
                |-- ANALYSES
                |-- FILES
                |-- ACQUISITIONS
                    |-- acquisition_label
                        |-- acquisition_label.flywheel.json
                        |-- FILES

View Files Before Syncing

To perform a test run to preview how the project will be synced, enter the following command in Terminal or Windows Command Prompt:

fw sync --dry-run [source-path] [destination-path]

Review the audit log for a preview of the sync.

Only Sync Certain Filetypes

You may want to sync only certain filetypes to your computer. For example, you know that you want to sync DICOM files because you plan to run analyses locally. You can configure the sync to only include those filetypes.

In Terminal or Windows Command Prompt, enter the following command:

fw sync --include dicom [source-path] [destination-path]

Use Tags to Export a Subset of Data

This allows you to include or exclude data for download based on a subject, session, acquisition, analyses, or file tag.

fw sync --include-container-tags '{"container": ["some-tag"]}' [source-path] [destination-path]

Where Container is the location of the tag, and the options are: subject, session, acquisition, analyses, and file. Flywheel will sync that container and all children.

For example, if you want to download data only from subjects with the cohort1 tag, you would format it as: --include-container-tags '{"subject": ["cohort1"]}'.

When added to the command:

fw sync --include-container-tags '{"subject": ["cohort1"]}' fw://radiology/Study1 ~/Documents/ExportedData

Filtering on More Than One Tag

It is possible to filter on more than one tag. When adding multiple tags, Flywheel uses AND logic to filter the data. This means that all tags specified must be present to download the data.

On the same container

--include-container-tag '{"session": ["cohort1", "complete"]}'

In this example, only sessions with BOTH the cohort1 and complete tag are downloaded.

On more than one type of container

--include-container-tag '{"subject": ["example", "cohort1"], "session":["review","complete"]}'

In this example, only sessions with both the review and complete tags that also belong to subjects tagged with example and cohort are downloaded.

Verify Success

After the sync completes, verify your data was synced correctly:

1. Check the CLI Output

Review the sync summary in the terminal:

• Number of files synced
• Files created, updated, or deleted
• Any errors or warnings during sync

2. Verify Destination Structure

Navigate to your destination and check the folder structure:

For local filesystem:

1

ls -la /local/data/project1

For cloud storage, use appropriate tools (AWS CLI, gsutil, etc.)

Expected structure with --full-project flag:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

project_label/
├── project_label.flywheel.json
├── ANALYSES/
├── FILES/
└── SUBJECTS/
    └── subject_label/
        ├── subject_label.flywheel.json
        ├── SESSIONS/
        │   └── session_label/
        │       ├── session_label.flywheel.json
        │       └── ACQUISITIONS/
        └── FILES/

3. Check File Counts

Compare file counts between Flywheel and destination:

• Number of subjects matches
• Number of sessions per subject matches
• File counts per acquisition match
• Verify filtered data matches expectations (if using --include, --exclude, or tag filters)

4. Review Audit Logs (if enabled)

If you used --save-audit-logs flag:

1

cat /path/to/audit-log.csv

Check the audit log for:

• Files with status "completed" or "skipped"
• Any errors in the error_message column
• Actions taken (created, updated, deleted)

5. Test Subsequent Sync

Run sync again to verify only changed files are transferred:

1

fw sync [same-command-as-before]

Should show minimal or no files transferred if nothing changed.

Next Steps

After successfully syncing your data:

• Analyze locally: Use the synced data for local analysis or processing
• Automate syncs: Set up scheduled syncs using cron (Linux/Mac) or Task Scheduler (Windows)
• Upload results back: Use fw upload to add analysis results back to Flywheel
• Monitor changes: Run periodic syncs to keep local/cloud copies up to date with Flywheel

Usage

Optional Arguments

Sync

* Learn more about file types in Flywheel

General

* Learn more about how to create this file.

Common Errors

"No project found at "

Cause: The source path does not point to a valid Flywheel project or the project doesn't exist.

Solution:

• Verify the source path follows the format: fw://group/project
• Check the group ID and project label are correct (case-sensitive)
• Use fw ls to browse available groups and projects
• Ensure you're logged into the correct Flywheel site with fw status

"Destination perm-check failed" (Permission Error)

Cause: Cannot write to the destination location (local filesystem, S3, or Google Cloud Storage).

Solution:

• Local filesystem: Verify you have write permissions for the destination directory
• S3: Check AWS credentials are configured correctly (aws configure)
• S3: Ensure the IAM user/role has write permissions to the bucket
• Google Cloud: Verify credentials with gcloud auth login
• Google Cloud: Ensure the service account has write permissions to the bucket

"Group and project level container tag filters are not allowed"

Cause: Attempted to use --include-container-tags or --exclude-container-tags with group or project level filters.

Solution:

• Container tag filters only work at subject, session, acquisition, analyses, and file levels
• Remove group/project from your tag filter specification
• Example of valid filter: --include-container-tags '{"subject": ["cohort1"]}'

"Analysis container filtering only works with --analyses flag"

Cause: Used analysis-level container tags without including the --analyses flag.

Solution:

• Add the --analyses flag to your command
• Or use --full-project which includes analyses
• Example: fw sync --analyses --include-container-tags '{"analyses": ["reviewed"]}' ...

"Could not extract file" / "Possibly not a valid zipfile"

Cause: Downloaded zip file is corrupted or incomplete.

Solution:

• Check your network connection stability
• Try syncing again (command will retry 5 times automatically for network errors)
• If error persists, the file may be corrupted in Flywheel - contact support
• Check available disk space at destination

S3/GCS Credential Errors

Cause: Cloud storage credentials are not configured or are invalid.

Solution:

• AWS S3: Configure credentials with aws configure or set environment variables
• AWS S3: See AWS credential configuration
• Google Cloud: Authenticate with gcloud auth login
• Google Cloud: See Google Cloud authentication
• Verify the credentials have read/write access to the specified bucket

See Also

Related Commands:

• Download Files - Download individual files or containers
• Export BIDS Data - Export data in BIDS format
• Import Data - Guide for uploading data to Flywheel

Data Management:

• Understanding Metadata - Learn about metadata preservation during sync
• File Types in Flywheel - Understanding file type filtering

Troubleshooting:

• CLI Troubleshooting Guide - Common CLI issues and solutions