Migrating Project Data between Flywheel Sites

This article explains how to migrate Project data from one Flywheel site to another.

Applicability

The options presented in this article only support migrating the following data:

• Container Hierarchy
• Files
• Metadata

Other data, including (but not limited to) the items listed below, cannot be migrated using any of the options presented in this article:

• Gear Job History / Provenance
• Analyses
• Change Log History
• Project Permissions
• Gear Rules
• Session Templates
• Data Views

Overview

There are two recommended options for migrating projects between Flywheel sites:

1. Direct copy without any interim storage
2. Export/Import using interim storage

Direct Copy without Interim Storage

The ingest project command from the legacy CLI is designed for copying project data directly from one Flywheel site to another without needing to write data to an interim storage location. For more information about this option, refer to the fw ingest project documentation.

Specifically,

• If the --cluster option is not used, then all of the processing will occur on the local client machine where the fw ingest project CLI command was invoked (i.e., the user's local computer). This means,
  • The local client machine requires direct connectivity to both the source and destination Flywheel sites.
  • All of the data will be routed through the local client machine on its way between sites. Depending upon network layout and speeds, this could significantly slow down the transfer.
• If the --cluster option is used, then all of the processing will occur on the the destination Flywheel site. This means,
  • The destination Flywheel site needs direct network connectivity to the source Flywheel site (one way).
  • The data will be routed directly from one site to another without (i.e., not through the local client machine). Depending upon network layout and speeds, this could significantly speed up the transfer.

Export/Import using Interim Storage

The Bulk Export and Bulk Import features are designed to work together, so that an exported project can be subsequently imported with minimal (or no) changes required.

However, there are a few nuances that need to be accounted for:

1. Bulk Import does not create groups or projects.
2. Bulk Export creates a top-level folder representing the project, but Bulk Import does not expect this folder to exist.
3. Bulk Import cannot currently handle multiple DICOM series in the same source folder, but Flywheel Core allows data to be organized this way and Bulk Export preserves it exactly as-is.

To account for #1:

• Create the group and project in the destination site in advance.

To account for #2:

• Override the External Storage configuration when running Bulk Import to target the project folder instead of its parent.
  • For example, set --storage-config 'prefix: "/ACME Trial"' where ACME Trial is the project name.
  • Note that with Bulk Import the External Storage configuration can currently only be overridden via the new (BETA) CLI and not via the Web App.
  • Although this could also be accomplished using an inclusion filter rule (e.g., --include "/ACME Trial"), this approach can be far slower since the import process will still scan all the other folders before applying the filter rules.

To account for #3:

• Modify the Bulk Export configuration to insert one additional folder level for each individual file into the destination: --path '{project.label}/{subject.label}/{session.label}/{acquisition.label}/{file.name}/{file.name}'
  • This results in each DICOM Series being fully contained within its own folder upon export.
• Then, modify the Bulk Import mapping rule to ignore the additional "file name" folders at import time: --mapping 'path={subject.label}/{session.label}/{acquisition.label}/*/*'

Example Script

Putting it all together, a single script using the new (BETA) CLI to first export and then subsequently import a project from one Flywheel site to another could look like this

1
2
3
4
5
6
7

export FW_GROUP_NAME="ACME Research Group"
export FW_PROJ_NAME="ACME Trial"
export FW_STORAGE_ID="67ab8fabcdeff4a87775a532"

fw-beta --profile site1 export run --storage $FW_STORAGE_ID --project 'fw://$FW_GROUP_NAME/$FW_PROJ_NAME' --path '{project.label}/{subject.label}/{session.label}/{acquisition.label}/{file.name}/{file.name}'

fw-beta --profile site2 import run --storage $FW_STORAGE_ID --project 'fw://$FW_GROUP_NAME/$FW_PROJ_NAME' --storage-config 'prefix: "/$FW_PROJ_NAME"' --mapping 'path={subject.label}/{session.label}/{acquisition.label}/*/*'

Note that the following pre-requisites need to be completed before running this script:

1. Configure login profiles for site1 and site2 in the new (BETA) CLI. Refer to the new (BETA) CLI documentation on Profiles for more details.
   • Alternatively, run the export and import commands separately with a logout and login in between.
2. Change the FW_GROUP_NAME and FW_PROJ_NAME variables at the beginning of the script to match your group and project.
3. Register an External Storage for use as the interim storage location and change the FW_STORAGE_ID variable at the beginning of the script to match its assigned External Storage ID.
4. Create the corresponding group and project in the destination Flywheel site (site2) before running the script.