Copy a Project using Ingest Project

  • Updated

In version (14.4.0), Flywheel introduced a new way to copy an entire Flywheel project to a new location. The ingest project command gives users the ability to copy project data in a scalable and reliable way.  This allows for streamlined multi-site collaboration as well helping workflows where data is curated on a pre-production Flywheel site before being moved to a production site.

The Ingest project CLI command can be used to copy data:

  • From one Flywheel site to a Flywheel different site

  • From one group to a different group within the same Flywheel site

Unsupported upload method for site, group, or project de-id profiles

Use one of these supported methods if you use site, group, or project de-id profiles.

Keywords

Source: The Flywheel site or project that contains the data you want to copy

Destination: The Flywheel site or project where you want to copy the data to

Steps

If you have a cluster, or are using Flywheel V3, we recommend using the cluster for a faster ingest. See this section for steps.

  1. Sign in to the Source. You must have create container and upload data permissions on the source project.
    The Source is the Flywheel site or Project that contains the data you want to copy.

  2. From the Source, copy your API key. You will need this later on. To find your API Key, go to your Profile and copy the API Key.

    FIndAPIKey.gif

  3. Sign in to the Destination . You must have download permissions on the destination project.
    If you are copying a project to a new group from within the same Flywheel site, your Destination and Source are the same.

  4. In the Destination, create the Group and Project where Flywheel will place the data.

  5. Open Terminal or Windows Command Prompt, and sign in to the Destination with the CLI. Learn more about how to download and install the CLI.
    To see where you are signed in, enter the fw status command. To login, copy and paste the command from your Flywheel Profile page.

  6. From the Flywheel CLI enter the following command:

    fw ingest project [optional commands] SRC FW_WALKER_API_KEY

    • Learn more about the optional flags for this command below

    • SRC: The routing string from the Source Group and Project. For example: fw://alphagroup/alphaProject

    • FW_WALKER_API_KEY: The API Key from the Source Flywheel site. For example: flywheel.io:yiLr8PHLtpe33IdYJ5

    For example:

    fw ingest project --group betagroup --project betaProject fw://alphagroup/alphaProject flywheel.io:yiLr8PHLtpe33IdYJ5

    If you do not specify the group and project by adding the --group and --project flags, Flywheel will look for a Group and Project that matches the Source.

  7. Flywheel begins scanning the Source and displays the results in the CLI.

    ingestProjectinProgress.png

  8. Review the summary, and enter yes.

    Flywheel ingest project command will copy the Source data and metadata, but not:

    • Job history/provenance

    • Analyses

    • Change log history

    • Project permissions

    • Gear Rules

    • Session templates

    • Data Views

  9. Once the scan is complete, sign in to the Destination to view.
    Existing containers are not modified in destination project, and any existing files are not copied over.

Considerations

  • Empty containers in the Source will not be copied to destination

  • The File origins in the Destination show that the data was uploaded by a user

  • Only one project can be migrated per CLI command

  • The source project is copied to destination. The source project is not modified or deleted and remains on the Source site.

Use a cluster for a faster ingest

We recommend using a cluster if you have one available. Flywheel sites on V3 have can utilize the same cluster that the Flywheel Site is using. Follow the steps below for more information.

Steps

  1. Sign in to the Source as a Site Admin.

    The Source is the Flywheel site or Project that contains the data you want to copy.

  2. From the Source, copy your API key. You will need this later on. To find your API Key, go to your Profile and copy the API Key.

    15fb30e4308b91.gif

  3. Sign in to the Destination as a Site Admin.

    If you are copying a project to a new group from within the same Flywheel site, your Destination and Source are the same.

  4. In the Destination, create the Group and Project where Flywheel will place the data.

  5. Open Terminal or Windows Command Prompt, and sign in to the Destination with the CLI. Learn more about how to download and install the CLI.

    To see where you are signed in, enter the fw status command. To login, copy and paste the command from your Flywheel Profile page.

  6. From the Flywheel CLI enter the following command:

    fw ingest project --cluster URL --follow SRC FW_WALKER_API_KEY

    • --cluster CLUSTER URL: Use the --cluster flag along with the cluster URL. If you are using a Flywheel V3 site, the URL to your cluster is https://yourflywheelURL.com/ingest.

      For example: https://university.flywheel.com/ingest

    • --follow: Follows the progress of the ingest

    • SRC: The routing string from the Source Group and Project. For example: fw://alphagroup/alphaProject

    • FW_WALKER_API_KEY: The API Key from the Source Flywheel site. For example: flywheel.io:333333333333IdYJ5

    For example:

    fw ingest project --cluster https://yourflywheelURL.com/ingest --follow --group beta_group --project beta_project fw://alphagroup/alphaProject flywheel.io:333333333333IdYJ5

    If you do not specify the group and project by adding the --group and --project flags, Flywheel will look for a Group and Project that matches the Source.

  7. Flywheel begins scanning the Source and displays the results in the CLI.

    cliIngestProjectProgress.png

  8. Review the summary, and enter yes.

    Note

    Flywheel ingest project command will copy the Source data and metadata, but not:

    • Job history/provenance

    • Analyses

    • Change log history

    • Project permissions

    • Gear Rules

    • Session templates

    • Data Views

  9. Once the scan is complete, sign in to the Destination to view.
    Existing containers are not modified in destination project, and any existing files are not copied over.

    Reuse this command

    You can reuse the command you created in these steps if you need to copy new data from the source in the future. Flywheel will only copy over the difference in the projects.

Command reference

 

Argument

Help Text

Positional arguments

SRC

The path to the folder to import (e.g.: fw://group-name/project-name/)

FW_WALKER_API_KEY

Your API key for the Source Flywheel site (e.g.: flywheel.io:yiLr8PHLtpe33IdYJ5)

Optional arguments

-h, --help

show this help message and exit

--compression-level COMPRESSION_LEVEL

The compression level to use for packfiles -1 by default. 0 for store. A higher compression level number means more compression.

--copy-duplicates

Upload duplicates found using --detect-duplicates to a sidecar project instead of skipping them (default: False)

--de-identify

De-identify DICOM files (default: False)

--deid-profile NAME

Use the De-identify profile by name

--detect-duplicates

Identify duplicate data conflicts within source data and duplicates between source data and data in Flywheel. Duplicates are skipped and noted in audit log (default: False)

In the audit log the error categories are: 

- `DD01` multiple files have the same destination path
- `DD02` file name already exists in destination container
- `DD03` a single item contains multiple StudyInstanceUIDs
- `DD04` a single session contains multiple StudyInstanceUIDs
- `DD05` multiple sessions have the same StudyInstanceUID
- `DD06` StudyInstanceUID already exists in a different session
- `DD07` a single acquisition contains multiple SeriesInstanceUIDs
- `DD08` multiple acquisitions have the same SeriesInstanceUID
- `DD09` a single item contains multiple SeriesInstanceUIDs
- `DD10` SeriesInstanceUID already exists in a different acquisition
- `DD11` SOPInstanceUID occurs multiple times (image uids should be unique)

--detect-duplicates-project DETECT_DUPLICATES_PROJECT [DETECT_DUPLICATES_PROJECT ...]

Specify one or multiple project paths to use for detecting duplicates

--enable-project-files

Enable file uploads to project container (default: False)

--encodings ENCODINGS [ENCODINGS ...]

Set character encoding aliases. E.g. win_1251=cp1251

--exclude PATTERN [PATTERN ...]

Patterns of filenames to exclude

--exclude-dirs PATTERN [PATTERN ...]

Patterns of directories to exclude

-g ID, --group ID

The id of the destination group (default comes from SRC)

--ignore-unknown-tags

Ignore unknown dicom tags when parsing dicom files (default: False)

--include PATTERN [PATTERN ...]

Patterns of filenames to include

--include-dirs PATTERN [PATTERN ...]

Patterns of directories to include

--load-subjects PATH

Load subjects from the specified file

--no-audit-log

Skip uploading audit log to the target projects (default: False)

-p LABEL, --project LABEL

The label of the destination project (default comes from SRC)

--require-project

Proceed with the ingest process only if the resolved group and project exists (default: False)

--skip-existing

Skip import of existing files (default: False)

--symlink

Follow symbolic links that resolve to directories (default: False)

General

-y, --yes

Assume the answer is yes to all prompts (default: False)

--ca-certs CA_CERTS

The file to use for SSL Certificate Validation

-C PATH, --config-file PATH

Specify configuration options via config file

-d, --debug

Turn on debug logging (default: False)

--no-config

Do NOT load the default configuration file (default: False)

-q, --quiet

Squelch log messages to the console (default: False)

--timezone TIMEZONE

Set the effective local timezone for imports

-v, --verbose

Get more detailed output (default: False)

Reporter

These config options are only available when using cluster mode with the --follow argument or when using a local worker

--save-audit-logs PATH

Save audit log to the specified path on the current machine

--save-deid-logs PATH

Save deid log to the specified path on the current machine

--save-subjects PATH

Save subjects to the specified file

Cluster

These config options apply when using a cluster to ingest data.

--cluster CLUSTER URL

Recruits resources from the cluster your V3 Flywheel site is installed on. Use when uploading a large amount of data.

Your cluster URL is the URL you use to sign in followed by /ingest. For example: https://university.flywheel.io/cluster

-f, --follow

Follow the progress of the ingest (default: False)

Worker

These config options are only available when using local worker (--cluster is not defined)

--jobs JOBS

The number of concurrent jobs to run (e.g. scan jobs)

--max-tempfile MAX_TEMPFILE

The max in-memory tempfile size, in MB, or 0 to always use disk

--sleep-time SECONDS

Number of seconds to wait before trying to get a task