In version (14.4.0), Flywheel introduced a new way to copy an entire Flywheel project to a new location. The ingest project command gives users the ability to copy project data in a scalable and reliable way. This allows for streamlined multi-site collaboration as well helping workflows where data is curated on a pre-production Flywheel site before being moved to a production site.
The Ingest project CLI command can be used to copy data:
-
From one Flywheel site to a Flywheel different site
-
From one group to a different group within the same Flywheel site
Source: The Flywheel site or project that contains the data you want to copy
Destination: The Flywheel site or project where you want to copy the data to
Steps
-
Sign in to the Source. You must have create container and upload data permissions on the source project.
The Source is the Flywheel site or Project that contains the data you want to copy. -
From the Source, copy your API key. You will need this later on. To find your API Key, go to your Profile and copy the API Key.
-
Sign in to the Destination . You must have download permissions on the destination project.
If you are copying a project to a new group from within the same Flywheel site, your Destination and Source are the same. -
In the Destination, create the Group and Project where Flywheel will place the data.
-
Open Terminal or Windows Command Prompt, and sign in to the Destination with the CLI. Learn more about how to download and install the CLI.
To see where you are signed in, enter thefw status
command. To login, copy and paste the command from your Flywheel Profile page. -
From the Flywheel CLI enter the following command:
fw ingest project [optional commands] SRC FW_WALKER_API_KEY
-
Learn more about the optional flags for this command below
-
SRC: The routing string from the Source Group and Project. For example:
fw://alphagroup/alphaProject
-
FW_WALKER_API_KEY: The API Key from the Source Flywheel site. For example:
flywheel.io:yiLr8PHLtpe33IdYJ5
For example:
fw ingest project --group betagroup --project betaProject fw://alphagroup/alphaProject flywheel.io:yiLr8PHLtpe33IdYJ5
If you do not specify the group and project by adding the
--group
and--project
flags, Flywheel will look for a Group and Project that matches the Source. -
-
Flywheel begins scanning the Source and displays the results in the CLI.
-
Review the summary, and enter
yes
.Flywheel ingest project command will copy the Source data and metadata, but not:
-
Job history/provenance
-
Analyses
-
Change log history
-
Project permissions
-
Gear Rules
-
Session templates
-
Data Views
-
-
Once the scan is complete, sign in to the Destination to view.
Existing containers are not modified in destination project, and any existing files are not copied over.
-
Empty containers in the Source will not be copied to destination
-
The File origins in the Destination show that the data was uploaded by a user
-
Only one project can be migrated per CLI command
-
The source project is copied to destination. The source project is not modified or deleted and remains on the Source site.
We recommend using a cluster if you have one available. Flywheel sites on V3 have can utilize the same cluster that the Flywheel Site is using. Follow the steps below for more information.
-
Sign in to the Source as a Site Admin.
The Source is the Flywheel site or Project that contains the data you want to copy.
-
From the Source, copy your API key. You will need this later on. To find your API Key, go to your Profile and copy the API Key.
-
Sign in to the Destination as a Site Admin.
If you are copying a project to a new group from within the same Flywheel site, your Destination and Source are the same.
-
In the Destination, create the Group and Project where Flywheel will place the data.
-
Open Terminal or Windows Command Prompt, and sign in to the Destination with the CLI. Learn more about how to download and install the CLI.
To see where you are signed in, enter the
fw status
command. To login, copy and paste the command from your Flywheel Profile page. -
From the Flywheel CLI enter the following command:
fw ingest project --cluster URL --follow SRC FW_WALKER_API_KEY
-
--cluster CLUSTER URL: Use the --cluster flag along with the cluster URL. If you are using a Flywheel V3 site, the URL to your cluster is
https://yourflywheelURL.com/ingest
.For example:
https://university.flywheel.com/ingest
-
--follow: Follows the progress of the ingest
-
SRC: The routing string from the Source Group and Project. For example:
fw://alphagroup/alphaProject
-
FW_WALKER_API_KEY: The API Key from the Source Flywheel site. For example:
flywheel.io:333333333333IdYJ5
For example:
fw ingest project --cluster https://yourflywheelURL.com/ingest --follow --group beta_group --project beta_project fw://alphagroup/alphaProject flywheel.io:333333333333IdYJ5
If you do not specify the group and project by adding the
--group
and--project
flags, Flywheel will look for a Group and Project that matches the Source. -
-
Flywheel begins scanning the Source and displays the results in the CLI.
-
Review the summary, and enter
yes
.Note
Flywheel ingest project command will copy the Source data and metadata, but not:
-
Job history/provenance
-
Analyses
-
Change log history
-
Project permissions
-
Gear Rules
-
Session templates
-
Data Views
-
-
Once the scan is complete, sign in to the Destination to view.
Existing containers are not modified in destination project, and any existing files are not copied over.
Argument |
Help Text |
Positional arguments |
|
SRC |
The path to the folder to import (e.g.: fw://group-name/project-name/) |
FW_WALKER_API_KEY |
Your API key for the Source Flywheel site (e.g.: flywheel.io:yiLr8PHLtpe33IdYJ5) |
Optional arguments |
|
-h, --help |
show this help message and exit |
--compression-level COMPRESSION_LEVEL |
The compression level to use for packfiles -1 by default. 0 for store. A higher compression level number means more compression. |
--copy-duplicates |
Upload duplicates found using --detect-duplicates to a sidecar project instead of skipping them (default: False) |
--de-identify |
De-identify DICOM files (default: False) |
--deid-profile NAME |
Use the De-identify profile by name |
--detect-duplicates |
Identify duplicate data conflicts within source data and duplicates between source data and data in Flywheel. Duplicates are skipped and noted in audit log (default: False) - `DD01` multiple files have the same destination path |
--detect-duplicates-project DETECT_DUPLICATES_PROJECT [DETECT_DUPLICATES_PROJECT ...] |
Specify one or multiple project paths to use for detecting duplicates |
--enable-project-files |
Enable file uploads to project container (default: False) |
--encodings ENCODINGS [ENCODINGS ...] |
Set character encoding aliases. E.g. win_1251=cp1251 |
--exclude PATTERN [PATTERN ...] |
Patterns of filenames to exclude |
--exclude-dirs PATTERN [PATTERN ...] |
Patterns of directories to exclude |
-g ID, --group ID |
The id of the destination group (default comes from SRC) |
--ignore-unknown-tags |
Ignore unknown dicom tags when parsing dicom files (default: False) |
--include PATTERN [PATTERN ...] |
Patterns of filenames to include |
--include-dirs PATTERN [PATTERN ...] |
Patterns of directories to include |
--load-subjects PATH |
Load subjects from the specified file |
--no-audit-log |
Skip uploading audit log to the target projects (default: False) |
-p LABEL, --project LABEL |
The label of the destination project (default comes from SRC) |
--require-project |
Proceed with the ingest process only if the resolved group and project exists (default: False) |
--skip-existing |
Skip import of existing files (default: False) |
--symlink |
Follow symbolic links that resolve to directories (default: False) |
General |
|
-y, --yes |
Assume the answer is yes to all prompts (default: False) |
--ca-certs CA_CERTS |
The file to use for SSL Certificate Validation |
-C PATH, --config-file PATH |
Specify configuration options via config file |
-d, --debug |
Turn on debug logging (default: False) |
--no-config |
Do NOT load the default configuration file (default: False) |
-q, --quiet |
Squelch log messages to the console (default: False) |
--timezone TIMEZONE |
Set the effective local timezone for imports |
-v, --verbose |
Get more detailed output (default: False) |
Reporter These config options are only available when using cluster mode with the --follow argument or when using a local worker |
|
--save-audit-logs PATH |
Save audit log to the specified path on the current machine |
--save-deid-logs PATH |
Save deid log to the specified path on the current machine |
--save-subjects PATH |
Save subjects to the specified file |
Cluster These config options apply when using a cluster to ingest data. |
|
--cluster CLUSTER URL |
Recruits resources from the cluster your V3 Flywheel site is installed on. Use when uploading a large amount of data. Your cluster URL is the URL you use to sign in followed by /ingest. For example: https://university.flywheel.io/cluster |
-f, --follow |
Follow the progress of the ingest (default: False) |
Worker These config options are only available when using local worker (--cluster is not defined) |
|
--jobs JOBS |
The number of concurrent jobs to run (e.g. scan jobs) |
--max-tempfile MAX_TEMPFILE |
The max in-memory tempfile size, in MB, or 0 to always use disk |
--sleep-time SECONDS |
Number of seconds to wait before trying to get a task |