CLI command reference: Ingest folder

  • Updated

The ingest folder command should be used if you are importing data that you previously downloaded from Flywheel, or if you are importing data that was shared from another Flywheel site. The folder import command uses the folder structure to map your data to the Flywheel hierarchy. Learn how to run the ingest folder command.

Below is the basic Flywheel hierarchy:

hierarchyfolders.png

Usage

fw ingest folder [SRC] [optional arguments]

Required arguments

Required argument

Description

SRC

The parent folder for the directory you want to import

Optional arguments

Tip

If you are using multiple optional arguments in your command, consider creating a config file. See our article for more information on creating and using a config file.

Optional argument

Description

-h, --help

show this help message and exit

-g ID, --group ID

The id of the group, if not in folder structure 

-p LABEL, --project LABE

The label of the project, if not in folder structure

--dicom NAME

The name of dicom subfolders to be zipped prior to upload

--pack-acquisitions TYPE

Acquisition folders only contain acquisitions of TYPE and are zipped prior to upload

--root-dirs ROOT_DIRS

The number of directories to discard before matching

--no-subjects

no subject level (create a subject for every session)

--no-sessions

no session level (create a session for every subject)

--group-override ID

Force using this group id

--project-override LABEL

Force using this project label

--include-dirs PATTERN

Patterns of directories to include

Note

When S3 bucket is configured as source, this flag does not support regex wildcard match, only “starts with.”

Include-dirs: OHM/101-10

Will match:OHM/101-105, OHM/101-106, etc

--exclude-dirs PATTERN

Patterns of directories to exclude

--include PATTERN

Patterns of filenames to include

--exclude PATTERN

Patterns of filenames to exclude

--compression-level COMPRESSION_LEVEL

The compression level to use for packfiles -1 by      default. 0 for store. A higher compression level  number means more compression

--ignore-unknown-tags

Ignore unknown dicom tags when parsing dicom files

--encodings ENCODINGS

Set character encoding aliases. E.g. win_1251=cp1251

--de-identify

De-identify DICOM files

--deid-profile NAME

Use the De-identify profile by name. Learn more about how to create a de-id profile.

--skip-existing

Skip import of existing files

--no-audit-log

Skip uploading audit log to the target projects

--load-subjects PATH

Load subjects from the specified file

--detect-duplicates

Identify duplicate data conflicts within source data  and duplicates between source data and data in Flywheel. Duplicates are skipped and noted in audit log

In the audit log the error categories are:

- `DD01` multiple files have the same destination path
- `DD02` file name already exists in destination container
- `DD03` a single item contains multiple StudyInstanceUIDs
- `DD04` a single session contains multiple StudyInstanceUIDs
- `DD05` multiple sessions have the same StudyInstanceUID
- `DD06` StudyInstanceUID already exists in a different session
- `DD07` a single acquisition contains multiple SeriesInstanceUIDs
- `DD08` multiple acquisitions have the same SeriesInstanceUID
- `DD09` a single item contains multiple SeriesInstanceUIDs
- `DD10` SeriesInstanceUID already exists in a different acquisition
- `DD11` SOPInstanceUID occurs multiple times (image uids should be unique)

--require-project

Proceed with the ingest process only if the resolved
group and project exists (default: False)

Note

A new group/project will be created if the resolved group/project do not exist and this command is not invoked. 

--enable-project-files

Enable file uploads to project container (default: False)

Reporter

These config options are only available when using cluster mode with the  --follow argument or when using local worker.

Optional argument

Description

--save-audit-logs PATH

Save audit log to the specified path on the current machine

--save-deid-logs PATH

Save deid log to the specified path on the current machine

--save-subjects PATH

Save subjects to the specified file

Cluster config

These config options apply when using a  cluster to ingest data.

Optional argument

Description

--cluster CLUSTER URL

Recruits resources from the cluster your V3 Flywheel site is installed on. Use when uploading a large amount of data.

Your cluster URL is the URL you use to sign in followed by /ingest. For example: https://university.flywheel.io/cluster

-f, --follow

Follow the progress of the ingest (default: False)

Worker config

These config options are only available when using local worker (--cluster is not defined)

Optional argument

Description

--jobs JOBS

The number of concurrent jobs to run (e.g. scan jobs),  ignored when using cluster (default: 4)

--sleep-time SECONDS

Number of seconds to wait before trying to get a task  (default: 1)

--max-tempfile MAX_TEMPFILE

The max in-memory tempfile size, in MB, or 0 to always use disk (default: 50)The max in-memory tempfile size, in MB, or 0 to always use disk (default: 50)

General

Optional argument

Description

-C PATH, --config-file

Specify configuration options via config file. Learn more about how to create this file.

--no-config

Do NOT load the default configuration file

-y, --yes

Assume the answer is yes to all prompts

--ca-certs CA_CERTS

Set the effective local timezone for imports

--timezone TIMEZONE

Set the effective local timezone for imports

-q, --quiet

Squelch log messages to the console

-d, --debug

Turn on debug logging

-v, --verbose

Get more detailed output

 Related articles