Bulk Import - Customizing Import Rules

Overview

While Flywheel provides automatic import behavior that works well for many scenarios, you may need to customize how files are organized during import.

The new CLI provides extensive options for customizing:

Filtering - Control which files are imported
Mapping - Define how files are organized into Flywheel's hierarchy
Grouping - Control how DICOM files are packaged into ZIP archives
File Naming - Customize ZIP archive and individual file names

All customization options are available when using the flyw import run command.

Pattern Syntax Documentation

For comprehensive pattern syntax documentation and practical examples, see:

Filtering and Mapping Guide - Practical examples and common use cases
Pattern Quick Reference - Quick lookup for common patterns
Pattern Syntax Reference - Complete syntax documentation
Rule Files - Define complex rules in reusable YAML format

Scan Parameters

When importing from a cloud storage location that contains a very large number of items (for example, millions of files in a single storage prefix), you can use server-side prefix filtering to improve performance. Rather than listing all items and filtering client-side, prefix filtering tells the system to scan only the specified paths.

Prefix Filtering via the CLI

Use the --scan-params-prefix option to specify one or more path prefixes:

# Scan only specific prefixes
flyw import run -s <storage> -p <project> \
    --scan-params-prefix 'site-a/imaging' \
    --scan-params-prefix 'site-b/imaging'

Multiple --scan-params-prefix values can be provided in a single import run. Each prefix is scanned independently, and the results are combined.

Prefix Filtering via Rule Files

Prefixes can also be defined in a rule file or managed rule set using the scan_params.prefixes field:

- level: acquisition
  include:
    - 'ext=dcm'
  mappings:
    - 'path={sub}/{ses}/{acq}/*'
  type: dicom
  scan_params:
    prefixes:
      - site-a/imaging
      - site-b/imaging

The rule file approach is recommended for recurring imports because it keeps the prefix configuration alongside the rest of the import rules.

Filtering Files

Filters control which files are included or excluded from import based on file properties like name, size, path, or depth.

Include Filters

Include filters specify which files should be processed. At least one include filter must match for a file to be imported.

# Include only DICOM files
flyw import run -s <storage> -p <project> \
    --include 'ext=dcm'

# Include multiple file types
flyw import run -s <storage> -p <project> \
    --include 'ext=dcm' \
    --include 'ext=nii'

# Include files at specific depth
flyw import run -s <storage> -p <project> \
    --include 'depth=4'

Exclude Filters

Exclude filters specify which files should be skipped. If any exclude filter matches, the file is not imported.

# Exclude temporary files
flyw import run -s <storage> -p <project> \
    --exclude 'name=~*temp*' \
    --exclude 'name=~*backup*'

# Exclude by size
flyw import run -s <storage> -p <project> \
    --exclude 'size<1MB'

Combining Include and Exclude

# Import DICOM files but exclude localizers
flyw import run -s <storage> -p <project> \
    --include 'ext=dcm' \
    --exclude 'name=~*localizer*' \
    --exclude 'name=~*scout*'

For more filtering examples and available fields, see the Filtering and Mapping Guide.

Mapping Files to Flywheel Hierarchy

Mapping patterns define how source file paths are transformed into Flywheel's hierarchy structure (Subject, Session, Acquisition) and how container labels are derived.

Path-Based Mapping

Extract container labels from file path structure:

# Map directory structure to Flywheel hierarchy
flyw import run -s <storage> -p <project> \
    --mapping 'path={sub}/{ses}/{acq}/*'

# Skip root directory levels
flyw import run -s <storage> -p <project> \
    --mapping 'path=data/{sub}/{ses}/{acq}/*'

DICOM Metadata Mapping

For DICOM files, derive container labels from DICOM headers:

# Use DICOM tags for container labels
flyw import run -s <storage> -p <project> \
    --type dicom \
    --mapping 'subject.label={PatientID}' \
    --mapping 'session.label={StudyDescription}' \
    --mapping 'acquisition.label={SeriesDescription}'

# Combine multiple DICOM tags
flyw import run -s <storage> -p <project> \
    --type dicom \
    --mapping 'session.label={StudyDescription}_{StudyDate}' \
    --mapping 'acquisition.label={Modality}_{SeriesNumber}'

Default Values

Set default values when mappings don't extract a value:

flyw import run -s <storage> -p <project> \
    --mapping 'subject.label={PatientID}' \
    --default 'session.label=baseline'

Override Values

Force specific values regardless of extracted data:

flyw import run -s <storage> -p <project> \
    --mapping 'subject.label={PatientID}' \
    --override 'project.label=MyStudy'

For more mapping examples and available DICOM fields, see the Filtering and Mapping Guide.

Per-File Values from a Manifest

Mapping rules derive values from patterns that apply to many files at once. When values cannot be derived from paths or headers, supply them file by file in a manifest file.

Rules and a manifest work together in the same import:

flyw import run -s <storage> -p <project> \
    --rule-set <rule-set> \
    --manifest dataset1-manifest.csv

Each part of a rule plays a different role, and the manifest affects only one of them:

Filters (include and exclude) are applied while Flywheel scans the source location. They decide which files are imported, and the matching rule sets each file's hierarchy level. The manifest is not used at this stage, so it cannot bring in a file that the filters skip.
Grouping of files into archives, such as the files of a DICOM series, also happens without the manifest.
Mappings, including defaults and overrides, derive each file's labels and metadata. This is the only stage where the manifest applies. For each file the manifest lists, its non-empty values replace the values that the mappings derived, including values set by overrides. For files the manifest does not list, and for fields that a manifest row leaves empty, the mappings' values are used.

In other words, the filters always apply, and for the files they select, manifest entries take precedence over the mappings, which act as the fallback. Keep as much as possible in the rules, and use the manifest only for what the rules cannot derive. See Bulk Import - Default Behavior for how values are derived when no manifest is supplied.

DICOM-Specific Customizations

Custom Grouping Logic

Control how DICOM files are grouped into ZIP archives using the --dicom-group-by option:

# Group by study and series (default behavior)
flyw import run -s <storage> -p <project> \
    --type dicom \
    --dicom-group-by StudyInstanceUID \
    --dicom-group-by SeriesInstanceUID

# Custom grouping by date and series number
flyw import run -s <storage> -p <project> \
    --type dicom \
    --dicom-group-by StudyDate \
    --dicom-group-by StudyTime \
    --dicom-group-by SeriesNumber

Disable Zipping Single Files

By default, even single DICOM files are zipped. Use --no-zip-single to disable this:

flyw import run -s <storage> -p <project> \
    --type dicom \
    --no-zip-single

Configuring File Names

Control the naming of files during import using the file.name mapping field. This applies to both ZIP archives created from DICOM files and non-DICOM files:

# Custom ZIP archive naming for DICOM
flyw import run -s <storage> -p <project> \
    --type dicom \
    --mapping 'file.name={Modality}_{SeriesNumber}_{SeriesDescription}.dicom.zip'

# Rename non-DICOM files during import
flyw import run -s <storage> -p <project> \
    --mapping 'file.name={PatientID}_{file.name}'

Configuring Individual DICOM File Names

Control how individual DICOM files are named within ZIP archives using --dicom-instance-name:

# Custom naming for individual DICOM files in archives
flyw import run -s <storage> -p <project> \
    --type dicom \
    --dicom-instance-name '{SeriesDescription}_{InstanceNumber}.{Modality}.dcm'

Note

The --dicom-instance-name option only affects DICOM files packaged into ZIP archives. To customize the names of loose DICOM files when --no-zip-single is used, use the file.name mapping field instead (see Configuring File Names).

Using Rule Sets or Rule Files

For complex import scenarios with multiple sets of rules, you have two options:

Managed Rule Sets (Recommended)

Use a centrally managed import rule set stored in Flywheel. Rule sets can be shared across your organization, are automatically versioned, and provide a full audit trail.

# Use a managed rule set by ID
flyw import run -s <storage> -p <project> \
    --rule-set <rule-set-id>

See Import and Export Rule Sets for details on creating and managing rule sets.

Local Rule Files

Alternatively, define rules in a local YAML file:

# Use a local rule file
flyw import run -s <storage> -p <project> \
    --rule-set /path/to/import-rules.yaml

Rule files support:

Multiple rules with first-match-wins logic
All filtering and mapping options
DICOM-specific configurations
Reusable configurations across imports

See Rule Files for complete documentation on the YAML format and examples.

Complete Example

Here's a comprehensive example combining multiple customization options:

flyw import run \
    --project "fw://group/MyProject" \
    --storage my-storage-id \
    --type dicom \
    --include 'ext=dcm' \
    --include 'size>1MB' \
    --exclude 'name=~*localizer*' \
    --exclude 'name=~*scout*' \
    --mapping 'subject.label={PatientID}' \
    --mapping 'session.label={StudyDescription}_{StudyDate}' \
    --mapping 'acquisition.label={SeriesNumber}_{Modality}_{SeriesDescription}' \
    --dicom-group-by StudyInstanceUID \
    --dicom-group-by SeriesInstanceUID \
    --dicom-instance-name '{SeriesNumber}_{InstanceNumber}.{Modality}.dcm'

This command:

Imports DICOM files larger than 1MB
Excludes localizer and scout images
Derives container labels from DICOM headers
Groups files by study and series
Customizes individual DICOM file naming within archives

Testing Your Configuration

Before running a full import, test your configuration using:

# Test on a single file
flyw import test /path/to/sample.dcm \
    --type dicom \
    --include 'ext=dcm' \
    --mapping 'subject.label={PatientID}'

# Dry-run mode (processes small subset without actual import)
flyw import run \
    --project "fw://group/MyProject" \
    --storage my-storage-id \
    --dry-run