Rule Files

Rule files provide a way to define complex import and export rules in a reusable YAML format. They are especially useful when command-line arguments become unwieldy or when you need multiple sets of rules with first-match-wins logic.

When to Use Rule Files

Rule files are ideal for:

Complex rule sets - When command-line options become too long or difficult to manage
Reusable configurations - Save and reuse rule sets across multiple import/export jobs
Multiple rule priorities - Define multiple rules where the first matching rule is applied
Depth-based logic - Apply different rules based on directory depth or file location
Team sharing - Share standardized import/export configurations with your team
Version control - Track changes to import/export logic over time

Understanding Rule Options

Rule files use the same filtering, mapping, and configuration options available via command-line arguments.

Before creating rule files, review:

For Import Rules: Bulk Import - Customizing Import Rules
For Export Rules: Bulk Export Settings
For Pattern Syntax: Pattern Syntax Reference

Basic Structure

Rule files are YAML documents containing a list of rule objects. Each rule is evaluated in order, and the first rule that matches a file is applied.

Import Rule File

- level: acquisition
  include:
    - 'ext=dcm'
  exclude:
    - 'name=~*temp*'
  mappings:
    - 'path={sub}/{ses}/{acq}/*'
    - 'subject.label={PatientID}'
  type: dicom

Export Rule File

- level: acquisition
  include:
    - 'file.type=nifti'
  exclude:
    - 'session.label=~test'
  path: '{project.label}/{subject.label}/{session.label}/{file.name}'
  unzip: true

Rule Structure

Each rule in a rule file is a YAML object containing:

level (required) - Container level: project, subject, session, or acquisition
Filter fields (include, exclude) - Control which files match this rule
Configuration fields - Vary between import and export rules

Import Rule Structure

Import rules support the following field groups:

Filters: include, exclude
Mappings: mappings, defaults, overrides
Processing: type, zip, zip_single, group_by
DICOM-specific: dicom_instance_name, dicom_group_by, dicom_split_localizer

For detailed field descriptions and examples, see Bulk Import - Customizing Import Rules.

Export Rule Structure

Export rules support the following field groups:

Filters: include, exclude
Path template: path
Extraction: unzip, unzip_path
Metadata: metadata

For detailed field descriptions and examples, see Bulk Export Settings

Using Rule Files

Import with Rule File

# Using a rule file for import
fw-beta import run \
    --project "fw://group/project" \
    --storage storage-id \
    --rules-file /path/to/import-rules.yaml

Export with Rule File

# Using a rule file for export
fw-beta export run \
    --project "fw://group/project" \
    --storage storage-id \
    --rules-file /path/to/export-rules.yaml

Rule Evaluation Order

Rules are evaluated in the order they appear in the file. The first rule that matches a file is applied. This allows you to handle different file types or directory structures with different logic.

# Rule 1: Handle DICOM files at depth 4
- level: acquisition
  include:
    - 'depth=4'
    - 'ext=dcm'
  mappings:
    - 'path={sub}/{ses}/{acq}/*'
  type: dicom

# Rule 2: Handle NIfTI files at depth 3
- level: acquisition
  include:
    - 'depth=3'
    - 'ext=nii'
  mappings:
    - 'path={sub}/{ses}/*'

# Rule 3: Catch-all for other files
- level: acquisition
  include:
    - 'depth<=5'

Real-World Examples

Example 1: Multi-Depth Research Data Import

Complex import rules handling different directory structures based on depth:

# Project-level documentation files
- level: project
  exclude:
    - 'path=~.DS_Store'
    - 'path=~**/backup/*'
  include:
    - 'depth=1'
    - 'name=~README*'
    - 'name=~*.pdf'
  mappings:
    - 'path=[*/]{file}'

# Deep hierarchy: subject/session/acquisition structure (depth 4)
- level: acquisition
  exclude:
    - 'path=~.DS_Store'
    - 'path=~**/backup/*'
    - 'depth=1'
  include:
    - 'depth=4'
    - 'ext=dcm'
  mappings:
    - 'path={sub}/{ses}/{acq}/*'
    - 'subject.label={PatientID}'
    - 'session.label={StudyDate}'
  type: dicom
  defaults:
    - 'acquisition.info.import_rule=depth4_dicom'

# Shallow hierarchy: subject/acquisition structure (depth 3)
- level: acquisition
  exclude:
    - 'path=~.DS_Store'
    - 'path=~**/backup/*'
    - 'depth=1'
  include:
    - 'depth=3'
  mappings:
    - 'path={sub}/{acq}/*'
  defaults:
    - 'session.label=baseline'
    - 'acquisition.info.import_rule=depth3_baseline'

Example 2: Structured Data Export

Export data with separate rules for different file types and metadata:

# Project documentation files
- level: project
  include:
    - 'file.name=~README*'
    - 'file.name=~CHANGES*'
    - 'file.name=~*.pdf'
    - 'file.info.documentation=true'
  exclude:
    - 'file.info.exclude=true'
  path: '{project.label}/docs/{file.name}'

# Processed imaging data (NIfTI and metadata)
- level: acquisition
  include:
    - 'file.type=nifti'
    - 'file.name=~*.json'
    - 'file.info.processed=true'
  exclude:
    - 'file.type=dicom'
    - 'session.label=~test*'
  path: '{project.label}/data/{subject.label}/{session.label}/{acquisition.label}/{file.name}'
  metadata: true

# Raw DICOM source data (keep zipped)
- level: acquisition
  include:
    - 'file.type=dicom'
    - 'file.name=~*.dicom.zip'
  exclude:
    - 'file.info.exclude=true'
    - 'session.label=~test*'
  path: '{project.label}/sourcedata/{subject.label}/{session.label}/{acquisition.label}/{file.name}'
  unzip: false

Example 3: Multi-Site Clinical Study

Different import logic based on directory structure:

# Site A: DICOM organized by StudyDate
- level: acquisition
  include:
    - 'path=~SiteA/*'
    - 'ext=dcm'
  mappings:
    - 'path=SiteA/{sub}/{StudyDate}/{acq}/*'
    - 'subject.label={PatientID}'
    - 'session.label={StudyDate}'
  type: dicom
  defaults:
    - 'subject.cohort=SiteA'

# Site B: NIfTI organized by subject/timepoint
- level: acquisition
  include:
    - 'path=~SiteB/*'
    - 'ext=nii'
  mappings:
    - 'path=SiteB/{sub}/{ses}/{acq}/*'
  defaults:
    - 'subject.cohort=SiteB'

# Site C: Mixed data with metadata JSON
- level: acquisition
  include:
    - 'path=~SiteC/*'
  exclude:
    - 'name=~*metadata.json'
  mappings:
    - 'path=SiteC/{sub}/{ses}/{acq}/*'
  defaults:
    - 'subject.cohort=SiteC'

Best Practices

Rule Organization

Order by specificity - Place more specific rules before general ones
Use comments - Document the purpose of each rule
Group related rules - Keep rules for similar data types together
Test incrementally - Test each rule addition with a small dataset

Performance Optimization

Minimize rules - Combine similar rules when possible
Efficient filters - Use exclude filters to skip files early
Depth-based filtering - Use depth filters to quickly categorize files

Maintainability

Version control - Store rule files in git alongside your analysis code
Document assumptions - Add comments explaining data structure expectations
Use descriptive names - Name rule files clearly (e.g., bids-import-rules.yaml)
Test with dry-run - Always test rule changes with --dry-run first

Troubleshooting

Files Not Matching Any Rules

If files aren't being processed, check:

Rule order - Is a more general rule matching first?
Filter syntax - Are your include/exclude patterns correct?
Depth filters - Does the depth match your directory structure?
Field availability - Are you using fields available for import vs export?

Use fw-beta import test to see how a single file matches your rules:

fw-beta import test /path/to/file.dcm \
    --rules-file /path/to/rules.yaml

Debugging Rule Evaluation

To understand which rule matches a file:

Add explicit depth filters to each rule for clarity
Use the --dry-run flag to see what would happen without actually transferring files
Check the import/export report for details on how each file was processed