Skip to content

Create an Ingest Template

This guide walks you through creating a template file to map your dataset structure to Flywheel's project hierarchy.

Before you begin: If you are new to templates, read Understanding Ingest Templates to learn when and why to use them.

Prerequisites

  • Flywheel CLI installed and configured (installation guide)
  • A plain text editor (TextEdit, Notepad, Sublime, VS Code, etc.)
  • Your dataset organized in folders on your local machine
  • Understanding of your dataset's folder structure and naming conventions

Before You Begin

Ensure you have:

CLI Setup:

  • Flywheel CLI installed on your computer
  • Valid API key and authenticated CLI - verify with fw status
  • Stable internet connection for data upload

Permissions:

You need the following project-level permissions in the target group/project:

  • Single File Upload/Create or Bulk File Upload - Required for CLI ingest commands
  • Create Container - Required if creating new subjects, sessions, or acquisitions

To check your permissions, navigate to the project in the web UI and check your role. Learn more about user roles and permissions.

Data and Template Requirements:

  • Dataset organized in consistent folder structure on your local machine
  • Understanding of your dataset's folder naming patterns and hierarchy
  • Template file saved as .yaml (e.g., my-template.yaml)
  • For DICOM data: Files must be valid DICOM format
  • For non-DICOM data: Specify appropriate packfile_type in template (e.g., nifti, zip)

Template Validation:

Before running ingest:

  • Validate YAML syntax at yamllint.com
  • Test template on a small subset of data first
  • Use --dry-run flag to preview what will be imported without uploading

For PHI/Sensitive Data:

If your data contains Protected Health Information (PHI), configure de-identification before upload:

For Automated Processing (Optional):

To automatically process data as it arrives, set up gear rules before uploading:

  • Configure gear rules to run analysis automatically on new data
  • Test rules on a small dataset first to verify they work as expected

Step 1: Start with a Basic Template

Every template must define labels for subjects, sessions, and acquisitions. Create a new file in your text editor and start with this basic structure:

1
2
3
4
5
template:
  - pattern: "{subject}"
  - pattern: "{session}"
  - pattern: "{acquisition}"
      scan: dicom

This is the simplest template for uploading DICOM files with a three-level folder structure.

Step 2: Define the First Pattern

The first - pattern: line maps your dataset's top-level folder. Choose one of these approaches:

Option A: Use entire folder name as label

1
2
template:
  - pattern: "{subject}"

Example:

Folder name Result
sub-01 Subject label: sub-01

Option B: Extract part of folder name

1
2
template:
  - pattern: "{subject}-subject-Nov-2021"

Example:

Folder name Result
001-subject-Nov-2021 Subject label: 001
002-subject-Nov-2021 Subject label: 002

Option C: Add custom metadata from folder name

1
2
template:
  - pattern: "001-subject-{session.info.dataset}"

Example:

Folder name Result
001-subject-Nov-2021 Adds metadata dataset: Nov-2021 to session

Option D: Skip this folder level

1
2
template:
  - pattern: .*

Use regex .* to skip a folder level and move to the next.

Option E: Extract from filename

1
2
3
4
5
template:
  - pattern: "{subject}"
      scan:
      name: filename
      pattern: "(?P<acquisition>)[^.]*"

Example:

Filename Result
SAG T1.dicom Acquisition label: SAG T1

Tip

Use angled brackets <> instead of curly brackets {} when using regex to assign variables. This is a Python regex requirement.

Step 3: Add Patterns for Subfolders

Add a - pattern: line for each subfolder level in your dataset. Use any of the methods from Step 2.

Example:

1
2
3
template:
  - pattern: "{subject}"
  - pattern: "{session}"

Step 4: Package Files for Upload

At the deepest folder level containing your files, specify how to package them:

For DICOM files

1
2
3
4
5
template:
  - pattern: "{subject}"
  - pattern: "{session}"
  - pattern: "{acquisition}"
      scan: dicom

When scan: dicom is used:

  • Flywheel parses all .dcm files
  • Metadata is extracted from DICOM headers
  • Files are compressed into a zip for upload
  • Invalid DICOM files stop the import (use --ignore-scan flag to skip invalid files)

For non-DICOM files

1
2
3
4
5
template:
  - pattern: "{subject}"
  - pattern: "{session}"
  - pattern: "{acquisition}"
      packfile_type: jpg

Replace jpg with your file extension. This compresses all files in the folder into a single zip.

For mixed folders (select specific files)

1
2
3
4
5
6
template:
  - pattern: "{subject}-{session}-{acquisition}"
  - select:
    - pattern: "*.dcm"
      packfile_type: dicom
    - pattern: .*

This example only processes files ending in .dcm.

Step 5: Review and Validate

  1. Verify you have included each required variable exactly once:
  2. {subject}
  3. {session}

  4. {acquisition}

  5. Check your YAML syntax using YAML Lint

  6. Save your template file with a .yaml extension (e.g., my_template.yaml)

Step 6: Use Your Template

Run the ingest command with your template:

1
fw ingest template /path/to/template.yaml /path/to/dataset

For additional options and flags, see the fw ingest template command reference.

Common Template Patterns

Three-level hierarchy with metadata

1
2
3
4
5
template:
  - pattern: "{subject}"
  - pattern: "{session.label}-{session.info.visit_date}"
  - pattern: "{acquisition}"
      scan: dicom

Skip middle folder level

1
2
3
4
5
6
template:
  - pattern: "{subject}"
  - pattern: .*
  - pattern: "{session}"
  - pattern: "{acquisition}"
      packfile_type: nifti

Complex metadata extraction

1
2
3
4
5
6
template:
  - pattern: "{subject}-subject-Nov-2021"
  - pattern: .*
  - pattern: "amyg_s*_amyg_{session}_{subject.info.protocol}"
  - pattern: "{acquisition}"
      packfile_type: dicom

Verify Success

After the ingest completes, verify your template worked correctly:

1. Check the CLI Output

Look for confirmation messages in the terminal:

  • Number of files processed
  • Containers created (subjects, sessions, acquisitions)
  • Any warnings about pattern mismatches or skipped files

2. View Data in Flywheel Web UI

  1. Sign in to your Flywheel site
  2. Navigate to your project: Groups → [Your Group] → [Your Project]
  3. Verify:
  4. Subject labels match template pattern extraction
  5. Session labels are correct
  6. Acquisition labels are as expected
  7. File counts match what you uploaded

3. Check Metadata Extraction

If your template extracts custom metadata:

  1. Click on a subject, session, or acquisition
  2. View the "Info" tab
  3. Verify custom fields were populated correctly (e.g., {session.info.visit_date})

4. Validate File Organization

Check that files are organized correctly:

  • Files are in the correct acquisitions
  • File types are set correctly (DICOM, NIfTI, etc.)
  • No files were skipped unexpectedly

5. Review Audit Logs (if enabled)

If you used the --save-audit-logs flag:

  • Review the audit log file for any errors or warnings
  • Check for pattern matching issues
  • Verify all expected files were processed

Next Steps

Troubleshooting

Issue: Template syntax errors

  • Solution: Validate YAML at yamllint.com
  • Check for proper indentation (use spaces, not tabs)

Issue: Missing subjects, sessions, or acquisitions in Flywheel

  • Solution: Verify each variable ({subject}, {session}, {acquisition}) appears exactly once in your template

Issue: Invalid DICOM files stop ingest

  • Solution: Add --ignore-scan flag to skip invalid files and continue processing

Issue: Metadata not appearing in Flywheel

  • Solution: Use dot notation syntax: {container.info.fieldname} (e.g., {session.info.visit_date})