Skip to content

Step 1: Create a Template for File Upload

Introduction

The template is a set of instructions for how to map your dataset to a Flywheel project. 

All projects in Flywheel are made up of subjects, sessions, and acquisitions. Using the template in the config file, leverage the folder and file structure on your local machine to map your dataset to subjects, sessions, and acquisitions. The template also allows you to use folder or filenames from your dataset as additional metadata fields.

Note: This is different from the ingest dicom command, which uses the DICOM file's metadata to organize data.

Instruction Steps

Mapping Dataset Folder Structure to Flywheel

To successfully upload your dataset, you must define the labels for subjects, sessions, and acquisitions. Below is an example of the most basic template for uploading DICOM files.

The steps below explain how to build your own template for labeling subject, sessions, and acquisitions.

1
2
3
4
5
6
7
8
# Template and Group/Project Settings
#####

template:
  - pattern: "{subject}"
  - pattern: "{session}"
  - pattern: "{acquisition}"
      scan: dicom
  1. To begin, open a plain text editor such as TextEdit, Notepad, Sublime, etc.

  2. Copy and paste the first step of your template:

    1
2
    template:
- pattern:

    This first pattern field corresponds to the top folder of your dataset.

  3. Configure how your first folder and any data inside it will map to the Flywheel project. Below is an overview of the available options. See the Reference section at the end of this article for more details.

  4. Use the {subject}, {session}, or {acquisition} variables to create labels

    • To successfully upload your dataset, you must define the labels for subjects, sessions, and acquisitions in your dataset. You can use all or part of the folder name for the label. To use all of the folder name for the subject label:
    1
2
    template:
- pattern: "{subject}"

    For example if you had the following folder name it would become a subject label:

    Example folder name Subject label
    sub-01 sub-01

    To use only part of the folder name, replace a portion of the folder with the "{subject}" variable. For example:

    1
2
    template:
- pattern: "{subject}-subject-Nov-2021"
    Example folder name Subject label
    001-subject-Nov-2021 001
    002- subject-Nov-2021 002

  5. Add metadata

    • Metadata has many useful applications in Flywheel including being used to properly categorize data, create collections, or curate a dataview. The template allows you to pull folder or file names to add as metadata. For example:
    1
2
    template:
- pattern: "001-subject-{session.info.dataset}"

    For example

    Example folder name Metadata
    001-subject-Nov-2021 dataset: Nov-2021

    Note: this metadata field is added to the sessions of your project.

  6. Skip this folder: Use regex to skip this folder level and move on to the next:

1
2
template:
- pattern: .*

  • Use filenames for labels in Flywheel

    • Oftentimes the filename has useful information. To leverage the filename for a label, add a scan step for filenames.

    The filename scan allows for regex pattern matching, so you can pull out only the relevant information in each filename. For example:

    1
2
3
4
5
    template:
- pattern: "{subject}"
    scan:
    name: filename
    pattern: "(?P<acquisition>)[^.]*"

    The above template would take the filename without the extension and make it the acquisition label. For example:

    Example filename Acquisition label
    SAG T1.dicom SAG T1

    !!! tip

    1
       Use angled brackets **<>** instead of curly brackets **{}** when using regex to assign variables. This is a requirement of Python regex.

  • Add a second -pattern: to the template. This corresponds to the first subfolder of your dataset. You can use any of the above mapping methods. For example:

    1
2
3
    template:
- pattern: "{subject}"
- pattern: .*

  • Continue for each new subfolder level in your dataset until you reach the last folder you wish to upload. To package files for upload, use packfile_type or scan:

  • Compress all files into a packfile

    • If you don't have DICOM data, you should still compress all of your images for upload using packfile_type: <filetype>. This creates a single zip file for upload. For example:
    1
2
3
4
5
    template:
- pattern: "{subject}"
- pattern: "{session}"
- pattern: "{acquisition}"
    packfile_type: jpg

    or

    1
2
3
4
5
6
    template:
- pattern: "{subject}-subject-Nov-2021"
- pattern: .*
- pattern: "amyg_s*_amyg_{session}_{subject.info.pcol}"
- pattern: "{acquisition}"
    packfile_type: dicom

  • Validate and package DICOM data: If you have DICOM data, and you would like to organize data using the seriesinstanceuid tag, use the dicom scan step:

    1
2
3
    template:
- pattern: "{acquisition}"
    scan: dicom

    When dicom scan is enabled, Flywheel parses all files with the .dcm extension in that folder. Flywheel pulls out relevant metadata from the DICOM files to use as Flywheel metadata and compresses all data into a zip file for upload.

    If the file is not a valid DICOM file, the file is not uploaded and the import stops by default. To determine if a file is valid DICOM, we look for a DICM string at byte 128. However, you can use the --ignore-scan flag in your CLI command to set it so that Flywheel only ignores the invalid DICOM file and continues to upload valid files.

  • "Select" folders for upload: When there are multiple folders at the same level, you can use select: to set different rules for each folder:

    1
2
3
4
5
6
    template:
- pattern: "{subject}-{session}-{acquisition}"
- select:
    - pattern: "*.dcm"
    packfile_type: dicom
    - pattern: .*

    This example only scans files that end in the common DICOM file extension, .dcm.

  • Review your template. Verify that you have added each of these variables once:

  • {subject}

  • {session}

  • {acquisition}

  • Verify that your template is valid YAML using an online tool such as YAML lint.

Reference

See the [Ingest Template - Parameters Reference](./ingest_template_parameters_reference.md] guide for more information about the various options that can be set within the ingest template.