Creating a configuration file for fw ingest

  • Updated

The fw ingest command allows you to upload large amounts of data to your Flywheel instance. The config file for the Ingest command controls how your data is uploaded, organized, and de-identified.

Creating a configuration file is helpful if you:

  • Use many command line arguments to properly sort and import data

  • Share your import methods with collaborators

  • Plan to automate pipelines for data import

This article explains how to use the config file as well as an example config file you can customize to fit your ingest needs.

Customize the config file for your data

  1. Copy and paste the example config file below in to a text editor.

  2. Update the config to fit your data as well as the command you are using. For more information on the sections within this config, see the following articles:

  3. Save with the .yaml extension.

    Use an online YAML validator (such as YAMLlint ) to make sure the format is correct. 

Example config file

Below is an example ingest config file. Learn more about customizing this example below.

#####
# Template and Group/Project Settings
#####

template:
  - pattern: "{project}"
  - pattern: "{subject}"
  - pattern: "{session}"
  - pattern: "{acquisition}"
    packfile_type: dicom

#####
# Optional includes/excludes for directories and files 
#####

# Patterns of directories to include
include-dirs:  
  - "*-DCM"

# Patterns of filenames to exclude
exclude:  
  - "*.txt"
  - "*.xml"

  #####
# De-identification Settings
#####

deid-profile: custom

deid-profiles:
- name: custom

  dicom:

    # What date offset to use, in number of days
    date-increment: -17
    
    # Remove all private tags
    remove-private-tags: true

    # Set patient age from date of birth
    patient-age-from-birthdate: true

    # Set patient age units as Years
    patient-age-units: Y

    fields:
      # Remove a dicom field (e.g.remove PatientID)
      - name: PatientID
        remove: true
      # Increment a date field by -17 days
      - name: StudyDate
        increment-date: true
      # One-Way hash a dicom field to a unique string
      - name: AccessionNumber
        hash: true

How to use the config file

To use the config file in your ingest command:

  1. Instead of setting the optional flags in the command, you can combine all of the settings in to the config file option. To call this config file in your command:

    $ fw ingest template ~/Documents/DataToImport TemplateExample --config-file ~/Documents/ExampleConfig.yaml

  2. You'll see the following output when the Flywheel CLI is using the config file:

    $ fw ingest template ~/Documents/DataToImport TemplateExample --config-file ~/Documents/ExampleConfig.yaml

$ Reading config options from: Users/AvaFranklin/Documents/ExampleConfig.yaml

CLI arguments override anything set in the config file. For example:

If you add the flag --group Psychology , but the config file includes the line group: PsychLab, Flywheel will use Psychology as the group name.

Customization FAQs

Can I change the order of the settings in the config file?

The example config above uses sections to organize the settings. These sections can be removed or reordered. However, there are special considerations for the options under the Template and Group/Project Settings section (see our template article for more information).

If I have created a template or de-id profile already, can I reuse that in the ingest config file?

Yes. If you created a template, de-id profile, or config file to use with the import command, those files can be integrated in the ingest config. This means if those files were valid with the import command, they are also valid for the ingest command.

Are there more settings I can add to the config file?

Yes. The valid settings for the config file depend on which fw ingest command you use. The optional flags in the reference articles are the same settings you can add to the config file. See the following articles for more information:

External resources

Online YAML validator: YAML lint