Bulk Export Settings

This article explains the various options available when starting a new Bulk Export job.

All of these settings are available when starting the export job from the Web App or from the new (BETA) CLI.

Export Job Settings

• Export Name: (Optional) Provide a unique label so that you can identify your export run.
  • If you do not provide an Export Label, one will be automatically generated later.
• Description: (Optional) Provide a more detailed description of the resulting export data.

• Export Destination: Select either "My Computer" or one of the configured External Storages as the destination to where the files should be exported.

  Missing External Storage

  If you do not see any external file store destination options in the dropdown, ask your Flywheel Site Admin to add external storage providers.

• Overwrite Handling: Select how existing files located in the destination at the same path as the planned export should be handled.

  • Auto: (Default) The timestamps of the file being exported are compared with those of the existing file located in the destination at the target path, and the most recent file is used.
  • Never: Existing files in the destination at the target path are never overwritten.
  • Always: Existing files in the destination at the target path are always overwritten.
• Delete Pre-Existing Files in Destination: In some cases you may already have other files located in the destination path. When this option is enabled, all files located in the destination path will be deleted before the export begins even if the existing files do not directly conflict with the files being exported.
  • This option is useful if, for example, you change the filter rules on a subsequent export of the same project data, so that less data is exported in the later export. This way, the destination storage will only contain exactly the files from the latest export and none of the "extra" files from a previous export.

Export Rules

Selecting Files

Export rules are used to select which files to export. Multiple export rules can be specified for each export.

Files will only be selected for export if they match at least one export rule, otherwise they will be skipped.

Each export rule consists of:

• A required Container Level context defining the scope in which the filters are applied, and
• A optional set of Include and/or Exclude filters for selecting which files to export.

Export rules are evaluated in order for every file. The first rule matching the following conditions will be used:

• The Container Level matches the file level, and
• Any of the include filters match (if specified), and
• None of the exclude filters match (if specified).

The available options for Container Level are:

• Project
• Subject
• Session
• Acquisition (Default)

Filters

There are two types of filters that can be used in any combination:

• Include: Only files matching this filter will be exported unless they also match an exclude filter.
  • If no include filter is specified at all, then all files will be selected unless they match an exclude filter.
• Exclude: Any file matching this filter will not be exported (regardless of any include rules).

Each filter (whether include or exclude type) consist of the following three components:

• Field
• Operator
• Value

Supported Fields for Export Filters

The following fields are available for use in export filters are:

Supported Operators for Export Filters

The following logical operators are supported for use in export filters are:

Placement Rules

Placement rules are used to define how files will be organized and named when placed into the destination storage. The following options are available:

• Default Destination Path: Defines a template for the destination path for the files.
  • The default will use the standard hierarchy labels to create the path: {project.label}/{subject.label}/{session.label}/{acquisition.label}/{file.name}
  • Optionally, you can customize the path by replacing the text in the path. You can enter fixed text, use other fields to define the path, shorten the path by deleting containers, or lengthen the path by adding fields. For example this would be a valid path template: my_data/{project.label}/{subject.mlset}/{subject.label}/{acquisition.label}/{file.name}
• Include Metadata: If enabled, a "sidecar" JSON file will be placed next to every file. This sidecar file contains the Flywheel metadata describing the file itself and each of its parent containers.
• Unzip Files: If enabled, archive files will be unzipped when writing to the export destination.
  • Zip naming: Options for handling the unzipped file path in the extract to allow collapsing the path.
    • Original: Use the path and the name as it was archived.
    • Underscore: Combine the path and filename into a filename using underscores.
    • Basename: Ignore the path and just use the file name from the archive.
    • E.g., for the archive file:  /data/image/3D/rfMRI_REST.dcm
      • Original: /data/image/3D/rfMRI_REST.dcm
      • Underscore: data_image_3D_rfMRI_REST.dcm
      • Basename: rfMRI_REST.dcm

Timepoint (Snapshot)

The export is created based on a "snapshot" of the state of the project data in Flywheel at a specific point in time (timepoint).  This helps ensure the content of the export is repeatable and consistent.

You can choose to either:

• Export from the current date and time (default), or
• Export from a previous timepoint at which an snapshot was previously created.
  • This is useful if you want to reproduce an export from a prior run.

If you select the current date and time, a new snapshot will be created. This new snapshot can then be re-used as a "previous timepoint" in subsequent export jobs.