How to Run Gears as a Batch: Batch Run Gears

Introduction

This document assists with executing a gear on multiple datasets at once.

Instruction Steps

How do I run a gear on multiple sessions or acquisitions at the same time in Flywheel?

Executing a gear on multiple datasets at once is done by using the "Batch Run Gear" action. When viewing data on the Session Dashboard, you can select entire sessions, or specific acquisitions within a session, on which you want to run a Gear.

Under the Actions menu, select the Batch Run Gears option. A modal then asks you to select the gear type and then you can select from the gear you want from the list of those available on your site. You then are able to choose the version of the gear and set any configuration options available for that gear.

After pressing the Run Gear button, you are then notified how many jobs of that gear you are starting based on the datasets you selected. Selecting 'Confirm' pushes these jobs into your site's job queue and their progress will appear on the provenance tab.

How is the number of jobs queued determined?

When determining which jobs will be queued as part of a Batch Gear run, each acquisition container is evaluated for matches to the specified Gear inputs. (For SDK Gears with no inputs please see the next section.) Thus, when 'Batch Run Gear' is executed on a selection of sessions or acquisitions Flywheel will examine the files within each acquisition container and queue a job for each acquisition that has exactly one file that matches each type of expected inputs, as determined by the Gear's manifest. If the acquisition has multiple files that match a given input file's type, or if multiple inputs require a file of the same type, a job will not be queued for that acquisition.

For example, for a Gear that has a single input and expects a NIfTI file (as indicated by the Gear author in the Gear's manifest) the Batch Job Resolver will look at each acquisition to find those which have a single NIfTI file - matching the file type of the input. Note that to avoid ambiguity there should only be a single NIfTI file in that acquisition - acquisitions with multiple matches for a given Gear input will not run, as which file to choose as input is ambiguous in that case.

What if my Gear is SDK-enabled and has no inputs?

If your gear is SDK-enabled and has no inputs specified in the manifest, the system will assume that the Gear itself will determine which acquisitions and files will be used as input. In this case the system will only queue a single job per session, with the destination and type set to the session (regardless of which acquisitions are selected - note this behavior is the same for acquisitions within a collection as well).

When executed in this way the config.json file provided to the Gear at runtime will contain the following information (example assumes no configuration options):

{
 "config": {},
 "inputs": {
 "api-key": {
     "key": " <job_api_key>",
     "base": "api-key"
 }
 },
 "destination": {
     "type": "session",
     "id": "<session_id>"
 }
}

Why does batch run not queue my job?

If there are no files of the expected type (as indicated by the Gear author in the Gear's manifest) found in the acquisition, Flywheel will not be able to queue a job for that acquisition. Also, If the acquisition has multiple files that match a given input file's type, or if multiple inputs require a file of the same type, a job will not be queued for that acquisition. This is because the Batch Resolver cannot logically determine which file is to be used as input when multiple matches exist in the same acquisition container.