Skip to content

Running a Flywheel Job Locally with Docker

2020-05-12 Update: This example no longer requires users to be a site admin. While site admins may still use this example to run any job, normal users and developers may use this code to run their own jobs locally NOTE: The input arguments have changed - the gear_name is now required as input. Please see the help for more.

Background

It is occasionally beneficial for a Gear Developer to re-run an existing Flywheel joblocally to debug, or otherwise improve, their algorithm. To do this we provide here a simple python script which uses the Flywheel SDK to generate (and/or download) all of the input and configuration files required to run the job locally, using Docker.

Requirements

Privileges

This code no longer requires that the user have Site Admin privileges. If you are a site admin you can grab any job across the system, however, if you are not a site admin you can still run the code below if the job you are attempting to run locally is your own.

Software Dependencies

To run the script and execute the resulting job locally you will need:

Code Inputs

To run the provided code, you will need three things, your API Key, the name of the gear, and the Job ID for the job you want to run locally.

  • API Key: Your API Key can be found on the Profile Page from within the Flywheel UI. Learn more about creating User API Keys from the Profile page
  • Gear Name: The Gear Name can be found at the top the job log, within the Flywheel UI. In the example below it's "mriqc".
  • Job ID: The Job ID can be found at the bottom of the job log, within the Flywheel UI.

Usage

Download the code in the next block, modify permissions to make it executable, and run with python. See the code comments for more background.

Once the code has been run, a 'run.sh' file will be generated, which contains the Docker command that can be used to run the job locally.

Development Notes

  • You can modify the 'run.sh' file to change the version of the gear used to run the job. This is particularly useful if you are iterating on versions of the gear. For example, if you build a new version of the gear you can modify the tag within the 'run.sh' file to run the new version.
  • You can modify configuration parameters in the generated 'config.json' file, to see how those changes affect the algorithm.
  • If you modify the algorithm and add configuration parameters, be sure to also add those to the 'config.json' file.

fwutil_job_run_local.py