Skip to content

Logo Logo

Gear Building Tutorial Part 2: Creating Your First Gear

Introduction

The purpose of this tutorial is to help you create your very first gear.  This gear will be as basic as possible, and will focus mostly on the mechanics of gear development and the steps involved in an effort to familiarize you with the process.  Hopefully, this tutorial will teach you enough that you'll be able to start debugging more advanced use cases on your own.   More advanced tips, tricks, and best practices will be covered in later sections.

Instruction Steps

This section serves an introduction to the different types of files and software Flywheel uses for its gears, and the role each component plays when running the gear itself.  At its core, a gear consists of three files, each performing a very important function:

  1. The run script:

The run script is an executable file, usually named "run.py" for a python script.  This is the first command to be called. It will carry out all the processing of any input data, call other functions and files that are present in the gear, handle intermediate files, and generally control the entire order of execution of the computations.

2. The manifest:

  • The manifest is a JSON schema. It acts as both a header for the gear, containing various pieces of metadata and information about the gear, as well as a bridge between user input and the run script.

  • Keys in the manifest fall into two categories:

  • Metadata (gear name, version number, gear author, etc)

  • Inputs.  Inputs can either be files stored in flywheel that the gear uses when run (input values), or settings that the user can enter manually to configure how the gear runs (config values).

For example, imagine a gear that applies spatial smoothing to an fMRI image.  An input would be a T2* image from a subject's session (a file on flywheel), and a config setting would be the size of the smoothing kernel (a value the user can set).  

3. The Dockerfile:

The Dockerfile is a series of commands that is read by the Docker engine.  These commands set up a specific computing environment for the gear to run, which is called a Docker image. This image allows you to run the gear on any computer.  Regardless of the software or environment of the computer, the docker image will always run with the same identical environment as you set in the Dockerfile.  

The way flywheel uses these three files to run gears is as follows:

  1. The Manifest is used to generate a UI so that inputs and configuration can be set by the user.
  2. Once these inputs are set, the gear engine generates a config.json file, which contains all the values set by the user (or if not set, containing the default values).
  3. A Docker container is created from the Docker image name provided in the manifest, and the config file is generated from the inputs and placed into main working directory.
  4. A command (set by the user in the manifest, or by default “python run.py”) is called, starting the execution of the gear’s main script.

By the end of this section, you'll use the tools installed from Part I, and build a very simple gear to run on your machine.  

Workflow

Good gear workflow is important for streamlined gear development.  Creating a dockerfile can take time, especially if it requires a lot of software.  Additionally, compiling the dockerfile into a Docker image can also be a lengthy process.   There are ways to minimize build times, which we'll cover later in the tutorial, but in general you want to build your Docker image as few times as possible.  

There are many different ways to optimize your workflow, and the best way may change depending on your situation.  For example, if using a specific piece of software to process some data, you may wish to set up your docker image first, and sort out all your dependencies so that you can develop directly in your docker image.  On the other hand, if you’re unsure of exactly what tools/libraries/environments you may need, it may be better to get a run script functioning on your computer, and then generate a dockerfile based on the tools you ended up using.

For this example, we’ll start with generating a simple run script.  For the sake of the tutorial, it will help you understand the way we set up our docker file if you understand the run script, and exactly what it’s doing.

We’ll develop this example gear in the following steps:

  1. Develop a working run script
  2. Generate a Manifest
  3. Generate a dockerfile
  4. Test the gear locally
  5. Upload to Flywheel

We'll now cover each of these steps in detail, working through a real gear example as we go.  By the end, we'll have created a fully functioning gear.  Each gear will need its own directory to be developed in.  This directory will have your manifest, dockerfile, and run script.  Any additional files/scripts you'd like to use should go in here as well.

Your Gear’s Main Directory

Let’s create a folder on your computer to work in for the gear:

  1. Navigate to the directory you'd like to work in.
  2. Create a new folder called "GearTutorial".

From here on out, all our files will be created in this directory.  Each of the main gear files will be covered in the following sections, starting with the run script.

Part 1: Environment Setup

Part 2: Gear Intro

Part 2a: The Flywheel Environment

Part 2b: The Run Script and Logging

Part 2c: The Manifest

Part 2d: The Dockerfile

Part 2e: Testing/Debugging