Gear Building Tutorial Part 2: Creating Your First Gear

Introduction

The purpose of this tutorial is to help us create our very first gear. This gear will be as basic as possible and focus mostly on the mechanics of gear development and the steps involved in an effort to familiarize us with the process. Hopefully, this tutorial will teach us enough that we'll be able to start building more advanced use cases on our own. More advanced tips, tricks, and best practices will be covered in later sections.

Main gear files

This section serves an introduction to the different types of files and software Flywheel uses for its gears, and the role each component plays when running the gear itself. At its core, a gear consists of three files, each performing a very important function:

The run script:

The run script is an executable file, usually named run.py for a python script. This is the first command to be called. It will carry out all the processing of any input data, call other functions and files that are present in the gear, handle intermediate files, and generally control the entire order of execution of the computations.
The manifest:

The manifest is a JSON schema. It acts as both a header for the gear, containing various pieces of metadata and information about the gear, as well as a bridge between user input and the run script. Keys in the manifest fall into two categories:
- Metadata: gear name, version number, gear author, etc
- Inputs: Inputs can either be files stored in Flywheel that the gear uses when run (input values), or settings that the user can enter manually to configure how the gear runs (config values).
For example, imagine a gear that applies spatial smoothing to an fMRI image. An input would be a T2* image from a subject's session (a file on Flywheel), and a config setting would be the size of the smoothing kernel (a value the user can set).
The Dockerfile:

The Dockerfile is a series of commands that is read by the Docker engine. These commands set up a specific computing environment for the gear to run, which is called a Docker image. This image allows us to run the gear on any computer. Regardless of the software or environment of the computer, the Docker image will always run with the same identical environment as we set in the Dockerfile.

How Flywheel uses these files

The way Flywheel uses these three files to run gears is as follows:

The manifest is used to generate a UI so that inputs and configuration can be set by the user.
The gear engine generates a config.json file, which contains all the inputs and configuration values set by the user (or if not set, containing the default values).
A Docker container is created from the Docker image name provided in the manifest.
A command (set by the user in the manifest, or by default python run.py) is called, starting the execution of the gear’s main script.

By the end of this tutorial series, we'll use the tools installed from Part 1 and build a very simple gear to run on our machine.

Gear development workflows

Good gear workflow is important for efficient gear development. There are many different ways to optimize your workflow, and the best way may change depending on the situation. For example, if using a specific piece of software to process some data, it may be easier to set up the Docker image and sort out all the dependencies first, so that you can develop directly in the Docker image. On the other hand, if you’re unsure of exactly what tools/libraries/environments you may need, it may be better to get a run script functioning on your local machine, and then generate a Dockerfile based on the tools you end up using.

Optimizing Dockerfiles and minimizing build times

Creating a Dockerfile can take time, especially if it requires a lot of software. Additionally, compiling the Dockerfile into a Docker image can also be a lengthy process. There are ways to minimize build times, but in general you want to build your Docker image as few times as possible.

For this example, we’ll start with generating a simple run script. Choosing this workflow will help us understand the way we set up our Dockerfile, as we will first have a chance to understand the run script and exactly what it’s doing.

We’ll develop this example gear in the following steps:

Develop a working run script
Generate a Manifest
Generate a Dockerfile
Test the gear locally
Upload to Flywheel

We'll now cover each of these steps in detail, working through a real gear example as we go. By the end, we'll have created a fully functioning gear. Each gear will need its own directory to be developed in. This directory will have our manifest, Dockerfile, and run script. Any additional files/scripts we would like to use should go in here as well.

Setting up the main gear directory

While this tutorial is designed to be simple enough for us to create all of the necessary folder structure and files from scratch, in general, most gear development starts with some sort of template gear directory with skeleton versions of the run script, manifest, and Dockerfile. As this is our first time building a gear, we have created a GitLab repository with fleshed out "skeleton" versions of these three main files, along with an input text file we will use later on in our run script.

Let's get started!

Below are two options for getting a copy of our skeleton Gear Building Tutorial gear directory. Option 1 assumes you have some comfort level with cloning git repositories and switching between branches. Option 2 assumes you do not have much experience with git repositories and would prefer just to download the skeleton gear directly. Pick whichever option sounds better to you.

Option 1: Cloning the GitLab repository

If you are familiar with working with GitHub or GitLab repositories, you can start by cloning the Gear Building Tutorial repository and checking out the hello-world branch (hint: git checkout hello-world).

Option 2: Downloading a zip file

If you are completely new to GitHub and GitLab, no worries. You can navigate to the same Gear Building Tutorial repository. Make sure that you are in the "hello-world" branch from the upper-left pull-down menu and select "zip" from the blue "Code" pull-down menu under the "Download source code" section. Once the file is finished downloading, unzip the file.

You are also welcome just to start from scratch and create all of the necessary files in a directory called, Gear-Building-Tutorial. Just note that the subsequent parts generally assume that you are working from these "Gear Building Tutorial" skeleton template files.

From here on out, all of the work we do will be saved in this Gear-Building-Tutorial directory. Each of the main gear files will be covered in the following sections, starting with the run script in Part 4.

Wrapping Up

In this part, we have learned a bit about the three main files needed to build a gear: the run script, the manifest, and the Dockerfile. We have also cloned or downloaded the "Gear Building Tutorial" skeleton template from GitLab. In Part 3, we will cover the Flywheel environment. The Flywheel environment is the folder structure that the Flywheel gear engine expects all gears to use. Understanding the Flywheel environment and how it handles inputs and outputs are key to successful gear development.

Previous Part 1 Developing Gears

Next Part 3 The Flywheel Environment