Gear Building Tutorial Part 7: Running a Gear Locally
Introduction
From the previous sections, you should have a gear directory with a complete:
- run script
- manifest
- Dockerfile
- message file
In this section, we will walk through how to run our example gear locally using the new Flywheel CLI. We will also cover how to upload a gear to Flywheel using our example gear.
Instruction Steps
- Gear Building Tutorial Part 7: Running a Gear Locally
- Introduction
- Instruction Steps
- Step 1. Using the Flywheel CLI to interact with a gear
- Step 2. Building the gear
- Step 3. Creating the config file
- Step 4. Adding config options to the config file
- Step 5. Adding inputs to the config file
- Step 6. Creating the input and output directories
- Step 7. Running the gear
- Step 8. Checking the output
- Step 9. Uploading a gear
- Running Your Gear
- Resources
- Navigation
Step 1. Using the Flywheel CLI to interact with a gear
The command we will be using the most in this part of the Gear Building Tutorial is: fw-beta gear. This command has a lot of powerful functionality for running gears locally, uploading gears to Flywheel, and copying gears between Flywheel instances. For this tutorial part, we will be focusing on the functionality related to running gears locally, as this will be useful when testing and debugging your own gears. Towards the end, we will touch on the command option for uploading gears to Flywheel.
Step 2. Building the gear
With the new Flywheel CLI, the first step to running a gear locally (or prepare it for upload to Flywheel) is to build the gear. The command we can use is:
This command is a wrapper script around Docker build with additional functionality to update the environment variable in the manifest with the environment specified in the Docker image.
To run this command for our example gear, navigate to inside the gear directory in a Terminal window and run:
Step 3. Creating the config file
Instead of being able to directly specify our input and other run.py input options, as we would if we were directly calling the script using python, we need to create a config.json file using the default config options and inputs we specified in our manifest.json file. This step is recreating what Flywheel does behind-the-scenes when running a gear on a Flywheel instance.
Creating the config.json file using fw-beta gear
The fw-beta gear config --create command will only generate config.json file entries for config options and inputs with default values set in the manifest.json file. Any config options or inputs without prespecified defaults will need to be added using either fw-beta gear config --config <key>=<value> or fw-beta gear config --input <key>=<value>.
The command we want to use is:
Let's check what is in our newly created config.json file and compare it to the inputs and config options we specified in our manifest.
We can see that it captured the "num_rep" config option correctly but is missing both the "my_name" config option under "config" and the input file under "inputs". This is because we only specified a default value for the "num_rep" option in the manifest. We did not give either the "my_name" config option or the input file a default value. So, we need to add both of these to our config file.
Step 4. Adding config options to the config file
Let's start by adding the missing "my_name" config option to our config.json file. Since this config option is quite simple, we could just manually update the config.json file ourselves by adding "my_name": "Maggie Simpson" to the existing list under "config".
However, it is useful to know that fw-beta gear config provides an option to automatically do this for us:
The VALUE of this config option can be whatever name we want, so that is easy enough to specify. In order to know the KEY, we can open up our manifest.json file and check what we called it there. Looking under the "config" section, we can see that we called this config option, "my_name". So, instead of manually updating our config file, we can use the following to automatically add both the "my_name" config option and specify the value we want to use:
Now if we look at our updated config file, we should see that it looks like what is shown above, with both the "num_rep" and "my_name" config options listed.
config.json
Step 5. Adding inputs to the config file
While it may seem daunting to have to figure out how to specify our message.txt file as the required input file directly to the config file, fortunately, fw-beta gear config has an option that can help us out.
The --input option shown below
will automatically update the config file with the necessary lines to specify input files for testing gears locally.
We know the VALUE of our input; that is just the path to our input text file. Since we are already inside of our gear directory in the Terminal window, we simply need to specify the name of our text file, message.txt. Similary, we can consult our manifest file to remember what we called the KEY for this input text file.
Open the manifest.json file again and scroll down to the section where we specified our input files to see that we helpfully called the KEY, message_file. Now that we know both the KEY and VALUE of our input, we can run:
Now if we check the contents of our config file again, we can see that an input file section has been added, and that our message.txt file has been specified as the message_file. In the below example of the config file we created, we have collapsed the "object" block to save space.
config.json
Step 6. Creating the input and output directories
The last step we need to perform before running our example gear for the first time is to create the Flywheel-expected input and output directory structure. As with Step 3 above, this is a step that is performed behind-the-scenes when running a gear on Flywheel.
But for the purposes of running a new gear locally, we do need to manually create both the input and output directory structure in our gear directory.
In the context of a gear, Flywheel expects a input directory structure where each input file is contained in its own subfolder below a gear-level input directory. Similarly, Flywheel expects a directory called output, where any gear results we want to access should be written to.
gear/
|- run.py
|- manifest.json
|- config.json
|- Dockerfile
|- input
|- input_file_1
|- input_file_2
|- output
For our example gear, we first need to create a new folder called input inside the gear directory. Then we need to create a new folder inside input with the same name as we called the input in our manifest. (Hint, we already looked us this KEY when adding the input message file to our config.json file.) So, this new subfolder should be called, message_file. Finally, we need to also create a new folder called output inside the gear directory.
Our gear directory should now look like:
Gear directory structure
We do not need to copy our message.txt file into input/message_file/, as Flywheel will do this automatically when we run our example gear. We just need to create the input folder structure.
Step 7. Running the gear
Now we are ready to test run our example gear for the first time! From inside our example gear directory, we can run:
If everything goes well, we should see the below screendump in our Terminal window:
Hello World gear output
> fw-beta gear run
/Users/flywheelio/Documents/GitLab/Gear-Building-Tutorial/work does not exist, skipping.
Populating inputs.
* Found file input with source path: ~/D/G/G/message.txt,
copying to destination: ~/D/G/G/i/m/message.txt
Creating container from homer/hello-world:0.1.0
Hello, Maggie Simpson!
Hello, Maggie Simpson!
Hello, Maggie Simpson!
Hello, Maggie Simpson!
Hello, Maggie Simpson!
This is a special message saved in the "message.txt" file.
Anyone can change this message by changing the "message.txt" file.
Oh, I see it's your birthday. Happy birthday!
It's probably not your birthday...
But that's going to freak someone out :)
Dumping run command in run.sh
If you are getting an error message at this point and are unsure what it might mean, don't worry. There is a complete working copy of this example gear that you can download here. If you downloaded the zip file in Part 2, just download the zip file for this hello-world-debugged version. If you cloned the GitLab repository in Part 2, then you should just be able to checkout the hello-world-debugged branch. Re-run the fw-beta gear build . step from inside the hello-world-debugged folder, and then skip to this fw-beta gear run step to get back on track.
Step 8. Checking the output
Remember in addition to setting our run.py script to print to the terminal screen, we also set it to print out the "Hello, ..." statement to an output text file called, hello.txt. If we check in the output folder, we can see our hello.txt file was successfully saved and available for us to view.
Checking the contents of our output file, we can see that the "Hello,..." statement was printed to the file the number of times specified by the num_rep config option.
hello.txt
Within the context of running a gear in Flywheel, any files written out to the output folder would be available under the gear's Results tab.
Step 9. Uploading a gear
The last step in building a gear is to upload our fully-functional test gear to our Flywheel instance. To upload the gear to a Flywheel instance, the key step is to make sure that we are logged into the desired Flywheel instance using fw-beta login and our API key for the desired Flywheel instance. If you have not already created an API key, follow the steps in this section.
Users with access to multiple Flywheel instances
If you have Flywheel access for multiple sites/instances, this step is extremely important, as Flywheel will upload the gear to whichever instance you are currently logged into. You can use fw-beta login --status to check which Flywheel instance you are currently logged into.
Now that we are sure that we are logged into the desired Flywheel instance, we can open a terminal window in our test gear directory and type the following:
We will see a series of messages on screen indicating the status of the upload. Gears with large docker images may take a long time to upload, however for our case this should be relatively quick.
That's it! Navigating to our Flywheel instance, we should now see the gear under Installed Gears on the left-hand navigation bar (under the COMPUTE section), as well as in the drop-down menu when choosing a gear to run. (You may need to refresh the Flywheel instance page if you had it open before running the gear upload step.)
Here are a couple of things to note about Flywheel gears on an instance:
Flywheel gears on an instance
- Gears cannot be overwritten. Every new upload must have a different version, no matter how small the change. While it is strongly recommended that versions follow a logical progression, this is not enforced by Flywheel.
- Gears cannot be deleted once uploaded. This is to maintain provenance of Flywheel instances. However, Flywheel support can hide old gears on an instance if desired, to avoid gear clutter. All logs and outputs of a hidden gear will still visible, though.
Running Your Gear
To run this gear on the Flywheel instance, we will need to either create a new project and upload a "message.txt" file or upload a "message.txt" file to an existing project to use as input. When launching the gear, remember to set the name and the number of reps in the config tab. The output of this gear will show up in the gear "logs", which we can view in the provenance tab for any session, as well as in the Analysis tab, since we specified our gear to be an analysis gear in the manifest. A detailed description on how to run any gear, view its status, logs, and examine gear outputs can be found in this article.
Resources
Navigation
Previous Part 6 The Dockerfile
Next Part 8 Logging