Gear Building Tutorial Part 7: Running a Gear Locally
Introduction
From the previous sections, you should have a gear directory with a complete:
- run script
- manifest
- Dockerfile
- message file
In this section, we will walk through how to run our example gear locally using the new Flywheel CLI. We will also cover how to upload a gear to Flywheel using our example gear.
Instruction Steps
- Gear Building Tutorial Part 7: Running a Gear Locally
- Introduction
- Instruction Steps
- Step 1. Using the Flywheel CLI to interact with a gear
- Step 2. Building the gear
- Step 3. Creating the config file
- Step 4. Adding config options to the config file
- Step 5. Adding inputs to the config file
- Step 6. Preparing the gear
- Step 7. Running the gear
- Step 8. Checking the output
- Step 9. Uploading a gear
- Running Your Gear
- Resources
- Navigation
Step 1. Using the Flywheel CLI to interact with a gear
The command we will be using the most in this part of the Gear Building Tutorial is: fw-beta gear
. This command has a lot of powerful functionality for running gears locally, uploading gears to Flywheel, and copying gears between Flywheel instances. For this tutorial part, we will be focusing on the functionality related to running gears locally, as this will be useful when testing and debugging your own gears. Towards the end, we will touch on the command option for uploading gears to Flywheel.
Step 2. Building the gear
With the new Flywheel CLI, the first step to running a gear locally (or prepare it for upload to Flywheel) is to build the gear. The command we can use is:
This command is a wrapper script around Docker build with additional functionality to update the environment
variable in the manifest with the environment specified in the Docker image.
To run this command for our example gear, navigate to inside the gear directory in a Terminal window and run:
Step 3. Creating the config file
Instead of being able to directly specify our input and other run.py
input options, as we would if we were directly calling the script using python
, we need to create a config.json
file using the default config options and inputs we specified in our manifest.json
file. This step is recreating what Flywheel does behind-the-scenes when running a gear on a Flywheel instance.
Creating the config.json file using fw-beta gear
The fw-beta gear config --new
command will only generate config.json
file entries for config options and inputs with default values set in the manifest.json
file. Any config options or inputs without prespecified defaults will need to be added using either fw-beta gear config --config <key>=<value>
or fw-beta gear config --input <key>=<value>
.
The command we want to use is:
Running an older version of fw-beta
?
If you are running a version of fw-beta
earlier than 0.15.3
, the command to create a new config.json
file is: fw-beta gear config --create
.
Let's check what is in our newly created config.json
file and compare it to the inputs and config options we specified in our manifest.
We can see that it captured the "num_rep"
config option correctly but is missing both the "my_name"
config option under "config"
and the input file under "inputs"
. This is because we only specified a default value for the "num_rep"
option in the manifest. We did not give either the "my_name"
config option or the input file a default value. So, we need to add both of these to our config file.
Step 4. Adding config options to the config file
Let's start by adding the missing "my_name"
config option to our config.json
file. Since this config option is quite simple, we could just manually update the config.json
file ourselves by adding "my_name": "Maggie Simpson"
to the existing list under "config"
.
However, it is useful to know that fw-beta gear config
provides an option to automatically do this for us:
The VALUE
of this config option can be whatever name we want, so that is easy enough to specify. In order to know the KEY
, we can open up our manifest.json
file and check what we called it there. Looking under the "config"
section, we can see that we called this config option, "my_name"
. So, instead of manually updating our config file, we can use the following to automatically add both the "my_name"
config option and specify the value we want to use:
Now if we look at our updated config file, we should see that it looks like what is shown above, with both the "num_rep"
and "my_name"
config options listed.
config.json
Step 5. Adding inputs to the config file
While it may seem daunting to have to figure out how to specify our message.txt
file as the required input file directly to the config file, fortunately, fw-beta gear config
has an option that can help us out.
The --input
option shown below
will automatically update the config file with the necessary lines to specify input files for testing gears locally.
We know the VALUE
of our input; that is just the path to our input text file. Since we are already inside of our gear directory in the Terminal window, we simply need to specify the name of our text file, message.txt
. Similary, we can consult our manifest file to remember what we called the KEY
for this input text file.
Open the manifest.json
file again and scroll down to the section where we specified our input files to see that we helpfully called the KEY
, message_file
. Now that we know both the KEY
and VALUE
of our input, we can run:
Now if we check the contents of our config file again, we can see that an input file section has been added, and that our message.txt
file has been specified as the message_file
. In the below example of the config file we created, we have collapsed the "object"
block to save space.
config.json
Hidden input mapping .json
file
When you run the fw-beta gear config --input
command, in addition to updating the config.json
file to populate the "inputs"
section, it will also create a hidden file called, .input_map.json
. This file contains information needed by fw-beta gear run --prepare
to correctly find the input message.txt
file in our gear directory and move it to the gear input/
directory.
Step 6. Preparing the gear
Now that we have built the gear, set up the new config file, and populated the config file with our config options and input file, we need to run one last step to create the gear structure in a temporary directory on our local machine. This step handles creating the Flywheel directory structure and copies any input file(s) specified in our config file into the correct place.
Before we run this prepation step, let's make sure that our gear directory has the following files:
Gear directory structure
To automatically prepare our temporary gear directory and populate our input file, we can run:
This command will print an output to the terminal screen:
We will need the path to the gear structure from the output, so go ahead and copy the path that comes after Prepared gear structure in
. So, for our example gear, this would be: /tmp/gear/hello-world_0.1.0
. Depending on your local system set up, the format of this path may be different.
Step 7. Running the gear
Now we are ready to test run our example gear for the first time! From inside our example gear directory, we can run:
Remember to replace the path, /tmp/gear/hello-world_0.1.0
, with the output from when you ran fw-beta gear run --prepare
in the previous step.
If everything goes well, we should see the below screendump in our Terminal window:
Hello World gear output
Error related to: "Could not find input map,..."?
If you get the following error when running your gear locally: "Could not find input map, inputs won't be populated. If you want to use inputs, try adding them with gear config
", check to make sure that the .input_map.json
file exists in your gear directory. If you accidentally deleted it, just re-run the fw-beta gear config --input
command from Step 5 and then the fw-beta gear run --prepare
command from Step 6.
If you are getting any other error message at this point and are unsure what it might mean, don't worry. There is a complete working copy of this example gear that you can download here. If you downloaded the zip file in Part 2, just download the zip file for this hello-world-debugged
version. If you cloned the GitLab repository in Part 2, then you should just be able to checkout the hello-world-debugged
branch. Re-run the fw-beta gear build .
step from inside the hello-world-debugged
folder, and then skip to this fw-beta gear run
step to get back on track.
Step 8. Checking the output
Remember in addition to setting our run.py
script to print to the terminal screen, we also set it to print out the "Hello, ..."
statement to an output text file called, hello.txt
. If we check in the output
folder of our temporary gear structure created in Step 6, we can see our hello.txt
file was successfully saved and available for us to view.
Checking the contents of our output file, we can see that the "Hello,..." statement was printed to the file the number of times specified by the num_rep
config option.
hello.txt
Within the context of running a gear in Flywheel, any files written out to the output
folder would be available under the gear's Results
tab.
Step 9. Uploading a gear
The last step in building a gear is to upload our fully-functional test gear to our Flywheel instance. To upload the gear to a Flywheel instance, the key step is to make sure that we are logged into the desired Flywheel instance using fw-beta login
and our API key for the desired Flywheel instance. If you have not already created an API key, follow the steps in this section.
Users with access to multiple Flywheel instances
If you have Flywheel access for multiple sites/instances, this step is extremely important, as Flywheel will upload the gear to whichever instance you are currently logged into. You can use fw-beta login --status
to check which Flywheel instance you are currently logged into.
Now that we are sure that we are logged into the desired Flywheel instance, we can open a terminal window in our test gear directory and type the following:
We will see a series of messages on screen indicating the status of the upload. Gears with large docker images may take a long time to upload, however for our case this should be relatively quick.
That's it! Navigating to our Flywheel instance, we should now see the gear under Installed Gears on the left-hand navigation bar (under the COMPUTE section), as well as in the drop-down menu when choosing a gear to run. (You may need to refresh the Flywheel instance page if you had it open before running the gear upload step.)
Here are a couple of things to note about Flywheel gears on an instance:
Flywheel gears on an instance
- Gears cannot be overwritten. Every new upload must have a different version, no matter how small the change. While it is strongly recommended that versions follow a logical progression, this is not enforced by Flywheel.
- Gears cannot be deleted once uploaded. This is to maintain provenance of Flywheel instances. However, Flywheel support can hide old gears on an instance if desired, to avoid gear clutter. All logs and outputs of a hidden gear will still visible, though.
Running Your Gear
To run this gear on the Flywheel instance, we will need to either create a new project and upload a "message.txt" file or upload a "message.txt" file to an existing project to use as input. When launching the gear, remember to set the name and the number of reps in the config tab. The output of this gear will show up in the gear "logs", which we can view in the provenance tab for any session, as well as in the Analysis tab, since we specified our gear to be an analysis gear in the manifest. A detailed description on how to run any gear, view its status, logs, and examine gear outputs can be found in this article.
Resources
Navigation
Previous Part 6 The Dockerfile
Next Part 8 Logging