Gear Building Tutorial Part 7: Running a Gear Locally

Introduction

From the previous sections, you should have a gear directory with a complete:

• run script
• manifest
• Dockerfile
• message file

In this section, we will walk through how to run our example gear locally using the new Flywheel CLI. We will also cover how to upload a gear to Flywheel using our example gear.

Instruction Steps

• Gear Building Tutorial Part 7: Running a Gear Locally
• Introduction
• Instruction Steps
  • Step 1. Using the Flywheel CLI to interact with a gear
  • Step 2. Building the gear
  • Step 3. Creating the config file
  • Step 4. Adding config options to the config file
  • Step 5. Adding inputs to the config file
  • Step 6. Preparing the gear
  • Step 7. Running the gear
  • Step 8. Checking the output
  • Step 9. Uploading a gear
  • Running Your Gear
  • Resources
  • Navigation

Step 1. Using the Flywheel CLI to interact with a gear

The command we will be using the most in this part of the Gear Building Tutorial is: fw-beta gear. This command has a lot of powerful functionality for running gears locally, uploading gears to Flywheel, and copying gears between Flywheel instances. For this tutorial part, we will be focusing on the functionality related to running gears locally, as this will be useful when testing and debugging your own gears. Towards the end, we will touch on the command option for uploading gears to Flywheel.

Step 2. Building the gear

With the new Flywheel CLI, the first step to running a gear locally (or prepare it for upload to Flywheel) is to build the gear. The command we can use is:

1

fw-beta gear build <path/to/gear>

This command is a wrapper script around Docker build with additional functionality to update the environment variable in the manifest with the environment specified in the Docker image.

To run this command for our example gear, navigate to inside the gear directory in a Terminal window and run:

1

fw-beta gear build .

Step 3. Creating the config file

Instead of being able to directly specify our input and other run.py input options, as we would if we were directly calling the script using python, we need to create a config.json file using the default config options and inputs we specified in our manifest.json file. This step is recreating what Flywheel does behind-the-scenes when running a gear on a Flywheel instance.

The command we want to use is:

1

fw-beta gear config --new

Let's check what is in our newly created config.json file and compare it to the inputs and config options we specified in our manifest.

1
2
3
4
5
6
7

{
    "config": {
        "num_rep": 5
    },
    "inputs": {},
    "destination": {}
}

We can see that it captured the "num_rep" config option correctly but is missing both the "my_name" config option under "config" and the input file under "inputs". This is because we only specified a default value for the "num_rep" option in the manifest. We did not give either the "my_name" config option or the input file a default value. So, we need to add both of these to our config file.

Step 4. Adding config options to the config file

Let's start by adding the missing "my_name" config option to our config.json file. Since this config option is quite simple, we could just manually update the config.json file ourselves by adding "my_name": "Maggie Simpson" to the existing list under "config".

However, it is useful to know that fw-beta gear config provides an option to automatically do this for us:

1

fw-beta gear config --config <KEY>=<VALUE>

The VALUE of this config option can be whatever name we want, so that is easy enough to specify. In order to know the KEY, we can open up our manifest.json file and check what we called it there. Looking under the "config" section, we can see that we called this config option, "my_name". So, instead of manually updating our config file, we can use the following to automatically add both the "my_name" config option and specify the value we want to use:

1

fw-beta gear config --config my_name="Maggie Simpson"

Now if we look at our updated config file, we should see that it looks like what is shown above, with both the "num_rep" and "my_name" config options listed.

1
2
3
4
5
6
7
8

{
    "config": {
        "num_rep": 5,
        "my_name": "Maggie Simpson"
    },
    "inputs": {},
    "destination": {}
}

Step 5. Adding inputs to the config file

While it may seem daunting to have to figure out how to specify our message.txt file as the required input file directly to the config file, fortunately, fw-beta gear config has an option that can help us out.

The --input option shown below

1

fw-beta gear config --input <KEY>=<VALUE>

will automatically update the config file with the necessary lines to specify input files for testing gears locally.

We know the VALUE of our input; that is just the path to our input text file. Since we are already inside of our gear directory in the Terminal window, we simply need to specify the name of our text file, message.txt. Similary, we can consult our manifest file to remember what we called the KEY for this input text file.

Open the manifest.json file again and scroll down to the section where we specified our input files to see that we helpfully called the KEY, message_file. Now that we know both the KEY and VALUE of our input, we can run:

1

fw-beta gear config --input message_file=message.txt

Now if we check the contents of our config file again, we can see that an input file section has been added, and that our message.txt file has been specified as the message_file. In the below example of the config file we created, we have collapsed the "object" block to save space.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

{
    "config": {
        "num_rep": 5,
        "my_name": "Maggie Simpson"
    },
    "inputs": {
        "message_file": {
            "base": "file",
            "location": {
                "name": "message.txt",
                "path": "/flywheel/v0/input/message_file/message.txt"
            },
            "object": {
                ...
            }
        }
    },
    "destination": {}
}

Step 6. Preparing the gear

Now that we have built the gear, set up the new config file, and populated the config file with our config options and input file, we need to run one last step to create the gear structure in a temporary directory on our local machine. This step handles creating the Flywheel directory structure and copies any input file(s) specified in our config file into the correct place.

Before we run this prepation step, let's make sure that our gear directory has the following files:

1
2
3
4
5
6
7

GearTutorial/  
|- run.py  
|- message.txt  
|- manifest.json  
|- Dockerfile
|- config.json
|- .input_map.json

To automatically prepare our temporary gear directory and populate our input file, we can run:

1

fw-beta gear run --prepare

This command will print an output to the terminal screen:

1
2
3
4
5
6

Building gear structure in /tmp/gear/hello-world_0.1.0.
Populating inputs.
* Found file input with source path: ~/G/message.txt, 
        copying to destination: /t/g/h/i/m/message.txt

Success! Prepared gear structure in /tmp/gear/hello-world_0.1.0    

We will need the path to the gear structure from the output, so go ahead and copy the path that comes after Prepared gear structure in. So, for our example gear, this would be: /tmp/gear/hello-world_0.1.0. Depending on your local system set up, the format of this path may be different.

Step 7. Running the gear

Now we are ready to test run our example gear for the first time! From inside our example gear directory, we can run:

1

fw-beta gear run /tmp/gear/hello-world_0.1.0

Remember to replace the path, /tmp/gear/hello-world_0.1.0, with the output from when you ran fw-beta gear run --prepare in the previous step.

If everything goes well, we should see the below screendump in our Terminal window:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

> fw-beta gear run
/Users/flywheelio/Documents/GitLab/Gear-Building-Tutorial/work does not exist, skipping.
Populating inputs.
* Found file input with source path: ~/D/G/G/message.txt,
        copying to destination: ~/D/G/G/i/m/message.txt

Creating container from homer/hello-world:0.1.0
Hello, Maggie Simpson!
Hello, Maggie Simpson!
Hello, Maggie Simpson!
Hello, Maggie Simpson!
Hello, Maggie Simpson!


This is a special message saved in the "message.txt" file.
Anyone can change this message by changing the "message.txt" file.
Oh, I see it's your birthday.  Happy birthday!  
It's probably not your birthday...
But that's going to freak someone out :)
Dumping run command in run.sh

If you are getting any other error message at this point and are unsure what it might mean, don't worry. There is a complete working copy of this example gear that you can download here. If you downloaded the zip file in Part 2, just download the zip file for this hello-world-debugged version. If you cloned the GitLab repository in Part 2, then you should just be able to checkout the hello-world-debugged branch. Re-run the fw-beta gear build . step from inside the hello-world-debugged folder, and then skip to this fw-beta gear run step to get back on track.

Step 8. Checking the output

Remember in addition to setting our run.py script to print to the terminal screen, we also set it to print out the "Hello, ..." statement to an output text file called, hello.txt. If we check in the output folder of our temporary gear structure created in Step 6, we can see our hello.txt file was successfully saved and available for us to view.

Checking the contents of our output file, we can see that the "Hello,..." statement was printed to the file the number of times specified by the num_rep config option.

1
2
3
4
5

Hello, Maggie Simpson!
Hello, Maggie Simpson!
Hello, Maggie Simpson!
Hello, Maggie Simpson!
Hello, Maggie Simpson!

Within the context of running a gear in Flywheel, any files written out to the output folder would be available under the gear's Results tab.

Step 9. Uploading a gear

The last step in building a gear is to upload our fully-functional test gear to our Flywheel instance. To upload the gear to a Flywheel instance, the key step is to make sure that we are logged into the desired Flywheel instance using fw-beta login and our API key for the desired Flywheel instance.  If you have not already created an API key, follow the steps in this section. 

Now that we are sure that we are logged into the desired Flywheel instance, we can open a terminal window in our test gear directory and type the following:

1

fw-beta gear upload

We will see a series of messages on screen indicating the status of the upload.  Gears with large docker images may take a long time to upload, however for our case this should be relatively quick.

That's it!  Navigating to our Flywheel instance, we should now see the gear under Installed Gears on the left-hand navigation bar (under the COMPUTE section), as well as in the drop-down menu when choosing a gear to run. (You may need to refresh the Flywheel instance page if you had it open before running the gear upload step.)

Here are a couple of things to note about Flywheel gears on an instance:

Running Your Gear

To run this gear on the Flywheel instance, we will need to either create a new project and upload a "message.txt" file or upload a "message.txt" file to an existing project to use as input. When launching the gear, remember to set the name and the number of reps in the config tab.  The output of this gear will show up in the gear "logs", which we can view in the provenance tab for any session, as well as in the Analysis tab, since we specified our gear to be an analysis gear in the manifest.  A detailed description on how to run any gear, view its status, logs, and examine gear outputs can be found in this article.

Resources

• Example Gear Repo
• Flywheel CLI

Navigation

Previous Part 6 The Dockerfile

Next Part 8 Logging