Gear Building Tutorial Part 2b: The Run Script and Logging
Introduction
This document provides information on running script.
Instruction Steps
Developing a Working Run File
The run file is the file that contains your run script. This is essentially the "main" section of your code. This file can be used to call other files and processes, but it must contain all commands necessary to complete execution of the gear.
Identify the Purpose of the Gear
First, you must identify exactly what your gear is going to do, what inputs it requires, and what output files you would like to generate. For this example gear, we're going to create a simple test script.
A typical test program is one that simply says "Hello, World!". Only, "Hello, World!" is a little too simple. To also give you some experience with inputs and configuration settings, we'll have it say "Hello,
Let's also add one more piece of complexity–we'll create a custom message in a text file, and have the gear print that message after all the Hellos. Let's consider these inputs and outputs in the context of a gear:
-
3 values passed to the function:
-
Your name
- The number of times to say hello
- A message.txt file
For Flywheel, since your name and the number of times to say hello are simply values you enter in, these are considered config settings. Since the message file is an actual file, this is considered an input setting.
- No output files saved, only messages printed to the screen.
Write a Run File to Run on Your Computer
When you run a gear in Flywheel, the system environment is a little different from the environment you might use on your computer. Sometimes, this can cause problems and errors that you'll need to debug when it comes time to do a test run in Flywheel. It's important to make sure that the algorithm in your script runs successfully on your machine before anything else. That way, if you encounter any errors later on in the development process, you'll know it's not your code, and is likely from environment differences (or input/output issues). The important thing is to ensure that your code itself is not the culprit.
Let's get started by creating a run file in our GearTutorial directory.
Step 1: Create your Run File
We'll start with a simple python script that takes your name and echos "Hello
#!/usr/bin/env python3
# The Shebang tells the computer what to call the file with when it runs.
# For more info:https://bash.cyberciti.biz/guide/Shebang
my\_name = "Homer Simpson" # The name we want to say hello to
num\_rep = 3 # The number of times to say hello
custom\_message = "message.txt" # A .txt file with a text message to print
while (num\_rep > 0): # While the num\_rep variable is greater than zero
print("Hello, {}!".format(my\_name)) # Print "Hello Name!" every loop
num\_rep -= 1 # Decrease the num\_rep variable by one
That takes care of the "Hello" message portion of our run script. Now we need to add the following lines to print our custom message. First, write the following code to read in a file and print the message:
message\_file = open(custom\_message,'r') # Open the file with the intent to read
print('\n') # Print a blank line to separate the message from the "hello's"
print('Custom Message:')
print('\n')
print(message\_file.read()) # Read and print the file
Save this file as run.py in your GearTutorialdirectory. Next, create the message:
- Create a file called message.txt
- Write any simple text message in it
- Save this file to your GearTutorial directory
Step 2: Run the script
Now we should be able to run this script. In a terminal window, navigate to your gear directory, and enter: python run.py
You should see the following output:
>Hello, Homer Simpson!
Hello, Homer Simpson!
Hello, Homer Simpson!
Custom Message:
This is a special message saved in the "message.txt" file.
Anyone can change this message by changing the "message.txt" file.
Oh, I see it's your birthday. Happy birthday!
It's probably not your birthday...
But that's going to freak someone out :)
By now your directory structure should look like this:
At this point, we have a working run script. However, if we make this our gear, we'll have no way of changing our name, the number of times to say hello, or the custom message file. To make this program more flexible, and run for various inputs and settings, we need to make some modifications.
Next steps: Modifying the run script to run on Flywheel
This part will require some knowledge about the manifest file. We want to change this script so that the user can input ANY name, set the script to repeat ANY number of times, and have it read ANY .txt file as the custom message.
We'll work these modifications in as we go through the next section: the manifest.
Logging
When a Flywheel gear is run, a gear job and its associated log are created. Logs are especially handy for troubleshooting errors and tracking progress. If a gear fails, the error traceback is typically (sometimes niche errors prevent the error from being logged) reported in the log automatically.
Instead of using print statements, the built-in python module logging is recommended. A minimum logging should be set; that level and all levels above will be reported (e.g., if INFO is set, DEBUG will not be reported). The table below shows all log levels, and the code below shows how to initialize logging in the run script.
| Level | Numeric value |
|---|---|
CRITICAL | 50 |
ERROR | 40 |
WARNING | 30 |
INFO | 20 |
DEBUG | 10 |
NOTSET | 0 |
import logging
#Initialize logging and set its level
logging.basicConfig()
log = logging.getLogger()
log.setLevel(logging.INFO)
If you have other modules/files that get called from the run script, the initialized logger can be imported in the following way:
Warning: too much logging!
Like many good things in life, moderation is key. Too much logging can actually overwhelm part of the Flywheel system, and that system can cause other gears to fail if it's not working properly. In addition, logging too much can also make it much more difficult to troubleshoot issues, since one now has to mine for data instead of quickly scanning through the log files.
Log files are recommended to stay below 1MB (and ideally in the < 10KB range). As a reference, a .txt file of Don Quixote is about 2MB (no images). If one wishes to track the progress of a specific process that repeats, it is recommended that one does so not too frequently, once every 1-10 seconds, depending on how long the process is expected to take/repeat.
Navigation
Part 2a: The Flywheel Environment