Building a MATLAB Gear
Introduction
Given the prevalence of MATLAB in the scientific community, it is rather common for Flywheel users to want to develop a Flywheel Gear that can execute MATLAB code. As MATLAB users will be aware, licensing considerations are seemingly unavoidable. However, many users don't realize that you can easily (well, relatively) compile your MATLAB code and share it with the community, who can then execute the code using the freely available MATLAB Compiler Runtime. It's this method that we take advantage of to build and generate MATLAB Gears. As a developer, you can compile your MATLAB code and include the resulting binary within a Gear that is built from a base MCR image (more on that later).
General Gear building principles are still to be followed with MATLAB gears, however there are a few things to keep in mind, requirements, etc., some of which are specific to MATLAB gears, and outlined below.
Instruction Steps
General MATLAB Gear Requirements
Of your MATLAB Code
- Must have no GUI interaction. Given that FW Gears run in a "headless" environment, code that requires a GUI, or any kind of user interaction mid processing stream, are not supported.
- Must run on Linux: As the FW Gear execution environment is container based, we require that your execution environment similarly be Linux-based.
Of your Development and Execution Environment
- Linux - you must be able to run and compile the code on Linux - currently, it is not possible to cross-compile for platforms. That means that MATLAB code cannot be compiled on OSX, or WIN.
- Docker - Install instructions for Ubuntu are here. You will need Docker to build and test your Gear.
Of your Docker Image
-
You need to have (or build) docker images that contain the MCR (MATLAB Compiler Runtime). Flywheel has some images that can be used as a base image for your project. MCR images can be browsed here. To reduce your development effort, it’s recommended to have a look at those images and use one that is already available.
Important notes on the MATLAB MCR
The version of MATLAB which you use to compile the code determines the version of the MCR that should be used to execute. They must match, or it will not run.
Available MCR images via Flywheel Docker Hub
Each of the MATLAB MCRs that have been pre-built and are available to be pulled from Docker Hub are listed in the table below.
MATLAB MCR Version | Docker Image Tag (available in Docker Hub) |
2013a | flywheel/matlab-mcr:v81 |
2013b | flywheel/matlab-mcr:v82 |
2014b | flywheel/matlab-mcr:v84 |
2015b | flywheel/matlab-mcr:v90 |
2016a | flywheel/matlab-mcr:v901 |
2017a | flywheel/matlab-mcr:v92.1 |
2017b | flywheel/matlab-mcr:v93.1 |
2018b | flywheel/matlab-mcr:v95 |
2019b | flywheel/matlab-mcr:v97 |
2020b | flywheel/matlab-mcr:0.1.0_2020b |
2024a | flywheel/matlab-mcr:0.1.0_2024a |
Example MATLAB Gear
The best way to learn how to build a MATLAB Gear is to look at examples. For this purpose, we developed a very simple Gear --matlab-mcr-cow-says-- that runs a MATLAB script. It still shows how to access input files and configuration options, as well as how to add the matlab-mcr
code to the Gear Dockerfile.
Using SPM12
(Contributed by Noah Mercer)
Using SPM as part of a MATLAB gear requires a couple of extra steps. Because MATLAB gears must be stand-alone executables(SAE's), SPM will need to be compiled into your SAE. SPM requires some special configuration to run as part of an SAE, however.
The simplest way to get ready for SPM compilation is to:
- Edit
<SPM_HOME>/config/spm_make_standalone.m
and comment out the line in theCompilation
section that begins withmcc
. This will skip the final compilation step that actually builds SPM into its own SAE, which isn't necessary. - Run
spm_make_standalone
at the MATLAB prompt. This will configure SPM to allow it to run as part of your SAE.
When incorporating SPM into your SAE it is important to make sure that you exclude any SPM code meant to make it backwards compatible with versions of MATLAB earlier than the one that you're running. You'll find this code in <SPM_HOME>/external/fieldtrip/compat
, which has a series of directories named matlabltXXXXX
. The 'lt
' stands for less than and the XXXXX
is a MATLAB release, indicating that that code should only be included if you're running a version of MATLAB less than that release. Including it in your SAE with later versions of MATLAB can conflict with the Flywheel SDK. If you get a stack trace like this it might be because you've inadvertently included one of these compatibility directories:
[text]
Error using strfind
PATTERN must be a string scalar or character vector.
Error in contains (line 36)
Error in flywheel.Finder/makeArgs\_ (line 90)
Error in flywheel.Finder/find\_ (line 69)
Error in flywheel.Finder/findFirst (line 29)
Error in flywheel.Finder/subsref (line 19)
Error in flywheel.Client/subsref (line 269)
Error in run_gear (line 51)
MATLAB:string:MustBeSingleString
IMPORTANT: When you configure your SAE build in MATLAB, be sure to include in your path SPM, the Flywheel SDK (assuming that you're using it, of course) and *do not* include any compatibility directories that don't apply to your version of MATLAB.
Using PARPOOL
(Contributed by Geoff Aguirre)
Problem
Compiled MATLAB code running in a GCP VM does not make use of multiple cores when invoking the parallel pool for analysis.
Cause
The GCP VM is sustained by a single physical CPU, and hyper-threading is used to create two (virtual) cores, each with two (even more virtual) “sibling” threads by additional hyper-threading. MATLAB does not recognize the virtual cores as available for processing.
Solution
Using the MATLAB GUI interface for parpool management, create a parpool profile that allows the desired number of “workers”. (e.g., 2 or 4 workers for the default GCP VM used in Flywheel).
Export this profile to a file named (e.g.) flywheel.mlsettings
.
Within the MATLAB code, use this function to create the parpool.
The function:
- Imports the
flywheel.mlsettings
profile and makes it the default - Examines
/proc/cpuinfo
to determine the number of “cpu cores” - Calls the MATLAB function
maxNumCompThreads
with the found number of cores - Creates the
parpool
with the specified profile
In the run script that invokes the compiled MATLAB code, include the -mrcuserdata
key-value. E.g.:
where myMatlabApp
is the compiled MATLAB code and /usr/flywheel.mlsettings
is the path to the mlsettings profile as it is stored within the Gear (Docker image).
Summary
This procedure has allowed us to make use of the hyper-threaded cores found on GCP VMs. In testing, we have discovered that attempting to run a job using the set of 4 “siblings” within the default VM does not produce any further increase in speed. This may be dependent a bit on the nature of our calculation, which is almost entirely CPU dependent with little time for data I/O.