Using Pre-built Singularity Images (.sif) with HPC
Introduction
When a gear job is queued to run on your HPC, Flywheel's HPC Client--i.e., Cast--uses a docker image link to build a singularity image (SIF) file.
The first time that the gear is called it will create cached blobs for future runs.
While building the SIF file, more resources (e.g., RAM) and singularity tools (such as singularity's mksquashfs) are required. Rather than having such a large fluctuation of resources and tooling that may only be available in HPC development nodes but not in compute nodes, Cast can reference pre-built SIF files instead.
In its current form, the HPC Client does not have the ability to read a path to a pre-built SIF file as a gear configuration parameter, but Flywheel staff can change the docker image link (located in the backend database) that a gear uses to run a job.
Warning: changing this link (rootfs-url) to reference a local SIF file will prevent the docker version of the gear from running on your Flywheel instance.
In order to run a gear on both the HPC and the instance, you would have to create two versions of the same gear- one configured to run on the instance and one on HPC.Future developments to Cast may include a gear config option (e.g., singularity_file_path) to be used instead.
Instruction Steps
First install an HPC-compatible gear on your instance.
Upload the docker gear to your Flywheel instance, either through the gear exchange or with fw gear upload for a custom gear.
Get the gear's docker image link
The gear's docker image link can be found several ways.
From the Engine Logs of an HPC Job
If the gear has been run on your HPC, the engine logs (which are different from the gear logs that are recorded on your Flywheel instance) should contain the docker image link that was used to create the SIF file. You can search for "Command args", and it will look similar to the following (but in one line):
t=2022-03-03T04:52:05+0000 lvl=crit msg="Command args" job=6220496ff454f14006a981c4 args="[
exec
--containall
--pwd /flywheel/v0
--bind /tmp/flywheel/assets/6220496ff454f14006a981c4output-015771016:/flywheel/v0/output:rw
--bind /tmp/flywheel/assets/6220496ff454f14006a981c4-2-592537485/config.json:/flywheel/v0/config.json:rw
docker://ga.ce.flywheel.io/bids-fmriprep@sha256:2964e8b9d58ec1edc099acf9d8f8fd3fb7b994ede0bfcc98ff28150427c1b2f3
bash -c python run.py
]" caller=singularity.go:126
From the above, the corresponding docker image link is:
docker://ga.ce.flywheel.io/bids-fmriprep@sha256:2964e8b9d58ec1edc099acf9d8f8fd3fb7b994ede0bfcc98ff28150427c1b2f3
From your Flywheel Site's Database
Contact Flywheel support to provide you with the rootfs-url that points to the docker image for the gear and version that you want to change. They will need mongoDB access to your Flywheel site.
From Flywheel's Docker Hub
If the gear was installed from the exchange, then there should be a docker image on Flywheel's docker hub. Verify that the gear name and version are tagged on docker hub. As an example, here are all the tagged versions for bids-fmriprep.
Click on the version to see the docker image landing page.
If you have verified that the gear image and its version are tagged, the the docker link should be as follows:
Example:
Build the SIF file
Now that you have the docker image link, you can create the SIF file.
-
On your HPC, change directories to where you would like to place the SIF file. This directory should be readable by any HPC node (either compute or development nodes) on the HPC job scheduler/cluster (at this time, only slurm is supported) that executes your job.
-
Use the command
singularity pull <gear_name>:<gear_version>.sif <docker_url>.
Note: it can take several hours to build a SIF file from the docker image.
Example:
singularity pull bids-fmriprep::1.2.4_20.2.6.sif docker://docker.io/flywheel/bids-fmriprep:1.2.4_20.2.6
Or
singularity pull bids-fmriprep:1.2.5_20.2.6.sif docker://ga.ce.flywheel.io/bids-fmriprep@sha256:2964e8b9d58ec1edc099acf9d8f8fd3fb7b994ede0bfcc98ff28150427c1b2f3
Note: this step assumes that you already have a singularity authorization token setup.
Change the gear's rootfs-url on your instance's database
After you have created the SIF image, you can use the full file path to the SIF file (e.g./home/jiavila/sif_images/bids-fmriprep:1.2.5_20.2.6.sif) as a replacement to the gear's rootfs-url. As mentioned previously, this step can only be done by Flywheel staff and your gear will no longer be able to directly run on your Flywheel site (because the docker image is no longer referenced).