Software and Computing Requirements for HPC Client System

Introduction

Flywheel’s HPC Client allows Flywheel jobs (created by running gears) to be run on HPC systems.

The main purposes of the HPC Client are to check if any HPC jobs have been queued on Flywheel, create and run executable scripts (.sh files) that submit jobs to the HPC job scheduler (e.g., Slurm), and enable communication (i.e., logging and file transfers) between Flywheel and the job being run on the HPC.

The HPC Client should be installed on a system (i.e., a computer, VM, or login/head node) that has the ability to submit jobs to the HPC job scheduler.

Compute nodes should have access to the same directories as this system. This document provides minimum software and computing requirements/recommendations for this system.

Instruction Steps

Minimum Requirements

RAM: 32 GB

After the HPC Client submits a job to the cluster, Docker containers from Flywheel must be converted into Singularity containers (SIF files) the first time a gear is run; they are cached for future runs.

Depending on the gear, this process may require up to 32 GB of RAM (e.g., bids-fmriprep).

This conversion is typically done on compute nodes, but you may want to pre-build singularity containers on the system (or head/login node) if compute nodes do not have all singularity tools required for building SIF files. Learn more about using prebuilt singularity images.

Local Storage Drive: 64 GB

This assumes that the main storage drive is mapped, and this mapped drive contains all the cached singularity images.

As mentioned above, the compute notes should have access to this same directory/mapped drive.

If no mapped drive will be used, then the local storage drive should be sufficiently large to accommodate the cached singularity files, downloaded files to be processed, intermediate files when run, etc. ( ~2 TB).

CPUs: 4

If the system is only submitting jobs to compute nodes, then 4 CPUs should be sufficient; however, more is recommended (e.g., 16) if you anticipate building SIF files on the system.

Operating System: Linux

Common Linux distributions should work with the HPC Client; testing has been done on Ubuntu, version 20.04.3 LTS (Focal Fossa).

Software

The versions below have been verified. Unless otherwise instructed, use these versions or later.

• Singularity: version 3.8.1
• Cron for setting how frequently the HPC Client searches for HPC jobs on Flywheel
• Slurm: slurm-wlm 19.05.5.
• Python 3: version 3.8.10.
• Git: version 2.25.1.