Flywheel HPC-Client: Minimum software and computing requirements

  • Updated

Purpose

Flywheel’s HPC Client allows Flywheel jobs (created by running gears) to be run on users’ HPC systems. Its main purposes are to check if any HPC jobs have been queued on Flywheel, create and run executable scripts (.sh files) that submit jobs to the HPC job scheduler (e.g., Slurm), and enable communication (i.e., logging and file transfers) between Flywheel and the job being run on the HPC.

mceclip0.png

 

The HPC Client should be installed on a system (i.e., a computer, VM, or login/head node) that has the ability to submit jobs to the HPC job scheduler. Compute nodes should have access to the same directories as this system. This document provides minimum software and computing requirements/recommendations for this system.

 

 

Minimum Requirements

RAM: 32 GB

After the HPC Client submits a job to the cluster, Docker containers from Flywheel must be converted into Singularity containers (SIF files) the first time a gear is run; they are cached for future runs.  Depending on the gear, this process may require up to 32 GB of RAM (e.g., bids-fmriprep). This conversion is typically done on compute nodes, but one might want to pre-build singularity containers on the system (or head/login node) if compute nodes do not have all singularity tools required for building SIF files. Reference this document for more.

Local storage drive: 64 GB

This assumes that the main storage drive would be mapped, and this mapped drive would contain all the cached singularity images. As mentioned above, the compute notes should have access to this same directory/mapped drive. If no mapped drive will be used, then the local storage drive should be sufficiently large to accommodate the cached singularity files, downloaded files to be processed, intermediate files when run, etc. (e.g., ~2 TB).

CPUs: 4

If the system is only submitting jobs to compute nodes, then 4 should be sufficient; however, more is recommended (e.g., 16) if one anticipates building SIF files on the system.

Operating System: Linux

Common Linux distributions should work with the HPC Client; testing has been done on Ubuntu, version 20.04.3 LTS (Focal Fossa).

Software

The versions below have been verified. Unless otherwise instructed, one should use at least these versions.

  • Singularity: version 3.8.1 
  • Cron for setting how frequently the HPC Client searches for HPC jobs on Flywheel
  • Slurm: slurm-wlm 19.05.5.
  • Python 3: version 3.8.10.
  • Git: version 2.25.1.