Flywheel Notebooks User Guide

Introduction

Flywheel Notebooks provide an interactive compute option that allows researchers to easily apply Python data science tools within the Flywheel platform.

Flywheel integrates with the familiar Jupyter Lab/Notebooks to combine Python code, reports, and documentation in a single document.

The notebook servers are managed within Flywheel Projects and controlled via User permissions. The compute resources are configurable and managed by Flywheel.

Flywheel Notebooks have a wide array of applications, including the following:

• Data and Image Curation
• ML and AI Training and Inference
• Analytics and Reporting
• Automate Administrative Functions

Instruction Steps

How To Enable Flywheel Notebooks on a Project

Flywheel Notebooks are enabled in the Project settings under 'Workspaces'. Once enabled project users with correct permissions can use Flywheel Notebooks.

How To Use Flywheel Notebooks

From the Flywheel project, select the "Workspaces"

Next, select Create Jupyterlab Server on the far right of the screen.

Now, name your new server.

From the server, Launch JupyterLab. There is the option of selecting one of the configured compute resources.

After the server has been launched, the following message will appear,as shown in the image below.

Now click Open on the created server.

Once a selection is made, a new compute resource will be started. A new tab will appear for JupyterLab.

One can now use JupyterLab to create a new notebook.

Once a Notebook has been created/named, you may save the notebooks and other files as required.

It may prompt for the user to rename the file as shown in the image below.

IMPORTANT: When you are finished working with your session, select Publish to Flywheel to allow future collaboration on the Jupyter home directory.

The notebook created will now display on the far left hand screen in list form as shown in the image below. Your JupyterLab session can now be closed.

Back on the Jupyterlab Workspace, you can click "Stop" to shut down the server and free up the compute and storage resources.

JupyterLab Servers

Each Flywheel project can have one or more JupyterLab Servers. Each have file storage and allow usage of JupyterLab. All users with access to the project and with the permission to Manage Workspaces can create, modify, start, or stop any of the Servers. This approach allows a project team to collaborate on a set of Servers containing Jupyter Notebooks.

Create a Jupyterlab Server

Jupyterlab Servers in Flywheel are used to launch Jupyterlab servers and also maintain the home directories of Jupyterhub servers. They support team collaboration on the server contents by providing project team members with appropriate access to any server. Team members can edit notebooks and other files and then publish those to Flywheel to share work.

Server Status

Servers show the status of the Jupyterlab servers, indicating which are running and who last published the Server contents back to Flywheel.

Downloading from Jupyterlab Server

Jupyterlab Servers allow users to download a zip archive of the Server home directory contents. This file can be used to share the contents with other researchers.

Downloading and importing Server contents

To download and import server content, please follow the workflow instructions below:

• Download selected workspace zip file to local computer

• Now go to another project, go to the Workspace Tab and create a new JupyterLab Server.

• After creating the new server, one will launch Jupyterlab

• The user will now need to upload the zip file.

• Now using the Jupyterlab Terminal, use the linux command unzip to restore the files in the new workspace.
• Publish to Flywheel to save the workspace into Flywheel, where it can then be loaded again.

Starting JupyterLab from the Jupyter Workspace

Selecting the Server to Launch

Go to the Flywheel project and select a server from the Jupyter Workspace and launch JupyterLab. A JupyterLab instance should start in a separate tab after some startup time. The file contents that was previously published should appear in the JupyterLab instance. If you happen to close the new browser window for Jupyterlab, you can reopen it, to do so, go to the associated server and launch Jupyterlab. An instance should start again with the file contents loaded.

Select the Size/Type of Server

Users can select a specific size/type of Server to create when they launch JupyterLab. The specifications of the server are provided in terms of Central Processing Unit (CPU) count, Random-Access Memory (RAM) size, and whether it has a Graphics Processing Unit (GPU).

Options will vary depending on the cloud provider and how the site is configured (contact Flywheel to learn more). See image below for specifications of the four example compute resources.

Server Startup

After a server startup, an indicator of progress while waiting for the server to start will be shown on the screen. Sites are configured to have a certain number of warm servers ready to use.

The default is 1. If a warm server is available the user will get their server more or less immediately. If there are none available a new server will be requested, in this case provisioning may take two to ten minutes, depending on the size of the server.

Servers with GPU’s are never kept warm due to cost and these servers may take more time to be provisioned, the amount of time depends on the cloud provider’s provisioning time. When the server is starting, it can’t be stopped. Once it is running, it can be stopped. The status column will show the status of the server.

• Stopped - no server is running. A server can be started.

• Is starting… no action is possible.

• Is running… a server is running and the Open button will appear.

Launching JupyterLab

Once a Server is running, click the "Open" button to launch JupyterLab in a separate browser window. If there were any files published to Flywheel from a past server session of JupyterLab, they will appear in the Jupyterlab instance.

Stopping a Server

Stopping a server causes the server to be stopped and server storage to be deleted. The Flywheel storage remains so that you can start a new server and work where you left off. The user should remember to use the JupyterLab File menu item "Publish to Flywheel" before stopping a server to ensure work is saved to the Flywheel storage. Before stopping another user’s Server, the best practice is to contact a user before stopping their server so that they do not lose their work.

Deleting a Server

Users with appropriate permissions can delete a Server to avoid wasting compute resources. To do this, follow the steps below:

1. Go to the list of Servers on the far left hand side of the screen and click on one to delete it.

2. Stop the server if it is running and then delete it. This will remove the server from the list. Deleting a server deletes the Flywheel storage and only users with the ‘Delete’ permission can delete servers.

Using JupyterLab

JupyterLab is an open source tool and there are many useful sources of material found online. The official documentation for JupyterLab is here.

You can read more about getting started with Notebooks here.

Currently Flywheel uses version v3.6.3 of JupyterLab. Flywheel’s JupyterLab kernel contains the Jupyter/scipy-notebook image along with some additional packages, such as the current version of the Flywheel SDK to streamline access to flywheel.

Logging Out

When a user logs out of Flywheel, they will also log out of the Notebook. Using the log out option in JupyterLab to log out will log you out of JupyterLabs, but not Flywheel. The notebook server will stay running and not be stopped by this action.

Note: There may be some delay in this action due to the inherent design of the Jupyter applications.

Using JupyterLab

Launching JupyterLab

The files and folders stored in the Flywheel archive are automatically uploaded into the JupyterLab server instance. This allows users to pick up where they left off using JupyterLab.

This also allows a different user to start a JupyterLab instance in a workspace using another user’s files and collaborate on notebooks and other work.

If there are no files/folders in the server it is because it was newly created, so the user will see an empty folder named "Data".

The purpose of the Data folder is that any files put in this folder are assumed to be Flywheel data or temporary files, importantly files in the "Data" folder will not be brought back to Flywheel when the user Publishes to Flywheel.

Publishing Back to Flywheel

Publishing the home directory back to Flywheel allows one to pick up where they left off on and allow other users to collaborate on them with you. Before stopping the JupyterLab server, users need to publish their work back to Flywheel in order to preserve their changes.

Under the JupyterLab File menu there is a menu item called Publish to Flywheel to save your work to Flywheel.

This will first perform a local save-all so that the user's latest edits are saved into the local files. This will also publish all the folders and files back to Flywheel, except the folder named Data.

Anything in the Data folder will be excluded. Users can use the Data folder for holding data files on a temporary basis. Since these data files are also in the Flywheel database, it may not make sense to also publish these back to Flywheel.

If users want to intentionally publish data files, then they can create another folder with a different name and it will be published back to Flywheel. Empty folders and hidden files/folders with names starting with a . will also be published to the workspace.

Relevant Permissions

The following are the current relevant permissions:

• Users with Manage Workspace permission are able to edit/create/start/stop servers. The Workspace tab and the list of JupyterLab Servers is not visible if you do not have this permission.
• Users with Workspace Delete Permission are be able to Enable workspaces and delete Jupyter Servers.
• Users with the default Admin Role have the following two permissions for this feature:
• Workspaces:
  • Manage Workspaces - Create/Modify/Start/Stop Workspaces
  • Delete Workspaces - Enable the JupyterLab workspace for a project and delete servers

Using the Python SDK and CLI within a Flywheel Notebook

The Flywheel Notebook has a few convenience features when working with the Python Software Development Kit (SDK) and Command Line Interface (CLI). A compatible version of the Flywheel Python SDK and CLI is automatically installed into the JupyterLab server for users to use.

Using the CLI in JupyterLab

1. In JupyterLab open the terminal from the Launcher page.

2. Use the following command to log in to the Flywheel instance with your Application Programming Interface (API) key:

fw login $FW\_HOSTNAME":"$FW\_WS\_API\_KEY

1. This should return -

"*You are now logged in as: (Username)!"*

1. You can now use the CLI as usual

2. Since the CLI is running on the JupyterLab server, it is in the same cloud account as the Flywheel application so this maximizes performance.

3. Using this approach for CLI operations vs your local machine means your local machine is not tied up for long operations. Users do not have to maintain the CLI on their local machine.
4. Flywheel maintains the compatibility of the CLI version and the Flywheel application version so the users does not have to.

Using the Flywheel Python SDK in Notebooks

When users are in a Notebook they can use the Flywheel SDK client without having to specify the API key. Also, Git and Conda tools are installed in the server to allow working with source code repositories and installing Conda compatible packages. There are also convenience features for accessing and using Flywheel Exports as a data source for the notebook.

Imported Flywheel SDK and Preconfigured Flywheel Client

In the notebooks, the Flywheel SDK is automatically imported and the variable fw is already initialized with user credentials, so they will not need to be provided.

Example usage:

fw.get_current_user().id 

This will return your user id.
Also in the notebook, the flywheel project is already initialized. This can be referenced by using either fw.workspace_project or fw_project in the notebook.

Using Open Source Tools Git and Conda From the JupyterLab Terminal

• The Python package installer pip is available in the JupyterLab server.
• Git is installed in the JupyterLab server
• Git is available from the JupyterLab terminal using the command line: $ git
• Conda is installed in the JupyterLab server
• Conda is available from the JupyterLab terminal using the command line: $ conda

Using Flywheel Exports with a Flywheel Notebook

Flywheel's Project Exports can also be used to make data in Flywheel accessible to the notebook via cloud storage. This requires setting up an external storage provider at the group or project level.

In the notebook there is a simplified way to access the cloud storage using the Flywheel Storage Client, which is already installed in the notebook Server.

The high level steps are as follows:

1. Define an External Storage at the group or project level for the project(s) you wish to export. It must be at the group or project level, not the site level. If more than one is selected, the first one is used.
2. Run an Export in the project to the external storage with the desired data following the instructions for Project Exports.
3. Access the External storage using the Flywheel storage client in the notebook. Access keys and authentication will be managed by Flywheel behind the scenes. See details in the section below.

Using the Flywheel Storage Client in the Notebook

The Flywheel storage client technical documentation can be found at flywheel-io / tools / lib / fw-storage · GitLab.

In the Notebook, the Flywheel storage client is already initialized as fs and has access to the defined Storage Destination.

• Example usage to write a local file to the storage bucket

  fs.set(“test/image.jpg” ,“./image.jpg”)
  

• Example usage to read a file from the storage bucket

  file  = fs.get(“test/image.jpg”)
  

Example Notebooks

Some notebooks that show basic examples of the above features can be found at: gitlab:flywheel-notebooks-getting-started.

Technical Information

Flywheel Notebooks are supported on the following providers:

• Google Cloud Platform
• Amazon Web Services
• Microsoft Azure Sites have flexibility to configure the 4 types of Jupyter server instances shown to the user at install time so that they can leverage different cloud vendors Virtual Machine (VM) families more effectively. Compute Instances:
• Supports 4 levels of server (General Purpose, Compute, Compute-GPU, Larger-Compute-GPU)
• Specs are dependent on cloud provider and account
• Single GPU
• Single Jupyter ‘kernel’ - jupyter-scipy-notebook Compute Resources:
• Separate kubernetes cluster different from Flywheel Enterprise - no resource overlap
• One warm cloud VM (w/o GPU) is ready for users to reduce wait time (site configurable)
• Wait time for VM w/ GPU and additional servers is cloud dependent (~10min)
• Idle Jupyterlab servers are shut down after 2 hrs (site configurable)

Terminology

The section provides Jupyterhub terminology.