Skip to content

External Storage Providers Overview

External storage providers are used as either the source for Bulk Import or the destination for Bulk Export. Therefore, an external storage provider must be configured in order to use the Bulk Import and Bulk Export features.

Supported Technologies

Flywheel currently supports using any of the following storage technologies as external storage providers:

  • Amazon Simple Storage Service (S3),
  • Azure Blob Storage,
  • Google Cloud Storage, or
  • File System (e.g., NFS drive) mounted into the Flywheel infrastructure.

Credentials

Tip

The credentials for the storage account should grant only the minimum necessary permissions to users or developers. Ensure these credentials are rotated regularly according to your policies, with a recommended rotation period of at least every 90 days.

Use Connector's Pre-configured Credentials (Amazon and Azure only)

For Amazon and Azure storages, Flywheel offers a unique way of authenticating to cloud storage that uses pre-configured service account roles instead of Access Keys.

Benefits

Using pre-configured credentials improves overall security and reduces maintenance burden by eliminating the need for the person configuring a new External Storage registration within Flywheel to know, secure, or share secret credentials.

More details about this feature can be found in the "Use Connector's Pre-configured Credentials" option document.

You may be able to use this features immediately if:

  1. Your Flywheel site is hosted in either AWS or Azure, and
  2. Your cloud storage bucket is located in the same cloud account/subscription as your Flywheel site.

Otherwise, if your cloud storage bucket is located in a different cloud account/subscription from your Flywheel site, you may need assistance from Flywheel Support.

This feature is not available if your Flywheel site is hosted outside of AWS or Azure.

Flywheel Permissions

External Storage registrations within Flywheel have permissions that are internal to Flywheel and define how the storage is allowed to be used, specifically:

  • Flywheel is allowed to Import data from the storage
  • Flywheel is allowed to Export data to the storage

File System Permissions (File Systems only)

The following features are only available when using a File System (e.g., NFS drive) mounted into the Flywheel infrastructure as an external storage provider.

Unix Users and Groups

For File System storages, configuration options are available for specifying the Unix-style User (UID) and Groups (GID) necessary to interact with the file system. The Flywheel system switches itself to assume to these UID and GID values when interacting with file-system storages.

  • User ID (UID): Unix user who will own the files after they are written during an Export. Only numerical values are accepted. Defaults to 1000 (root) if not specified.
  • Group ID (GID): Unix group who will own the files after they are written during an Export. Only numerical values are accepted. Defaults to 1000 (root) if not specified.
  • Supplemental Groups (GIDs): Additional groups which may be required to grant sufficient permissions to interact with the files or directories on the file system. This value is entirely optional and has no affect on the ownership of files after they are written during an Export. Defaults to nothing if not specified.

Unix File Permissions

For File System storages, configuration options are available for specifying the Unix-style file permissions that are applied to the files after they are written to the storage.

Flywheel defaults to using 644 (-rw-r--r--) for file permissions, which translates to:

  • Owner (User): Read/Write
  • Owner (Group): Read
  • Everybody Else: Read

Note

File permission settings are currently only available in the following locations:

  1. When configuring an external storage via the new (BETA) CLI using the fw admin storage command, and
  2. When starting a new Bulk Export and overriding the external storage settings.

Cloud Access Policy (Cloud Storage only)

Certain cloud permissions are required depending upon which Flywheel permissions are assigned to the External Storage.

Why Delete Object permission is required

The permission to delete objects is required for Project Exports to support the "sync" workflow where the intent is to maintain a up-to-date copy of an entire Project in an external storage location.

In this case, an Export job is set to recur on a fixed schedule writing to the same location. Older files need to be cleared out in the external storage location each time the export runs in order to maintain an accurate copy of the Project when files have been deleted within Flywheel.

AWS Access Policy

The minimum S3 permissions required for all external storages are:

  • ReadWriteBucketlevel
    • s3:ListBucket,
    • s3:GetBucketLocation
  • ReadWriteObjectlevel
    • s3:GetObject*

If Flywheel Export permissions are granted for the storage, then the following additional permissions are required:

  • ReadWriteObjectlevel
    • s3:DeleteObject*,
    • s3:PutObject*

Azure Access Policy

The minimum Azure role is required for all external storages:

Status Check

When creating or modifying an External Storage, Flywheel performs a status check to ensure Flywheel is able to perform all the required actions against the storage. The status check involves:

  1. Listing the contents of the bucket
  2. Writing a (temporary) file to the bucket
    • Skipped if "Export" permissions are not granted
  3. Reading a file from the bucket
  4. Deleting the (temporary) file from the bucket
    • Skipped if "Export" permissions are not granted

Further Reading

  1. How to Configure an External Storage
  2. Pre-configured Credentials (AWS, Azure)