Discovery to Enterprise Data Export

Table of Contents

1. Introduction
2. Overview
3. Instruction Steps
   • Data Export Container Setup
   • Data Export
   • Monitoring Data Export
4. Resources
5. FAQs

Introduction

This document will provide the export process steps to transfer data from Discovery to Enterprise. Once data has been populated into the desired project on Discovery, it can be exported to Enterprise by launching a container on Discovery which will send the data to Enterprise via ingestion through an intermediary S3 bucket. Additionally, this document will cover all parts of the data export workflow, including setting up the export container and monitoring the export process.

Overview

This document contains details of the following:

1. Data Export Container Setup: Describes how to enable an export container on a Project in Discovery
2. Exporting Data: Describes how to perform batch launch of export jobs to begin the data transfer process
3. Monitoring Data Export: Describes how to monitor the data transfer process to Enterprise

Instruction Steps

These Instructions Steps will walk the user through three simple steps involved in exporting data from Discovery to Enterprise. The steps involved can be performed to send data from a Discovery system that has been pre-configured for data export to an Enterprise system that has been pre-configured to have appropriate permissions for the ingest service to access the exported data. Flywheel support will set this up on both systems.

Data Export Container Setup

This section describes how to enable an export container on a Project in Discovery. Once the requested data is in the appropriate Project on Discovery Admin, follow the steps below to set up the export container and transfer data to Enterprise. Please note that the Project ID on Discovery Admin should match the Destination Project ID on Enterprise. The workflow assumes that the appropriate Group and Project(s) have been pre-created on Enterprise and that any required de-identification profiles have already been set up. Neglecting to address requirements may result in failed exports.

From the Project Landing Page on Discovery Admin, select Project Settings from the Actions Menu on the right side.

 

 

Now Toggle the button to enable the Export Session Data from Discovery-Admin to Enterprise container.

 

 

Note: One will need to perform the steps above once for any new Project that is created on Discovery-Admin prior to being able to send data to Enterprise from that project.

Exporting Data 

This section describes how to batch launch export jobs to begin the data transfer process. Follow the below steps to begin exporting any number of sessions to Enterprise from Discovery.

Once the export container is enabled for the project, hover your cursor over Processing Dashboard in the Actions menu and select a Session type (for ex, say MR Sessions) to navigate to the Batch Launch Page.

 

 

From the Batch Launch Processing Page, select disc2ent from the dropdown menu.

 

 

Click the Toggle All checkbox to select all sessions or use the checkboxes to select individual sessions desired to export to Enterprise; then click the Launch job menu button.

 

 

In the Container Launch Menu, set the Group-Id field as per the Group ID for that Project on Enterprise and click Run Container(s) to start the Data Export.

 

 

Repeat the Batch Launch Process above for the other Session types as required (MR, CT, US etc.)

Monitoring Data Export 

This section demonstrates how to monitor the Export Process as data is transferred from Discovery to Enterprise. One can monitor the progress of the export process via the User Dashboard table on Discovery-Admin.