Discovery to Core Data Export
Introduction
This document will provide the export process steps to transfer data from Discovery to Core. Once data has been populated into the desired project on Discovery, it can be exported to Core by launching a container on Discovery which will export the data to Core. Additionally, this document will cover all parts of the data export workflow, including setting up the export container and monitoring the export process.
This document contains details of the following:
- Data Export Container Setup: Describes how to enable an export container on a Project in Discovery
- Exporting Data: Describes how to set up either (i.) an event service to tigger data export or (ii) perform batch launch of export jobs to begin the data transfer process
- Monitoring Data Export: Describes how to monitor the data transfer process to Core
Instruction Steps
These Instructions for setup will walk the user through three simple steps involved in exporting data from Discovery to Core. The steps involved can be performed to send data from a Discovery system that has been pre-configured for data export to an Core system. Flywheel support will set this up for you at the outset as part of the Discovery Implementation.
Data Export Container Setup
This section describes how to enable the export container on a Project in Discovery. You may have created this project in Discovery Admin prior to pulling and linking your requested data via the Discovery Control Panel.
The workflow assumes that the appropriate Group and Project(s) have been pre-created on Core, any required de-identification profiles as well as any required gears that you wish to run on the data incoming into Core have already been set up. Neglecting to address these requirements may result in failed exports.
Once the project is created, navigate to your project on Discovery Admin, and follow the steps below to set up the export container. From the Project Landing Page on Discovery Admin, select Project Settings from the Actions Menu on the right side.
Now Toggle the button to enable the Export Session Data from FW Discovery-XNAT to FW Core container.
Note: One will need to perform the steps above once for any new Project that is created on Discovery-Admin prior to being able to send data to Core from that project.
Exporting Data
Depending on your particular use case , there are a couple of different paths you could take to export data to Core.
-
Data is de-identified on Core thus requiring a De-ID profile setup and additional validation steps prior to data export. If this is your workflow, then you may want to skip to the Batch Launch Export Container section below.
-
Data is being exported to Core without applying any de-identification on Core. If this is your preferred workflow, you should skip to the Setup Event Service to Export Data section below.
Batch Launch Export Container
If your workflow requires sending a small batch of N sessions initially to Core as a way to establish and validate a De-Identification profile that has been setup on your Core project, then you can accomplish this by using the Batch Launch workflow described below.
Once the export container is enabled for the project, hover your cursor over Processing Dashboard in the Actions menu and select a Session type (for ex, say MR Sessions) to navigate to the Batch Launch Page.
From the Batch Launch Processing Page, select export_data_disc2core from the dropdown menu.
Select "N" number of sessions to test and validate your De-ID profile on Core by checking the checkbox next to N number of sessions (10 sessions should suffice in most cases), then click the Launch job menu button.
In the Container Launch Menu, set the Project-Id and Group-Id fields as per the destination project and group IDs on Core and click Run Container(s) to begin the Data Export.
Once you've established a satisfactory De-ID profile on Core , you can use the Batch Launch feature to filter and select the remaining sessions to export to Core.
TIP: For example, sessions that are ready to be sent can be filtered using the 'Ready' keyword, followed by then selecting all the filtered sessions using the 'Toggle all' checkbox as shown below.
TIP : Sessions already exported to Core will show up as 'Complete'. If any failures occur, those sessions can be selected by filtering using the 'Failed' keyword and re-exported.
TIP: Repeat the Batch Launch Process above for the other Session modalities as required (MR, CT, US etc.)
### Setup Event Service to Export Data
If your workflow requires that data being exported to Core need not be de-identified on Core, then you can avail the Discovery Admin's Event Service feature to automatically route data to Core when doing a data request.
Once you've set up your project and enabled the export container as described in the section titled
Data Export Container Setup,you can navigate to the Administer - > Event Service setup page. Under Event Subscriptions,click on Add New Event Subscriptionand set up as per your requirements. Below is an example of an event service that will automatically launch and run the export_data_disc2core container on any sessions that are sent to the BANDWIDTH3 project. Additionally it also has Core routing attributes set up to route to the appropriate group-id and project-id on Core.
Once the above Event Subscription rule is setup and enabled, any data requested and pulled into the BANDWIDTH3 project (in this example), will be automatically sent to Core!
TIP: If you want to auto-export to work as mentioned above, you would need to set up the event subscription rule prior to submitting you data request via the Discovery Control Panel!
Monitoring Data Export
This section demonstrates how to monitor the Export Process as data is transferred from Discovery to Core. One can monitor the progress of the export process via the User Dashboard table on Discovery-Admin.
Jobs that are queued for export will have Pending status, and will switch to Running status once data export begins. Once session data export is successfully finished, the job status will switch to Complete.
Question
1. Could a user automate data exports from Discovery to Core even when needing to first validate a De-ID profile on Core?
Yes, to some extent! The user could follow a “hybrid” approach for this which combines an initial Batch Launch, followed by an subsequent Event Subscription triggered export. Say you have a large data request , but first need to validate the De-ID profile you set up on Core.
One could: * Create a separate, small, data request with a sub-sample of 10 or 20 of your studies (sessions) * Once this subset of sessions is linked to a project, use the Batch Launch Export Container method described above to send those sessions and validate De-ID profile on Core. * Rinse and repeat above two steps until a satisfactory De-ID profile exists on Core destination project * Enable event service auto transfer for your project by following steps highlighted in the Setup Event Service to Export Data section on this page. * Submit your larger data request. Data will automatically get exported to Core as per your Event Subscription settings when the data is pulled and linked to your project via the Discovery Control Panel.
2. How can I track data exports?
Data exports status can be monitored on Discovery as described in the section above titled Monitoring Data Export
Note that session level 'audit logs' with details about the export are stored as CSV files saved under Project Resource on Discovery. From the Project Landing Page on Discovery Admin, select Manage Files from the Actions Menu on the right side. You will find the audit logs as in a folder called 'core-export-audit-logs' as shown below.
Additionally, you may also want to track the gear jobs on the Core instance for those cases when you have gear rules setup to apply to any incoming data (as is very commonly done!)