Conflict Handling with Bulk Imports
The Bulk Import system performs conflict detection automatically and surfaces conflict scenarios to a Data Manager for review and resolution before placing the affected data into Flywheel Core.
Detection of Conflicts
The following data conflict scenarios are automatically detected when importing files in bulk.
File Path Collision
- A file already exists with the same name at the intended destination path.
- Multiple files with the same name are being imported into the same destination path either within the same import or across multiple simultaneous imports.
Container UID Duplication (Session or Acquisition)
When locating the intended destination container path for an incoming file by matching on a container UID (Session or Acquisition):
- Multiple containers of the same type exist with the same UID in any location (pre-existing UID duplication), or
- Only one container of the same type already exists with the same UID, but
- It is located under a different parent container than the incoming data specifies (new UID duplication), or
- It has a different label than the incoming data specifies (label mismatch)
Container Metadata Conflicts (Subject, Session, or Acquisition)
When locating the intended destination container path for an incoming file by matching on a container label (Subject, Session, or Acquisition):
- Multiple containers of the same type exist in the same location with the same UID (pre-existing label duplication), or
- Only one container of the same type already exists in the same location with the same label, but
- It has a different UID than the incoming data specifies (UID mismatch)
Configurable Scope for UID Duplication
When determining Container UID duplication, the search scope is configurable to be any one of the following three options:
- Site: The same UID is used on more than container of the same type anywhere in the entire Flywheel site
- Group: The same UID is used on more than container of the same type anywhere within the same Group
- Project: The same UID is used on more than container of the same type anywhere within the same Project
The default choice is for the UID duplication scope to be set to Project.
To change the scope for UID duplication, contact Flywheel support.
Quarantining of Conflicts
By default, imports do not fail when conflicts are detected. Instead,
- Files or containers that do not cause conflicts are imported normally even if other files within the same import job do cause conflicts, and
- Files or containers that do cause conflicts are are quarantined within Flywheel but outside of the destination project, so that the pre-existing data is unaffected until a Data Manager is able to resolve the conflicts.
Tools for Resolving Conflicts
Tools are provided for data managers to resolve conflicts.
For each conflict occurrence, the system explains what caused the conflict, including the source and destination paths:
- For File Path Collisions: Source path, destination path, and existing file name
- For Container UID or Metadata duplication: Source file path, source UID, source UID, destination path, and for each conflicting file with matching UID (there may be multiple conflicting files):
- Path to existing file in Flywheel hierarchy
- UID of existing file
For each conflict occurrence, provide data managers options for resolving the conflict and clearing the occurrence from the quarantine area:
- Update Existing: Update existing item with new information
- Keep All: Keep both the existing and incoming data but without overwriting the existing data. (i.e., either allow duplication or rename the incoming data item to avoid conflict),
- For File Path Collisions, automatically append a suffix to the incoming file name to avoid the collision
- For Container Label Duplication, automatically append a suffix to the incoming container to avoid the duplication
- For Container UID Duplication, allow the UID duplication to exist by creating the additional container with the same UID
- Skip: Reject new item (i.e., ignore incoming item, leave existing item unaffected), or
- Retry: Run the conflict detection logic again for this item to determine if the conflict still exists or not (i.e., check again).
Conflicts Report
Each conflict detected during an import job run is recorded and made available in a Conflict Report for future audit purposes even after the conflict is resolved. If an action was taken to resolve a conflict, that action is also recorded in the Conflict Report.
The Conflict Report details every conflict that was detected for the associated import job along with which action was taken, if any, to resolve each conflict.
Downloading the Conflicts Report
Via the Web App
The conflict report can be found by navigating to the Bulk Import jobs list page, locating the import job, and then either:
-
Selecting the "Download Conflict Report" option in the overflow menu on the import job line, or
-
Opening the Import job details dialog from the overflow menu on the import job line and then selecting the "Download Conflict Report" button
The conflict report is available in either CSV or JSON Lines (JSONL) format.
- The CSV format can be opened in any standard spreadsheet software and may be the simplest option.
-
The JSON Lines format is best when reading the report programmatically.
Via the New (BETA) CLI
The import audit report can be downloaded via the new (BETA) CLI using the following command:
Where:
<id>
is replaced with the ID of the import job to be monitored, and<format>
is eithercsv
orjsonl
If you do not know the import job ID, see the article locating the import job ID.
Tip
The output will be written to the terminal (stdout), so you may want to redirect the output to a file for storage instead. E.g., fw-beta import get 682ccd2079fed7cd5c0cd46e --report jsonl > ~/Downloads/import-audit-report-682ccd2079fed7cd5c0cd46e.jsonl
For more information, see the new CLI documentation for import get
.
Formats for the Conflicts Report
The import conflict report is available in either CSV or JSON Lines (JSONL) format.
CSV
The CSV version of the import conflict report contains the following columns:
id
-- Unique identifier for the specific conflict occurrence.created
-- Date and time when the conflict was detected.level
-- Container level at which the conflict was found (e.g.,session
).type
-- Type of conflict detected (e.g.,multiple_uid_matches_found
).upload_path
-- Path to where the incoming file was to be placed in the Flywheel Hierarchy.resolved
-- Date and time when the conflict was resolved.resolved_by
-- User (or device) who resolved the conflict (e.g.,user jdoe@flywheel.io
).action
-- Action taken to resolve the conflict (e.g.,reject_incoming
).action_error
-- Description of error that occurred when applying the resolution action, if any.action_result
-- Result of action taken to resolve the conflict.
Import Conflict Report -- CSV Example
The following table shows an entire CSV-formatted Import Conflict Report.
id | created | level | type | upload_path | resolved | resolved_by | action | action_error | action_result |
---|---|---|---|---|---|---|---|---|---|
67fea90c3837b0d07e1ca0ba | 2025-04-15T18:44:28.314000+00:00 | file | file_name_already_exists | PGBM-017/FLYW FH MR RCBV/32602 - dT1/32602 - dT1.dicom.zip | 2025-04-15 18:46:10.225000+00:00 | user jdoe@acme.com | replace_existing | nan | PGBM-017/FLYW FH MR RCBV/32602 - dT1/32602 - dT1.dicom.zip |
67fea90c3837b0d07e1ca0bb | 2025-04-15T18:44:28.670000+00:00 | session | uid_match_belongs_to_a_different_parent | FLYW-017/RA FH MR RCBV OP/16 - T1post/16 - T1post.dicom.zip | 2025-04-15 18:48:10.090000+00:00 | user jdoe@acme.com | reject_incoming | nan | nan |
67fea9383837b0d07e1ca0bd | 2025-04-15T18:45:12.295000+00:00 | session | label_match_already_has_a_different_uid | PGBM-017/RA FH MR RCBV OP/37613 - FLAIR_reg/37613 - FLAIR_reg.dicom.zip | nan | nan | nan | nan | nan |
67fea961d8262959911ca0b2 | 2025-04-15T18:45:53.884000+00:00 | file | file_name_already_exists | PGBM-017/RA FH MR RCBV OP/37619 - nCBF_reg/37619 - nCBF_reg.dicom.zip | 2025-04-15 18:46:30.158000+00:00 | user jdoe@acme.com | replace_existing | nan | PGBM-017/RA FH MR RCBV OP/37619 - nCBF_reg/37619 - nCBF_reg.dicom.zip |
67fea9633837b0d07e1ca0bf | 2025-04-15T18:45:55.191000+00:00 | file | file_name_already_exists | PGBM-017/RA FH MR RCBV OP/37616 - T1pre_reg/37616 - T1pre_reg.dicom.zip | 2025-04-15 18:46:30.256000+00:00 | user jdoe@acme.com | replace_existing | nan | PGBM-017/RA FH MR RCBV OP/37616 - T1pre_reg/37616 - T1pre_reg.dicom.zip |
67fea98c3837b0d07e1ca0c2 | 2025-04-15T18:46:36.568000+00:00 | file | file_name_already_exists | PGBM-017/RA FH MR RCBV OP/37619 - nCBF_reg/37619 - nCBF_reg.dicom.zip | nan | nan | nan | nan | nan |
67fea98eaafa9876cd1ca0b7 | 2025-04-15T18:46:38.688000+00:00 | file | file_name_already_exists | PGBM-017/RA FH MR RCBV OP/37619 - nCBF_reg/37619 - nCBF_reg.dicom.zip | nan | nan | nan | nan | nan |
67fea9b73837b0d07e1ca0c3 | 2025-04-15T18:47:19.536000+00:00 | file | file_name_already_exists | PGBM-017/RA FH MR RCBV OP/37620 - sRCBV_reg/37620 - sRCBV_reg.dicom.zip | nan | nan | nan | nan | nan |
67fea9ba3837b0d07e1ca0c4 | 2025-04-15T18:47:22.867000+00:00 | file | file_name_already_exists | PGBM-017/RA FH MR RCBV OP/37620 - sRCBV_reg/37620 - sRCBV_reg.dicom.zip | nan | nan | nan | nan | nan |
67fea9e13837b0d07e1ca0c6 | 2025-04-15T18:48:01.992000+00:00 | file | file_name_already_exists | PGBM-017/RA FH MR RCBV OP/37621 - nRCBV_reg/37621 - nRCBV_reg.dicom.zip | nan | nan | nan | nan | nan |
67fea9e73837b0d07e1ca0c7 | 2025-04-15T18:48:07.810000+00:00 | file | file_name_already_exists | PGBM-017/RA FH MR RCBV OP/37751 - T2_reg/37751 - T2_reg.dicom.zip | nan | nan | nan | nan | nan |
67feaa0daafa9876cd1ca0bd | 2025-04-15T18:48:45.084000+00:00 | file | file_name_already_exists | PGBM-017/RA FH MR RCBV OP/5361 - Mask_Tumor/5361 - Mask_Tumor.dicom.zip | nan | nan | nan | nan | nan |
The following table shows a single line from a CSV-formatted Import Conflict Report. The line has been transposed for easier viewing.
id | 67fea90c3837b0d07e1ca0ba |
---|---|
created | 2025-04-15T18:44:28.314000+00:00 |
level | file |
type | file_name_already_exists |
upload_path | PGBM-017/FLYW FH MR RCBV/32602 - dT1/32602 - dT1.dicom.zip |
resolved | 2025-04-15 18:46:10.225000+00:00 |
resolved_by | user jdoe@acme.com |
action | replace_existing |
action_error | nan |
action_result | PGBM-017/FLYW FH MR RCBV/32602 - dT1/32602 - dT1.dicom.zip |
JSON Lines
The JSON Lines version of the import conflict report contains far more technical details about the each conflict.
Import Conflict Report -- JSONLines Example
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 |
|