BIDS curation tutorial part 1: preparing data

  • Updated

Our main objective in these tutorials is to curate a dataset so that it meets the BIDS specification.  Flywheel provides a gear, BIDS Curation, that does this. Before we can run this gear, however, we must ensure that data are in the right format. In this article, we will use a sample dataset to demonstrate how to prepare data for the BIDS Curation gear.

Step 1: Download example data

  1. Download example data from the Dcm2Bids Tutorial.
  2. Click on Code and then on Download ZIP. Unzip the folder. 

Step 2: Upload data to Flywheel

  1. If it has not yet been done, create a group and project where data will be uploaded.
  2. Open Terminal or Windows command prompt
  3. Use the 'ingest' command to upload the data. Learn more about how to use the ingest dicom command.  
fw ingest dicom [path_to_In_folder] [GROUP_ID] [PROJECT_LABEL]

The example below shows the folder structure of the uploaded data. Type "y" to continue with the upload. 

(base) jesusavila@C02DX6A1MD6M In % fw ingest dicom ~/Downloads/dcm_qa_nih-master/In jiavila bids_curation_test_01  --verbose

Created             [2021-06-16 10:37:41]

 

Configuring         [2021-06-16 10:37:41]

100.0% (2s)

 

Scanning            [2021-06-16 10:37:43]

610/610 files, 16MB (2s)

 

Resolving           [2021-06-16 10:37:45]

100.0% (1s)

 

In review           [2021-06-16 10:37:46]

Hierarchy:

Maximum 100 containers are displayed.

 

`- jiavila (16MB) (using)

   `- bids_curation_test_01 (16MB) (using)

      `- DEV (16MB) (creating)

         |- 2018-09-18 11_40_23 (14MB) (creating)

         |  |- 4 - Axial EPI-FMRI (Interleaved I to S) (3.6MB / 1 file) (creating)

         |  |- 5 - Axial EPI-FMRI (Sequential I to S) (3.6MB / 1 file) (creating)

         |  |- 6 - Axial EPI-FMRI (Interleaved S to I) (3.6MB / 1 file) (creating)

         |  `- 7 - Axial EPI-FMRI (Sequential S to I) (3.6MB / 1 file) (creating)

         `- 2018-09-18 12_12_29 (1.6MB) (creating)

            |- 3 - EPI PE=AP (421KB / 1 file) (creating)

            |- 4 - EPI PE=PA (422KB / 1 file) (creating)

            |- 5 - EPI PE=RL (422KB / 1 file) (creating)

            `- 6 - EPI PE=LR (422KB / 1 file) (creating)

 

  Groups: 1

  Projects: 1

  Subjects: 1

  Sessions: 2

  Acquisitions: 8

  Files: 0

  Packfiles: 8

 

Confirm upload? (yes/no): y

  1. After data are uploaded, check Flywheel to verify the structure. 
    1. Sign in to Flywheel.
    2. Navigate to the project.
    3. Click on the Sessions tab, and then click on the Subject icon SubjectViewIcon.png
      Note: Subject "DEV" has two sessions and all acquisitions
      001_Subject_DEV.png

Step 3: Run the DICOM MR Classifier gear on all acquisitions

Now that the data are in Flywheel, we will continue preparing it there. Run the DICOM MR Classifier gear. The gear reads the DICOM tags (in the DICOM headers) to determine and create classification metadata for each file. 

  1. Click on the Session icon.
    002_Sessions.png
  2. This gear should run on all acquisitions. To do this, go to the Actions menu on the top-left, and choose Batch Run Gear. 
    006_Actions_Batch_Run_Gear.png
  3. Select Utility Gear.
  4. Choose SciTran: DICOM MR Classifier, and select the latest version from the dropdown.
    007_SciTran_DICOM_MR_Classifier.png
  5. Click the Configuration tab to see what configuration options are available for this gear.
    008_Config_Options.png
  6. Keep the defaults, and click Run Gear.  Confirm that 8 jobs will be run: 
    009_Review_Jobs.png
  7. Click the Provenance tab. If the jobs are pending, click Refresh. 011_4_Jobs_Completed.png
  8. Earlier, we confirmed that 8 jobs would run, but there are only 4 job runs here. That is because this view only shows us one session. To view the jobs for both sessions, click the Subject icon SubjectViewIcon.png.012_Subject_Icon.png
  9. Click the subject DEV to see both sessions for this subject. Now you can see all 8 jobs and the input files for each.  
    013_Subject_Mode.png
  10. Click View Log. It's a good idea to search the log for the terms error and warning.  At the very end of the log, you'll be able to see how long it took the job to run.  Note that this time does not include the time it takes to create and start the virtual machine.
    014_View_Job_Log.png
  11. Click the Sessions tab to see what the gear did. The Classification gear used DICOM tags to determine what kind of scan each acquisition is. This is set as metadata attached to each file.  015_Classifications.png
  12. To see more details, click the information icon infoicon.png to view all metadata for that scan.

    Review the Modality, Classification (with Features, Intent, Measurement, and Custom fields), and the Custom Information.  The Custom Information field includes the DICOM tags and values.  All of these metadata fields can be edited if necessary.017_File_Info_and_Metadata.png

Now that the acquisitions have all been classified and the appropriate metadata has been added, we will convert the files into NIfTI.

Step 4: Run the DCM2NIIX gear on all acquisitions

The next step is to run the DCM2NIIX gear. This will take the DICOM zip archives and create NIfTI files. 

  1. Select all acquisitions. 
  2. Select the Actions menu, and choose Batch Run Gear.018_Select_Batch_Run_Gear.png
  3. Choose Utility, and select DCM2NIIX: dcm2nii DICOM to NIfTI converter. 019_dcm2niix_Utility_Gear.png
  4. Under the Optional Files section, choose Flexible.
    021_Flexible.png
    This tells the gear how to find input files. The dcm2niix algorithm is very good at knowing what to do with lots of files.  The gear expects to run with DICOM zip archives as input files, so those files will be matched and used as an input for the gear.

  5. Keep the default configuration options. 
  6. Click Run Gear, and confirm that 8 jobs will run.
  7. Select the Provenance tab to view the status of the jobs. 022_Running.png

 

In steps 3 - 4 we manually ran two gears on all acquisitions using batch run, but you can automate this process by creating Gear Rules for your project. Learn more about how to configure Gear Rules.

Continue to BIDS curation tutorial part 2: running the BIDS Curation gear to continue getting your data into the BIDS format.