After running the BIDS Curation gear, it is likely that the gear will not successfully curate all files, since file naming and types can vary among projects. In its current form, the gear job may indicate that it completed and report no errors in its log, but it does not necessarily mean that all files in the dataset were curated. In part 2 of the tutorial, we mentioned some of these curation failures.
The recommended workflow for fixing such failures is directly below. This is likely to be an iterative process.
In order to recognize curation failures, it is integral you understand the BIDS specification and BIDS curation tools (i.e., the bids curation gear and its associated gears, the template that defines the curation rules, etc.). Below are a few resources.
The following example demonstrates how to extend a template file to fix a curation failure.
When functional scans do not have the key word “run” and there are multiple runs of the same scan, copy the entire existing rule, and edit the “Run” initialization (note that parts of the rule are replaced with “...” below).
Which template should the bids_func_filerule be copied?
This example extends the “default” template. Add “.json” to that and look in the template directory, which at this point in time is in Gitlab (note that it is currently in the “release-candidate” branch). The template already has a Run attribute, so replace that with the above starting with "Run": { and ending with the matching } character. This block of text includes both "acquisition.label" and "$run_counter".
{
"extends": "default",
"description": "Handle functional files without ‘run‘ in them",
"rules": [
"id": "bids_func_file",
"template": "func_file",
"where": {
...
}
"initialize": {
"Task": {
...
}
"Run": {
"acquisition.label": {
"$take": true,
"$format": [
{
"$replace": {
"$pattern": ".+",
"$replacement": "+"
}
}
]
},
"$run_counter": {
"key": "func.{file.info.BIDS.Task}.{file.info.BIDS.Suffix}"
}
}
}
}
]
}
To test whether a new rule fixes the intended curation failure, use a development environment that has a debugging tool. Here's an example of how to setup PyCharm and its debugger.
Clone this Gitlab repo locally, and open PyCharm.
In PyCharm, click File > Open...
Select "debug-bids-curation" folder
-
Select an interpreter. Go to Preferences > Project > Project Interpreter, select the gear icon, and then click Add...
Create a new Conda or Pipenv environment.
After creating the new environment, add the package "flywheel-bids". Click the "+" button, and search for "flywheel". Select flywheel-bids.
Under the Project pane (top-left of window), right-click on the "runCurate.py" file, and select More Run/Debug > Modify Run Configuration...
For the Parameters, it is recommended that the following be added as arguments to the main function. If only one session needs to be curated, then a session id can be passed instead of the `-g` and `p` parameters.
--api-key <your_api_key> -g <your_group> -p <your_project> --reset --pickle_tree
Click OK to finish creating the configuration.
The breakpoints below are recommended to aid with debugging. The breakpoints in "runCurate.py" and "curate_bids.py" are mostly for stepping through the code the first time, so that one can understand how it works.
-
For file "runCurate.py", set one at main()
-
Right-click on the main() function. Select Go To > Declaration or Usages to open the "curate_bids.py" file and set the following breakpoints:
-
project = get_project_tree()
-
if fw:
-
-
Navigate to the function bidsify_flywheel.process_matching_templates(), right-click on it, and select Go To > Declaration or Usages. The breakpoint below is the most important one for debugging a newly added rule. Once this breakpoint is added, you can add a conditional breakpoint to only trigger if a condition is met.
For the following example, you can edit the rule.id condition to be the newly added rule id.
("file" in context) and ("dMRI" in context["file"].data["name"]) and (rule.id == "bids_diffusion_file")-
if rule.test(context):
-
Now that the breakpoints are set, you can start debugging by right-clicking on the "runCurate.py" file, and selecting Debug runCurate.
Next re-run the BIDS curation gear. Continue this process until you have completed curation.