1. 1 Automatic Processing Pipelines with XNAT and REDCap Vanderbilt University 2013-08-20 Benjamin Yvernault, Bennett Landman, Brian Boyd,

2. Requirements At this point, you should know how to : –Create project / sharing data –Browse data and view image previews –Search for projects, subjects, and data –Download imaging data, processed, and stored meta-data –Delete scans / sessions / projects –Use the Pre-Archive –Upload scans DICOMs –Quality Assurance 2

3. 3 After this short course, you will be able to: Create a project on REDCap Set up the design of your REDCap project Request API access for your project on REDCap What is a spider: -source code -Which spiders already exist? -Run spiders from your computer Run a spider on your local machine ACCRE: run spiders (status / failed / ….)

4. Setting REDCap project for XNAT processing 4

5. REDCap: Setup the main instrument 5 Variable Name: project_xnat -> type: Text Box Variable Name: subject_xnat -> type: Text Box Variable Name: experiment_xnat -> type: Text Box Variable Name: scan_xnat -> type: Text Box Variable Name: scan_type_xnat -> type: Text Box Variable Name: scan_sd_type -> type: Text Box Variable Name: script_name -> type: Text Box Variable Name: script_version -> type: Text Box Variable Name: pdf_report -> type: File Upload

6. REDCap: Upload main instrument 6 Variable Name: project_xnat -> type: Text Box Variable Name: subject_xnat -> type: Text Box Variable Name: experiment_xnat -> type: Text Box Variable Name: scan_xnat -> type: Text Box Variable Name: scan_type_xnat -> type: Text Box Variable Name: scan_sd_type -> type: Text Box Variable Name: script_name -> type: Text Box Variable Name: script_version -> type: Text Box Variable Name: pdf_report -> type: File Upload Take time to create all this fields each time you have a new project. Upload the main instrument dictionary as a first step. File location: http://xnat.vanderbilt.eduhttp://xnat.vanderbilt.edu -> Everything about spiders -> How to upload to REDCap -> link with the csv file.

7. REDCap: Download Shared Libraries 7

8. REDCap: API Key  To send data from XNAT to REDCap, you need to have an API key for your project on REDCap 8

9. REDCap Project Online Designer 9

10. Spiders …  Script executing a process on XNAT  Download images / data from XNAT  Execute a process on them  Copy the data to a directory  A specific spider to upload results on XNAT  All the spiders on http://xnat.vanderbilt.eduhttp://xnat.vanderbilt.edu  Source code in the project Masimatlab on NITRC  Three kind of Spiders :  Spider that creates results for a Scan  Spider that corresponds to an assessor  Infrastructure spiders 10

11. Scan Spiders  Converter Spider :  Spider_dcm2nii_all.py (using dcm2nii)  Spider_dcm2nii_all_v2.py (using r2agui)  Changing attributes:  Spider_set_scan_type.py  Spider_Series_Description.py  Extracting/creating files:  Spider_physlog.py  Spider_Make_Preview_from_nifti.py 11

12. Process Spiders  This spiders can work on a scan level (one assessor per scan) or an experiment level (one assessor per session)  Spider that generates an assessor:  Spider_fMRQA.py  Spider_VBMQA.py  Spider_dtiQA_v2.py  Spider_labelFusion.py  Spider_intra_sess_reg.py  Spider_nonrigi_reg_to_ATLAS.py  Spider_FreesurferReconAllWithQC.py  Spider_ON_segmentation.py 12

13. Infrastructure Spider  Spider_report_for_projects.py to create a report on projects (same than Xnatreport)  Spider_send_data_to_redcap.py to send data from complet processes on XNAT to REDCap  Spider_Process_Upload to upload the processes that finished. Any process Spiders will copy the results needed to a specific directory. You should run it when you need to upload processes. 13

14. Spider flow 14 Spiders Python script Any language script: - Matlab/SPM - Perl - shell script Folder UPLOAD_SPIDER_DIR Download data Run process Spider_process_upload.py / Xnatprocessupload

15. How to run a spider?  Spiders can run on your local machine if you have the requirements.  E.G: Most of the spiders use xvfb-run  python Spider_fMRIQA.py -h to see the options  Your.bashrc or.profile set up with your XNAT logs  Python packages install  Specific python package to run spiders on ACCRE develop by our lab. 15

16. ACCRE and Spiders  Advanced Computing Center for Research & Education  ACCRE is a Cluster for Vanderbilt University  Python script running every night on the cluster  Submitting one job per assessor  Status on the assessor related to the job status:  NEED_INPUTS-> the inputs files are not present on XNAT  NEED_TO_RUN-> ready to be submitted on ACCRE  JOB_RUNNING -> job submitted and running on ACCRE  JOB_FAILED -> job stop with an error  READY_TO_UPLOAD-> job ended and copy the results to the Upload directory  UPLOADING-> Spider_Process_Upload running and uploading results  READY_TO_COMPLETE-> a script will check and set the walltime and memory used by the job on ACCRE.  COMPLETE-> jobs done (processes finished and uploaded) with results on XNAT 16

17. ACCRE and CCI package 17 CCI Package Crontab job running every night a python script Folder UPLOAD_SPIDER_DIR Get assessors that need to run Run jobs for each assessors (Spider flow on ACCRE) Spider_process_upload.py / Xnatprocessupload

18. Status sync between XNAT and REDCap 18

19. 19 Interactive Assignment Add to your.bashrc the line (or.profile on Mac OS): export UPLOAD_SPIDER_DIR=path Create a project on REDCap Upload the dictionary from http://xnat.vanderbilt.eduhttp://xnat.vanderbilt.edu –Everything about Spiders -> How to upload on REDCap Download the shared libraries for fMRIQA / VBMQA / dtiQA Run the Spider VBMQA on one T1 in the project you created in the last courses (it’s going to take 10mins) Look at the results and upload them using Xnatprocessupload Delete the project