Using SHIWA Workflow Interoperability Tools for Neuroimaging Data Analysis Applications Vladimir Korkhov 1, Dagmar Krefting 2, Tamas Kukla 3, Gabor Terstyanszky 3, Matthan Caan 1 and Silvia Olabarriaga 1 1 Academic Medical Center of the University of Amsterdam, NL 2 Charité - Universtätsmedizin Berlin, DE 3 Centre for Parallel Computing, University of Westminster, UK EGI Community Forum, Munich, March 28, 2012

Outline Motivation SHIWA platform –Coarse-grained interoperability –SHIWA Simulation Platform: repository & portal Neuroimaging use-case –Workflow implementations –Interoperability experiments Conclusion EGI Community Forum, Munich, March 28, 2012

Motivation User perspective: applications (workflows) developed at different organizations –Share own, re-use workflows of others –Create and execute meta-workflows: workflow interoperability –Get more resources for own workflows by running on a different DCI (or several DCI at the same time) Different DCIs and workflow systems! EGI Community Forum, Munich, March 28, 2012 DCI – distributed computing infrastructure

Motivation: generic use-case EGI Community Forum, Munich, March 28, 2012 DCI A DCI B CE SE CE SE DCI – distributed computing infrastructure; CE/SE – computing/storage element

Motivation: generic use-case EGI Community Forum, Munich, March 28, 2012 DCI A DCI B CE SE CE SE combine WFs find and re-use with own data on own CE? DCI – distributed computing infrastructure; CE/SE – computing/storage element

Workflow interoperability Sharing and re-using workflows Creating and executing meta-workflows Utilizing heterogeneous DCIs (together) EGI Community Forum, Munich, March 28, 2012 SHIWA : SHaring Interoperable Workflows for Large-Scale Scientific Simulations on Available DCIs What is SHIWA?

Target: SHIWA Science Gateway EGI Community Forum, Munich, March 28, 2012 CGI – coarse-grained interoperability; FGI – fine-grained interoperability; IWIR - Interoperable Workflow Intermediate Representation

Coarse-grained interoperability CGI = Nesting of different workflow systems to achieve interoperability of execution frameworks EGI Community Forum, Munich, March 28, 2012

SHIWA Simulation Platform/CGI EGI Community Forum, Munich, March 28, 2012

SSP/CGI: Infrastructures and VOs EGI Community Forum, Munich, March 28, 2012

Use case: Neuroimaging Magnetic Resonance Imaging (MRI) Diffusion Tensor MRI (DTI) –Magnetic Resonance imaging modality enabling the identification of the orientation of human tissue. –Indirect measure of water diffusion in brain tissue –Used in comparative studies of brain diseases that are thought to cause local damage to brain tissue Data analysis workflows: –DTI-preprocessing (AMC) –FSL BedpostX (Charité and AMC) EGI Community Forum, Munich, March 28, 2012

Workflows: DTI pre-processing DTI-preprocessing (AMC) –Remove artifacts, noise, movement –In-house software based on Matlab –Inputs: DICOM, NIfTY, PARREC data –Output: multiple formats including bedpostX –Workflow MOTEUR WF engine, GWENDIA VleMed VO, ported to SHIWA VO EGI Community Forum, Munich, March 28, 2012

Workflows: BedpostX FSL BedpostX (Charité) –Reconstruct brain fibers, detect crossings –Based on FMRIB Software Library (FSL) –Inputs: image and metadata in specific formats –Outputs: directory with results and statistics –Workflows GWES and MOTEUR (AMC) MediGrid VO, ported to SHIWA VO EGI Community Forum, Munich, March 28, 2012

Interoperability scenarios 1. Publish executable workflows in the repo 2. Find and test workflows 3. Run workflows with own data 4. Create and execute meta-workflows 5. Perform parallel processing on multiple DCIs EGI Community Forum, Munich, March 28, 2012

Execution using SHIWA services Used SHIWA repository to: –Describe workflows –Share workflows Used SHIWA portal to: –Access and enact registered workflows –Compose and enact meta-workflows –Monitor workflows and meta-workflows execution –Retrieve results of the execution EGI Community Forum, Munich, March 28, 2012

Scenario 1: Publish WFs EGI Community Forum, Munich, March 28, 2012 SHIWA Repository –Provide WF description, describe inputs and outputs of WFs, upload files for implementations, and sample inputs/outputs (“configurations”)

EGI Community Forum, Munich, March 28, 2012 Scenario 2: Find and test WFs SHIWA Repository : Analyze description, inputs and outputs of published WFs SHIWA Portal : Instantiate WF from repo, execute with given sample data (inside P-GRADE workflow used as the Master WF system) 17

EGI Community Forum, Munich, March 28, 2012 Scenario 3: Run WFs with own data SHIWA Portal: Instantiate WF from SHIWA repo, execute with own data: –on the same VO/DCI –on different VO/DCI (inter-DCI transfers needed) copy data to execution VO generate WF inputs execute WF parse WF output copy data from execution VO

Scenario 4: Create and run meta-workflows EGI Community Forum, Munich, March 28, 2012 SHIWA Portal: several WFs running on different DCIs –Feed output of one to input of another –Different DCIs: data transfers of intermediate data needed execute WF1 process WF1 out copy data generate WF2 in execute WF2

Scenario 5: Parallel processing on different DCIs MOTEUR/EGI/SHIWA VO GWES/D-Grid FSL-BedpostX EGI Community Forum, Munich, March 28, 2012 –Meta-workflow with split-process-merge pattern –Processing in parallel with different implementations of WF

Scenario 5: Parallel processing on different DCIs EGI Community Forum, Munich, March 28, 2012 Inter-VO/DCI data transfers! execute WF prepare data copy data parse WF out gen WF in parse WF out collect res

Execution in SHIWA Portal EGI Community Forum, Munich, March 28, 2012

Discussion / future work Use of multiple DCIs: –handling multiple user credentials –data transfers Data exchange between sub-workflows Control of data formats Error handling Provenance EGI Community Forum, Munich, March 28, 2012

SHIWA for users Re-use others’ workflows from other WF systems/DCIs Get more resources for your WFs, transparently utilize multiple DCIs Compose heterogeneous meta-workflows for complex multi-step analysis Share your workflows with others so that they can use them right away even in different environment EGI Community Forum, Munich, March 28, 2012

Conclusion Sharing and re-use of interoperable workflows needed to promote scientific collaboration SHIWA provides a promising platform: –repository and portal –desktop and compilers coming Performed workflow interoperability experiments evaluate SHIWA solutions SHIWA is in progress –still much to be done but the current solution already facilitates workflow sharing EGI Community Forum, Munich, March 28, 2012

Acknowledgements Johan Montagnat, CNRS Tristan Glatard, CNRS Tram Truong Huu, CNRS Sarra Ben Fredj, CNRS Mario Wohlfahrt, Charite Shayan Shahand, AMC Mark Santcroos, AMC This work makes use of results produced by the SHIWA, an Integrated Infrastructure Initiative (I3) project co-funded by the European Commission (under contract number ) through the Seventh Framework Programme. EGI Community Forum, Munich, March 28, 2012

SHIWA Tutorial Friday, March & LRZ2 EGI Community Forum, Munich, March 28, 2012