1. Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow management in Remote Instrumentation Infrastructure – based on e-VLBI experiences

2. Introduction to the e-VLBI Introduction to the e-VLBI PSNC in EXPReS - FABRIC PSNC in EXPReS - FABRIC Grid – VLBI design & technology Grid – VLBI design & technology Summary Summary Outline

3. Introduction to the e-VLBI VLBI is a technique, in which physically independent and widely separated radio telescopes observe the same region of sky simultaneously, in order to generate very high-resolution continuum and spectral-line images of cosmic radio sources Telescopes are usually separated by thousands of kilometres Data from each telescope are digitally sampled and stored locally, using high-capacity magnetic tape systems and magnetic disk-array systems Data are sent and correlated at the central point (JIVE) The total flow of data into the central processor is approximately 10- 100 Terabytes per single observation, after processing this is reduced to 10-100 Gbytes.

4. Introduction to EXPReS EXPReS – the objective is to create a production-level electronic VLBI (e-VLBI) service, in which the radio telescopes are reliably connected to the central data processor at JIVE via a high-speed optical-fibre communication network. Project Details Three years, started March 2006 International collaboration Funded at 3.9 million EUR FP6, Contract #026642

5. PSNC in EXPReS EXPReS – a Real-time e-VLBI Radio Telescope - JRA1: Future Arrays of Broadband Radio-Telescopes on Internet Computing (FABRIC) - Grid – VLBI collaboration - Grid Workflow management - Grid Routing Creating solution for incorporating Grid resources for distributed correlation using existing infrastructure.

6. Once upon a time (1) Everything was slow Telescopes collected data on tapes Sent via postal mail Hard drive arrays slightly improved the situation The entire cycle could easily require 6 months or more

7. Once upon a time (2) Hardware correlator; the EVN MkIV data correlator at JIVE dedicated, purpose designed/built hardware a super computer; ~100 T ops/sec

8. Today / In the near future Data can be transferred over the network Each stage of the process can be speeded up GRID resources Software correlator e-VLBI - electronic VLBI

9. e-VLBI System - architecture

10. e-VLBI System – control flow

11. Workflow Manager module (WFM)

12. CCF – correlator configuration file

13. WFM – phase 1 Definition of radio telescopes – automatically based on the observation schedule

14. WFM – phase 2 Definition of file servers (each file sever is responsible for capturing data from RT)

15. WFM – phase 3 Definition of correlation nodes and data flows between components

16. WFM – properties Definition of resource properties

17. WFM (1) The WFM application has several purposes: Workflow creation – a user is given a list of building blocks. Each block represents the available resource. It is the users responsibility to design and construct the experiments workflow from the available building blocks. Workflow submission – the scenario prepared in the previous step can be submitted to the system. Workflow management and monitoring – allows to manage the executed workflow.

18. WFM (2) Obrazek z WFM -> przeplywy danych

19. Link definition

20. Type definition Radio telescope File server Correlator

21. Properties definition Dynamic forms Parameters stored in Json format

22. PSNC in EXPReS EXPReS – a Real-time e-VLBI Radio Telescope - JRA1: Future Arrays of Broadband Radio-Telescopes on Internet Computing (FABRIC) - Grid – VLBI collaboration - Grid Workflow management - Grid Routing Creating solution for incorporating Grid resources for distributed correlation using existing infrastructure.

23. Thank you for your attention http://www.expres-eu.org/ d.stoklosa@man.poznan.pl EXPReS is made possible through the support of the EC, 6 th FP, Contract #026642