Astro-WISE & Grid Fokke Dijkstra – Donald Smits Centre for Information Technology Andrey Belikov – OmegaCEN, Kapteyn institute University of Groningen

Target Non-commercial consortium to provide infrastructure in Northern Netherlands for scientific and business applications Many different applications (Life sciences, Cultural heritage, Astronomy,..) Support storage and computing infrastructure Industrial partners – a.o. IBM, Oracle Collaboration with Dutch NGI BiG Grid 19/10/20152

Target infrastructure 10 PB of storage: disk & tape – IBM GPFS file system managing this Compute facilities – University cluster (~3000 cores) – New Grid cluster (~900 cores) Users – LOFAR – KIDS/Viking – Other applications 19/10/20153

Why EGI Grid? Standards based – Computing – Storage – Authorization/authentication Resources available on national and international scale Sharing resources Challenges: – Abstraction layers make software more complex – Difficult to use for end user 19/10/20154

BiG Grid Dutch NGI, part of EGI Grid infrastructure based on gLite ~ 9000 cores 4 main sites 5 PB online storage > 8 PB nearline storage 19/10/20155

Local Grid node: status New Grid cluster at University of Groningen 58 node cluster – 16 cores – 64 GB memory – 4 TB local disk 10 GbE network Connected to Target storage infrastructure Used by a.o. OmegaCEN (KIDS/VIKING), LOFAR 19/10/20156

Astro-WISE Originally: scientific information system to process wide- field astronomical images Presently: system approach to development of information system for science (WISE technology) 19/10/20157

Astro-WISE - Grid Integration Workgroup OmegaCEN: Belikov, Vriend, Boxhoorn; CIT: Dijkstra, Dröge Astro-WISE  EGI, one way full integration Access and store data on EGI, launch Astro-WISE applications on EGI Authentication & Authorization (Grid certificate + Astro-WISE A&A) How-to available on Link from Metadata database to SRM Incorporation of third-party products into Astro-WISE (jLite, MyProxy, SRM) Use of Astro-WISE data models and data processing on the Grid Multiple copies of a file (dataserver, EGI Grid) 19/10/20158

Astro-WISE – Grid integration: Features Port the system to the Grid, not an application Easy-to-use approach Distributed Processing Unit 19/10/20159

Target goals Integrate 2 grid infrastructures: Astro-WISE Grid and external Grid (currently using gLite as middleware) Provide support in the use of Grid for existing and upcoming Target projects Use experience gained in the development of new information systems Hide complexity of Grid for users Collaboration with Big Grid important 19/10/201510

Grid - Jülich - Germany Nearline storage & processing Oracle DB Tier 2 Astro-WISE view Lofar Communities (Grid interfaces) SAS Archive Code base Euro VO LOFAR: User of integration 19/10/ Big Grid – SARA Amsterdam Station data Storage + compute Target - CIT – Groningen Blue Gene Files Metadata Tier 0 Tier 1 Ingest procedure

Current status KIDS/Viking pipelines running on Grid LOFAR – Software ported to the Grid – Integration with Astro-WISE ongoing Other applications being ported – Handwritten text recognition – Other astronomy projects in near future 19/10/201512

Future Goals Improve interfaces to Target storage Improving Astro-WISE DPU capabilities Less complexity in Authorization/authentication Better support for parallel jobs on Grid necessary Global solve for 240 subbands in LOFAR Multithreaded runs MPI runs of imager Improve software deployment Data location awareness I/O latencies Target can provide a local file system to local resources Data access from/to Grid (including DBMSes) IVOA interfaces Cloud computing: feasibility study Virtualization 19/10/201513

Proposal to community Knowledge dissemination Exchange with non-astronomical communities Public access to data stored on Grid (registry, interfaces, accessibility) Regular workshops/meetings for Astro community Information about experience with middleware and third-party software 19/10/201514

Acknowledgements Enabling Grids for E-sciencE III – EU FP7: INFSO-RI Target: – Provinces of Groningen and of Drenthe – European Fund for Regional Development – Dutch Ministry of Economic Affairs, Pieken in de Delta. – Samenwerkingsverband Noord Nederland. – Target operates under the auspices of Sensor Universe Big Grid – Netherlands Organisation for Scientific Research (NWO) 19/10/201515

Relevant publications K. Begeman, A.N. Belikov, D. Boxhoorn, F. Dijkstra, E.A. Valentijn, W. J. Vriend, Z. Zhao, Merging Grid technologies, J. Grid computing, 8, 2, (2010) K. G. Begeman, A. N. Belikov, D. Boxhoorn, F. Dijkstra, H. Holties, Z. Meyer- Zhao, G. A. Renting, E.A. Valentijn, W.J. Vriend, LOFAR Information System, 27, 3, (2011) A. N. Belikov, F. Dijkstra, J. A. Gankema, J. B. A. N. van Hoof, R. Koopman, Information Systems Playground - The Target Infrastructure, Scaling Astro- WISE into the Petabyte range, Experimental Astronomy, accepted for publication (2011) 19/10/201516