Innovative services and tools for NorduGrid (NGIn) Nordunet 2008, Jon Kerr Nilsen j.k.nilsen[at]fys.uio.no University of Oslo, Department of Physics PhD student, NGIn

2/6/2014www.nordugrid.org2 The NorduGrid Collaboration Nordic Grid Neighbourhood A research collaboration that: Develops and supports ARC middleware Develops and supports ARC middleware Provides middleware to research groups and national Grids Provides middleware to research groups and national Grids Coordinates the various inputs to the KnowARC code Coordinates the various inputs to the KnowARC code

06/02/2014www.nordugrid.org3 ARC middleware deployed on 50+ sites in 15+ countries all around the world

Large Hadron Collider 06/02/2014www.nordugrid.org4 40 million events/s per each of 4 experiments Reduced to few hundred good events/s ~15 petabytes of data per year

European Network Backbone 06/02/2014www.nordugrid.org5 Need to store and replicate 15 PB year ~5000 scientists in ~500 institutions and universities worldwide needs access Need Grid solutions Need dedicated network GÉANT2, N-RENs & al. Strong correlation with major European LHC centers

06/02/2014www.nordugrid.org6 Nordic Data Grid Facility NDGF pilot project started 2003 Goal: Create a Nordic Grid infrastructure Builds on history and competence of Nordic Grid collaboration NorduGrid/ARC middleware chosen as basis Funded (2 M.EUR/year) by National Research Councils of the Nordic countries Coordinates and hosts major Grid projects (e.g. the Nordic LHC Tier-1) Operates Nordic storage facility for major projects Develops grid middleware and services NOS-N DK SFN S Nordic Data Grid Facility

NDGF Facility Q3

06/02/2014www.nordugrid.org8 Univ. of OsloNorway Univ. of Lund Sweden Univ. of Copenhagen Denmark Univ. of Uppsala Sweden NIIF Hungary Geneva Univ. Hospitals Switzerland Univ. of KosiceSlovakia Univ. of Lübeck Germany science+computing ag Germany SUN Microsystems Hungary NorduGrid members and research teams in medicine, bioinformatics, physics, engineering, automotive industry apps, IT EU KnowARC project: 10 partners from 7 countries

06/02/2014www.nordugrid.org9 Goals of KnowARC Experienced middleware developers Existing ARC middleware Industry, academia application developers The mission of KnowARC is to create a novel, powerful Next Generation Grid middleware –extend and re-design ARC –ARC in major Linux distributions –New platforms: Windows, Solaris, Mac OS-X to promote Grid standardization and interoperability –interoperability with other Grid solutions (EGEE and OGF compliant grids) –possibilities to include (or to be included in) other infrastructures to prove usage in Health care, Industry and Science –Bioinformatics: Autoimmune Diseases –Medical Informatics: Lung Diseases –Automotive industry

Next Generation ARC 06/02/2014www.nordugrid.org10 Developed using SOA and existing and comming standards Web Services, SOAP, GLUE Non-intrusive (no need to change excisting computing resources) Client-Server communication mainly through Web Services

DENMARK: University of Copenhagen (NBI, DIKU) Southern Danish University, Dep. of Computer Science Aalborg University, Dep. of Computer Science SWEDEN: Lund University, Department of Physics Linköping University, National Supercomputer Center Umeå University, Department of Computing Science and HPC2N Uppsala University, Department of Radiation Sciences Royal Institute of Technology, NADA/PDC, KTH Stockholm University and KTH, Department of Physics NORWAY: University of Oslo, Physics institute University of Bergen, Department of Physics University of Tromsø, Department of Computer Science NTNU, Trondheim, Department of Computer and Information Science NTNU, Trondheim, IT division (ITEA) RUSSIA: St. Petersburg State University, Dep. of Physics Petrodvorets Telecommunication Center Dubna? ESTONIA: National Institute of Chemical Physics and Biophysics (NICPB) Estonian Educational and Research Network (EENet) Tartu University, Institute of Technology FINLAND: University of Helsinki, Helsinki Institute of Physics LITHUANIA: Vilnius University, Institute of Materials Science and Applied Research LATVIA ICELAND Nordic Grid Neighbourhood... 9 countries, 24 institutions Project: …, NO27, Nabo 05-06, Nordic Grid Neighbourhood Project no.: /V11 We have read and approved your final academic and financial project report for the funding period We are pleased that most of the planned goals of the network have been reached through, i.a. workshops, conferences and tutorials. We are impressed by the number of partners and participants in the various activities.

06/02/2014www.nordugrid.org Innovative services and tools for NorduGrid. PI: Farid Ould-Saada, University of Oslo. 4 year grant Oslo, Lund, Uppsala, Copenhagen, Helsinki, Reykjavik Mainly: Data Management: storage, indexing and movement Education and expert training in Grid computing 5 PhD grants starting January 2007 Grid school call for collaboration with Baltic and Russia … ?? NGIn for Research and Education Nordunet3-funded Internet Research Program

NGIn management 06/02/2014www.nordugrid.org13 Project leader Prof. Farid Ould-Saada, Univ. of Oslo, Norway Technical coordinator Prof. Alex Read, Univ. of Oslo, Norway Steering Group Prof. Farid Ould-Saada, Univ. of Oslo, Norway Prof. Paula Eerola, Univ. of Lund, Sweden Prof. Tord Ekelöf, Univ. of Uppsala Dr. Miika Tuisku, Helsinki Institute of Physics, Finland Dr. Anders Wäänänen, Niels Bohr Institute, Denmark Participating institutions Physics Department, Univ. of Oslo, Norway Physics Department, Univ. of Lund, Sweden Physics Department, Univ. of Uppsala, Sweden Department of Radiation Sciences, Univ. of Uppsala, Sweden Niels Bohr Institute, Univ. of Copenhagen and Univ. of Aalborg, Denmark Helsinki Institute of Physics (HIP), Univ. of Helsinki, Finland CSC, Finnish IT Centre for Science, Helsinki, Finland Univ. of Iceland, Reykjavik, Iceland

NGIn - training new Grid experts 06/02/2014www.nordugrid.org14 From the original application: The goal of this project is twofold: To extend the excisting middleware and to train new Grid experts... NGIn obtained 30% of funding applied for The primary goal of the NGIn project is now to train new Grid experts Training programme includes: Grid PhD students A Nordic Grid school (first held in Copenhagen in September 2007) The NGIn activities will go in line with the other NorduGrid related projects Budget 2008: PhD students:160 k Researcer training:15 k Other expenses:3 k Total:178 k

NGIn PhD students 06/02/2014www.nordugrid.org15 Four PhD students hired, fifth student soon to be hired In chronological order: PhD 1, Jon Kerr Nilsen, Univ. of Oslo, Norway PhD 2, Mika Silander, Helsinki Institute of Physics, Finland PhD 3, Salman Toor, Univ. of Uppsala, Sweden PhD 4, Phillippe Gros, Univ. of Lund, Sweden PhD 5, to be announced soon, Univ. of Oslo, Norway The students are funded 50% by Nordunet3, 50% by local sources All the PhD projects involves data management

Jon Kerr Nilsen, Oslo 06/02/2014www.nordugrid.org16 Thesis title: Data Management in Grid Middleware Address limitations and implement solutions in data management and data handling in ARC middleware Use the solutions in ATLAS data analysis Work in collaboration with both NGIn and the Scientific Computing Group at Univ. of Oslo Current status Got familiar with ARC and data management (and Grid acronyms) Survey on data management in conjunction with Grid technology presented at NorduGrid2007 Set up a test cluster with dCache and ARC Storage Element (in collaboration with Salman Toor)

Near future Storage in conjunction with virtualization Virtualization provides possibilities to run Linux applications in Windows Challenge to get data out of virtual machine and store them securely and efficiently Client side of the next generation ARC storage solution Will work with KnowARC in implementing client- side tools for accessing data in the next generation ARC Look at data management of local versus external dCache instances in current ARC Is there any gain in prefering dCache instances in the same network as the computing resource? Jon Kerr Nilsen, Oslo 06/02/2014www.nordugrid.org17

27/03/08www.nordugrid.org18 Problem: delegation is too coarse-grained in current middleware, e.g. Users delegate all rights (Impersonation) Limited proxy certificates (exclude shell access) Delegating the least set of rights is desirable Work so far on Analysing ARC components for types of operations and objects for rights modeling Analysing SAML for extending it to support fine- grained rights expressions Evaluation of existing solutions with applicable features, e.g. SecPAL Thesis title: Fine-grained rights delegation in Grids Mika Silander, Helsinki

27/03/08www.nordugrid.org19 Potential solution candidates SAML + XACML Pros: other Grid mw committed to these standards Cons: complexity in policy definition and evaluation, fine-grained rights not modeled Security Policy Assertion Language (SecPAL) Pros: broad set of security functionality including rights restrictions, policies expressed with easily understandable syntax Cons: restrictive license and implementation languages Mika Silander, Helsinki

27/03/08www.nordugrid.org20 Near future plans Explore possibility of combining pros of the two former for ARC: Easily understandable syntax similar to SecPAL, but, Implementation relying on C/C++ and adhering to pure SAML & XACML Ensure this is compatible with the security architecture of ARC v1.x being developed Evaluate SAML + XACML solution of gLite Applicable, parts reusable?

Salman Toor, Uppsala 06/02/2014www.nordugrid.org21 Thesis title: Data indexing and data movement in grid environments Topic: Management of scientific data in Grid systems Making a Grid Storage Resource Manager A layer on top of various Storage Managers (dCache, DPM, Castor, GPFS...) A complete cost based estimation system to find the best SM amongst the SMs registered

Salman Toor, Uppsala 06/02/2014www.nordugrid.org22 Progress Deployed dCache test site in Uppsala (with a pool- node in Oslo) Started study of SRM specifications in conjunction with different Storage Managers Deployed the full Globus toolkit and implemented test services and clients Next step write a service that gather the information (owner, permissions etc) of data stored in dCache replicate the data in a different Storage Manager

Philippe Gros, Lund Physics oriented project become an expert user of AliEn more Grid oriented project work on the AliEn-ARC integration Current status: AliEn expertise still a basic user, which is already a first step (submitting jobs on the Grid) AliEn-ARC integration project formulated recently studying the current state, existing simple interface Thesis title: Analysis and management of high energy physics data with NorduGrid

NGIn Grid School 06/02/2014www.nordugrid.org24 The NGIn training programme includes: Grid PhD students A Nordic Grid school The NorduGrid 2007 Grid school was held at Niels Bohr Institute in Copenhagen September 2007 Organized by CSC, Finland More than 35 present for the introductory lectures on first day Tutorials and demos had 15 participants Tutorials in topics from High Energy Physics to bio- informatics and medical imaging From beginner to specialized user in two days! Several participants spotted on the Grid later on!

NGIn School program Day 1 (General Grid Introduction) Intro to Grid Intro to ARC First steps with ARC (tutorials) Day 2 (Specialized tutorials) HEP distributed analysis ARC in bioinformatics, ARC in medical imaging ARC-gLite interoperability Grid Job Manager Dynamic Runtime Environments Day 3 (ARC Development) Sys-admin and developer training

Next Grid School Next Grid School planned to be held during NorduGrid2008 in Budapest, Fall 2008 Similar program as in the first school See you there!

Summary 06/02/2014www.nordugrid.org27 NorduGrid is a collaboration developing and deploying the ARC middleware NDGF provides infrastructure and deploys and contributes to the current ARC KnowARC develops the next generation ARC NGIn educates and trains new Grid experts, both through PhD projects and Grid schools The PhD projects contributes to both current ARC and the next generation ARC Next Grid school in Budapest, Fall 2008

Positions available in NorduGrid! 06/02/2014www.nordugrid.org28

06/02/2014www.nordugrid.org29 NorduGrid Collaboration Official Address: NorduGrid, University of Oslo, Department of Physics Post Office box 1048, Blindern 0316 Oslo, NORWAY Tel: , Fax: Steering Board: –Prof. F. Ould-Saada (Chair), OsloProf. F. Ould-Saada –Prof. P. Eerola, Lund –Prof. T. Eckelöf, Uppsala –Dr. Miika Tsuiku, Helsinki –Dr. A. Wänänen, Copenhagen –New members (MoU to be extended): Dr. Pekka Lehtovuori, CSC, Finland NN., NDGF NN., KnowARC Open to all ARC contributors Technical contacts: –B. Konya, KnowARC –J. Kleist, NDGF –A. Read, NGIn

06/02/2014www.nordugrid.org30 Title of Contract: Grid-enabled Know-how Sharing Technology Based on ARC Services and Open Standards Acronym:KnowARC Contract Nber: Instrument: STP - Specific Targeted Research Projects (aka STREP) Thematic priority/domain: Information Society Technologies (IST) Call title:IST Call 5 Call identifier: FP IST-5 Activity:IST Advanced Grid Technologies, Systems and Services Program: FP6 Duration:36 month Start date:June 1, 2006 Community Contribution: EUR Resources:17.5 FTEs in total, 12.5 financed Coordinator:University of OsloUniversity of Oslo Contact: Prof. F. F. Official Project information