EUFORIA FP7-INFRASTRUCTURES , Grant Scientific Workflows Kepler and Java API 4 HPC/GRID Hands on tutorial - ITM Meeting 2009 Michal Owsiak Marcin Plociennik PSNC

Outline - Tutorial overview - Quick intro to Kepler engine - Java API 4 HPC/GRID – brief introduction - Something about the Euforia (project ;) ) - Ok., but what is Grid computing about?

Tutorial agenda Time Description 14:00 Introduction ~30' - Description of Kepler tool and Java API 4 HPC/GRID - Introduction to Euforia project 14:30 Short description of Kepler ~10' 14:40 Installation of Kepler ~10' Installation of the Kepler application basing on Gateway pre- installed components 14:50 Kepler Basics ~30' - Kepler basics and execution of simple workflows 15:30 Talk regarding certificates within GRID ~10' 15:45 Coffee break ~15' 16:00 HPC + GRID jobs using Kepler GRID/HPC extension of Kepler (introduction) ~10' 16:15 GRID/HPC related workflows ~1h 17:15 Building workflows basing on predefined components ~30' 17:45 Submitting jobs into HPC/GRID in single workflow

What should you know after this tutorial - know the general idea of Kepler - know how to build simple workflows - be aware of Java API 4 HPC/GRID - know how to start simple HPC/GRID workflows

What is Kepler Open-source: Builds on Ptolemy II Collaborators: SEEK Project, SciDAC SDM Center, Ptolemy Project, GEON Project, ROADNet Project, Resurgence Project. Kepler is supported by the NSF-funded Kepler/CORE team, which spans several of the key institutions that originated the Kepler project: UC Davis, UC Santa Barbara, and UC San Diego.UC DavisUC Santa BarbaraUC San Diego Goals: Create powerful analytical tools that are useful across disciplines: Ecology, Engineering, Geology, Physics,...

What is Kepler

- Kepler is a Java based tool for building and executing workflows - Kepler allows you to use existing elements - Kepler allows you to extend it by creating new elements - Kepler is an Open Source project (BSD License)

Can I benefit from Kepler? Kepler can support you with: - building and executing workflows - executing tasks locally - executing tasks within GRID (Java API 4 GRID) - executing tasks at HPC (Java API 4 HPC) - intuitive and User Friendly interface

What Java API 4 HPC/GRID provides Java API 4 HPC/GRID provides: - set of actors that extends Kepler with HPC access - set of actors that extends Kepler with GRID access - already existing workflows that can be modified - set of composite actors for faster workflow building

Current work and Future plans -Integration with applications -Handling new use cases/requirements -Integration of UAL -Integration with Advance visualisation

EUFORIA 14 member Institutes 3.65M€ over 36 months - Management - Training - Dissemination - Grid and HPC infra- structure & support - Code adaptation & optimization - Workflows - Visualization

Grids and e-Science 2009 Project objectives Bringing a comprehensive framework and e-infrastructure to the fusion modelling community oriented to the development of ITER physics needs with particular emphasis on Grid and HPC activities Focus area: Edge and core transport and turbulence Bringing a comprehensive framework and e-infrastructure to the fusion modelling community oriented to the development of ITER physics needs with particular emphasis on Grid and HPC activities Focus area: Edge and core transport and turbulence

Work plan outline Jan 2008 Dec 2010 Mixed Workflows Grid Testbed Deployment Grid Appliance Proof of concept runs Application Porting Standardization and integration Development and deployment Migrating

Workflows The objective is to schedule jobs on the GRID and HPC infrastructures together with jobs running on other computing facilities. launch and control jobs in a transparent manner for the users. data communication: data transfers and visualizations are required at run time and to access the experimental and simulated data. middleware connectivity for Kepler Workflow (java Library for gLite, HPC) scheduler notification tools Integration with ITM tools (UAL – Universal Access Layer)

Grids and e-Science 2009 Portal Workflow/ Code Platform Grid HPC “Gateway” Local Cluster Local Data Ext. Data USER Portal + Platform + Resources Three Tiered Approach Portal defining access to resources (certificates) Databases will generally reside centrally (on the gateway) Resource allocation and workflow scheduling thro Visualization, monitoring & steering through platform EU fusion tools: standardized data structures, data access, interface definitions, Gateway, Workflow (Kepler)

Grid Infrastructure Largely based on the developments of the Interactive European Grid project ( Advanced Scheduling Capabilities (CrossBroker) Support to parallel jobs with MPI (mpi-start) Support to interactive jobs via i2glogin RAS in combination with Migrating Desktop Full interoperability with other gLite based infrastructures Like EGEE

The EUFORIA Services in the broader view GRID and HPC infrastructures

UNiform Interface to COmputing REsources seamless, secure, and intuitive Initial development started in two German projects funded by the German ministry of education and research (BMBF) 08/1997 – 12/1999: UNICORE project Results: well defined security architecture with X.509 certificates, intuitive GUI, central job supervisor based on Codine (predecessor of SGE) from Genias 1/2000 – 12/2002: UNICORE Plus project Results: implementation enhancements (e.g. replacement of Codine by custom NJS), extended job control (workflows), application specific interfaces (plugins) Continuous development since 2002 in several EU projects Open Source community development since Summer 2004 A bit on UNICORE

More than a decade of German and European research & development and infrastructure projects Any many others, e.g. Unicore Projects UNICORE UNICORE Plus EUROGRIDGRIP GRIDSTART OpenMolGRID UniGrids VIOLADEISANextGRIDCoreGRIDD-Grid IPEGEE-IIOMII-EuropeA-WAREChemomentumeDEISA PHOSPHORUS D-Grid IP 2SmartLM PRACE D-MON DEISA2ETICS2 WisNetGrid

– Grid driving HPC Used in DEISA (European Distributed Supercomputing Infrastructure) SKIF-Grid (Russian-Belarus HPC Infrastructure) National German Supercomputing Center NIC Gauss Center for Supercomputing (Alliance of the three German HPC centers) PRACE (European PetaFlop HPC Infrastructure) – starting-up But also in non-HPC-focused infrastructures (i.e. D-Grid) Taking up major requirements from i.e. HPC users HPC user support teams HPC operations teams

Open source (BSD license) Open developer community on SourceForge Contribution with your own developments easily possible Design principles Standards: OGSA-conform, WS-RF compliant Open, extensible, interoperable End-to-End, seamless, secure and intuitive Security: X.509, proxy and VO support Workflow and application support directly integrated Variety of clients: graphical, commandline, portal, API, etc. Quick and simple installation and configuration Support for many operating and batch systems 100% Java 5 –

OK., what is grid computing? “Distributed” or “grid” computing in general is a special type of parallel computing that relies on complete computers (with onboard CPU, storage, power supply, network interface, etc.) connected to a network (private, public or the Internet) by a conventional network interface, such as Ethernet. This is in contrast to the traditional notion of a supercomputer, which has many processors connected by a local high-speed computer bus. parallel computingInternetEthernetsupercomputercomputer bus The primary advantage of distributed computing is that each node can be purchased as commodity hardware, which when combined can produce similar computing resources to a multiprocessor supercomputer, but at lower cost. This is due to the economies of scale of producing commodity hardware, compared to the lower efficiency of designing and constructing a small number of custom supercomputers. The primary performance disadvantage is that the various processors and local storage areas do not have high-speed connections. This arrangement is thus well suited to applications in which multiple parallel computations can take place independently, without the need to communicate intermediate results between processors.commodity hardware multiprocessoreconomies of scale

Setup for these tutorial We are using: Gateway machine (via NX) HPC infrastructure: – Altamira (SPain, Santander) - IBM JS20 Blades – Jump(Julich, Germany) – IBM Power6 GRID infrastructure – Infrastructure of EUFORIA and int.eu.grid (gLite- EGEE compatible) Certificates: tutorial for itut vo

Thank you for your attention