BalticGrid Focus on Cracow contributions Mariusz Witek IFJ PAN

Cracow GRID Seminar – Outline PART I – About Project ■ A bit of history ■ Basic information: partners, objectives, activities ■ Cracow contibutions ■ HEP applications ■ BaltiGrid impact on IFJ PAN ■ Future PART II – Tools development ■ BAT ■ OCM-G ■ OCM-G Java API ■ CANDLE

Cracow GRID Seminar – History ■ 2003 – white places on the EGEE map  F. Galliardi, P. Öster, M. Turala – idea of EGEE extension to Baltic countries ■ 2004 – meeting in Vilnius  A. Kusznir, P. Nowakowski – presentations about CrossGrid ■ 2005 – meeting in Tallin,  proposal preparation ■ negotiations in Brussels

Cracow GRID Seminar – BalticGrid in One Slide ■ Partners:  10 Leading institutions in six countries in the Baltic Region and Switzerland (CERN) ■ Budget:  3.0 M€ ■ Coordinator:  KTH PDC, Stockholm ■ Compute Resources:  17 resource centres ■ Duration:  30 Months  Started 1 November 2005 SA - Specific Service Activities NA - Networking Activities JRA - Joint Research Activities

Cracow GRID Seminar – BalticGrid Partners ■ Estonia  Tallinn, Tartu ■ Lithuania  Vilnius ■ Latvia  Riga ■ Poland  Kraków, Poznan ■ Switzerland  Geneva ■ Sweden  Stockholm Details on

Cracow GRID Seminar –

7 Timeline First review Second Review P12P30P P18P EAC 1 st AHM 2 nd AHM3 rd AHM Summer School Kick-off

Cracow GRID Seminar – Seven Activities Integrated Infrastructure Initiative (I3) ■ Networking Activities  NA1: Management of the I3  NA2: Education, Training, Dissemination and Outreach  NA3: Application Identification and Support  NA4: Policy and Standards Development ■ Specific Service Activities  SA1: Baltic Grid Operation  SA2: Network Resource Provisioning ■ Joint Research Activity  JRA1: Account Service Level Agreements, Markets and Dynamic Account Management NA2, NA3, SA1 – significant Cracow contributions

Cracow GRID Seminar – Education Training Dissemination and Outreach Activity leader: Zofia Mosurska

Cracow GRID Seminar – Grid Operations Activity Leader: Lauri Anton Cracow Coordinator: Marcin Radecki (Status end of 2006)

Cracow GRID Seminar – Application Identification and Support Activity Leader: Algimantas Juozapavicius Application Support Coordinator: Tomasz Szepieniec IFJ PAN: 5 persons to support ~20 users (LHCb, Atlas)

Cracow GRID Seminar – HEP at IFJ PAN: applications for BalticGrid 1. Experimental applications ready to use on GRID (ATLAS, LHCb in Krakow )  Monte Carlo production of Atlas and LHCb ► Relatively easy installation of experimental software ► Comprehensive test of LCG 2. Statistical methods of data analysis (LHC experiments)  CPU intensive studies of expected sensitivity of a given measurement

Cracow GRID Seminar – HEP at IFJ PAN: LHCb experiment

Cracow GRID Seminar – HEP at IFJ PAN: ATLAS experiment

Cracow GRID Seminar – BaltiGrid impact on IFJ PAN ■ Start of LHC early next year  Grid infrastructure has to be ready on time ■ Development of local GRID installations  Access GRID from local UI ■ Support for HEP users  Installation of experimental applications  Development and tests of user algorithms  Submit jobs to GRID – distributed analysis ■ Mini cluster (blade technology)  Purchase from associated national project  32 cores, 2 GB RAM/core, 2 TB disks  To be extended in future (local Tier 3)

Cracow GRID Seminar – Future ■ Continuation – BG4Science, proposal submitted in May  Extension of the BalticGrid infrastructure to Belarus and eventually Ukraine ► New participants (13 in total: 10 old + 3 new) ■ 2 institutions from Belarus, 1 from Lithuania  Identifying and addressing specific needs of new scientific communities such as nano-science and engineering sciences  Establishing grid services for linguistic research, Baltic Sea environmental research, data mining tools for communication modeling and bioinformatics, the project will deploy an extended and reinforced e-Infrastructure to the scientific communities within the Baltic region

Cracow GRID Seminar – Outline PART I – About Project ■ A bit of history ■ Basic information: partners, objectives, activities ■ Cracow contibutions ■ HEP applications ■ BaltiGrid impact on IFJ PAN ■ Future PART II – Tools development ■ BAT ■ OCM-G ■ CANDLE

Cracow GRID Seminar – BAT = Batch Analysis Tool ■ Developed in BG SA1 ■ Site admin needs in-depth knowledge on site usage:  are VOs getting their contracted resources?  are priorities between VO respected?  are VOs utilizing resources or just blocking (wasting) them?  are specific VO requirements respected? e.g. to run job within 30 minutes. ■ Verification of a site resource-sharing policy is troublesome currently... ■ BAT is a grid tool that collects data from sites to central database and allows querying  User specifies time period  coarse/fine grain view  Various data views ► Site overview (all VOs summarized) ► Per VO view ► Job efficiency view (in VO scope) ► Summarized view (a kind of accounting)  Shows (examples): ► jobs queued, jobs running, ► jobs arrived and sent to queue, jobs arrived and started immediately ► percentage VO/job efficiency ► CPU/walltime consumed

Cracow GRID Seminar – BAT: Some views

Cracow GRID Seminar – BAT: Plans ■ Accounting  CPU normalization  User-level data – user privacy issues ■ Other batch systems  Sun Grid Engine ■ Scalability  Data caching  Smart algorithms for aggregation and summarization ■ Interested in using or installing BAT?

Cracow GRID Seminar – Tomasz Szepieniec Tomasz Duszka Jakub Janczak in

Cracow GRID Seminar – Application Support Objectives ■ By Application Support we mean:  Organize Application Expert Group to ► Organize and initialize communication between application experts and GRID experts and speed-up application adaptation ► Provide analysis of applications to show application developers the way to the GRID  Provide useful tools and support developers with using ► Graphical user interface for grid application – Migrating Desktop ► Performance analysis tools for grid application – OCM-G and G-PM ■ DISCLAIMER: Application Support is NOT for:  organizing support for user like help desk, call center, etc. User support is parts SA1.  developing grid-enabled extensions to applications. All alterations in applications should be done by application developers. ■ ONLY THIS SUMMER! BG Summer School on Grid enabling applications for gLite environment Details soon at

Cracow GRID Seminar – ■ OCM-G and G-PM  OCM-G is a grid-enabled application monitoring system enables possibility of on-line monitoring and steering of distributed application ► Special support for performance analysis of MPI applications  G-PM – tool for performance analysis  They enable possibility to study performance bottle-necks in grid applications  newly added: ► support for IA64 ► Support for Globus 4  Developed within CrossGrid and maintained now by IFJ PAN with cooperation with CYFRONET OCM-G + G-PM = On-line Monitoring

Cracow GRID Seminar – AP1 node1 site1 Tool LM SM node2 AP2AP3 LM site2 node3 AP4 LM SM :thread_stop([a_1]) Stop :thread_stop([p_1,p_2,p_3]) :thread_stop([p_4]) :thread_stop([p_1,p_2]) :thread_stop([p_3]) Stop Architecture and Request Distribution

Cracow GRID Seminar – Extensions in OCM-G ■ Integration with gLite  New options of installing OCM-G (not only RPM-based installation) ► Scripts for installing in Shared Workspace ► Quick installation with job  gLite job ID available internally ► Process list could be obtain using it ■ New services  Listing remote directory  Downloading files (supports parts of files)  Uploading files  Running shell command on remote nodes  Monitoring of CPU usage, free memory, free disk space, open files, etc.  Forking and managing other processes (including attaching to standard I/O)

Cracow GRID Seminar – ■ Full support of Globus Toolkit 2.4 and 3 and 4 ■ MCI reactivated!  Option to compile without Globus (pure sockets) ■ Partial support of IA64 ■ MPI instrumentation based on PMPI ■ Improved management of components life-cycle  Local Monitor now can be safely disconnected and re- connected Support of Other Platforms and Features

Cracow GRID Seminar – ■ Two non-standard use cases of OCM-G ■ A toolkit that enable launching application under control of OCM-G  recompilation of application not required  application watch-dog Facing BG Application – OCM-G Frameworks

Cracow GRID Seminar – ■ GAMESS – widely use computation chemistry application ► typically long running time – lost of data possible due to failure on worker node or break queue limits ► Feature to restart computation basing output files ■ Using the framework:  normal JDL as input  automatic transformation of JDL and OCM-G environment start-up  Automatic synchronization (downloading to UI) all output sandbox GAMESS Framework

Cracow GRID Seminar – ■ DNLP – MPI-Prolog based application to natural language (Latvian) syntax analysis  Interactive usage  Single task is relatively quickly computed ■ Framework enable:  Multi-site, dynamic, interactive ‘farm’ of jobs  OCM-G is used to distribute work between worker and collect results DNLP Framework

Cracow GRID Seminar – ■ Java package enabling access to OCM-G  Using COG (Java Globus API) or pure sockets ■ Multi-layer interface:  Layer 1 – handling connections GSI/MCI and sending/receiving text-based OMIS messages  Layer 2 – stateless objects handling tokens and operates on them  Layer 3 – stateful objects representing OCM-G tokens OCM-G Java API

Cracow GRID Seminar – ■ Managing and visualization of data collected by OCM-G ■ Using OCM-G Java API ■ Extensions of G-PM functionality:  Separation of data and visualization  Easy integration with web portals.  Support of dynamic applications  Better GUI ■ Development in progress  advanced prototype planned for August CANDLE – Successor of G-PM

Cracow GRID Seminar – Summary ■ BalticGrid is progressing well ■ Even such kind of project gives opportunity to develop useful tools  BAT  OCM-G  CANDLE ■ We hope to continue in BG4Science  and develop more e.g. RAP