Interactive European Grid Environment for HEP Application with Real Time Requirements Lukasz Dutka 1, Krzysztof Korcyl 2, Krzysztof Zielinski 1,3, Jacek Kitowski 1,3, Renata Slota 3, Wlodzimierz Funika 3, Kazimierz Balos 1, Lukasz Skital 1, Bartosz Kryza 1, Jan Pieczykolan 1 1 ACK Cyfronet AGH 2 Institute of Nuclear Physics Polish Academy of Sciences 3 Univeristy of Science and Technology AGH Cracow Grid Workshp 2006 – Krakow

2 Outline  int.eu.grid mission  HEP application real time requirements  int.eu.grid approach to support HEP  Summary

Cracow Grid Workshp 2006 – Krakow 3 Example: Ultrasound Computer Tomography A new method of medical imaging based on the reconstruction by numerical techniques of an image, using as input the data measured by a scanner of ultrasounds which surrounds the object of interest. The application requires analyzing about 20 Gb of data, which would take order of one month in a workstation… Researchers demand resources “Interoperable production-level e-Infrastructure for demanding interactive applications to impact the daily work of researchers” Distributed Parallel (MPI) Interactive Computing & Storage at the Tera level User Friendly Access Grid Interactive Desktop int.eu.grid VISION int.eu.grid in three slides

Cracow Grid Workshp 2006 – Krakow 4 A Real Challenge!  the int.eu.grid project aims to change the way researchers can use the available e-Infrastructure, exploiting the interactivity and collaboration possibilities  Researchers need to be convinced that they can: Transfer and process gigabytes of information in minutes Foresee more complex algorithms on larger statistics, test and tune them, use more powerful visualization techniques Collaborate across the network in a rewarding mode, from sharing information to discussing and presenting remotely through enhanced videoconference environments.

Cracow Grid Workshp 2006 – Krakow 5 Interactive European Grid ( Project acronym int.eu.grid Contract number Instrument I3 Duration 2 years may ´06- april ´08 “providing transparently the researcher’s desktop with the power of a supercomputer, using distributed resources” Coordinator: CSIC, Jesús Marco, IFCA, Santander, SPAIN

Cracow Grid Workshp 2006 – Krakow 6 Summary: the int.eu.grid Mission “To deploy and operate a production-quality Grid-empowered eInfrastructure oriented to service research communities supporting demanding interactive applications.”  Deployment of e-Infrastructure Oriented to interactive use Site integration support Grid operations service  Middleware for interactivity and MPI Adapt/integrate existing middleware guarantee interoperability with EGEE  Provide a complete interactivity suite Desktop roaming access scheduler with prioritization services complex visualization.  Support for interactive applications: setup of collaborative environment and VO consideration of performance interactivity and visualization requirements identification and selection of research oriented interactive applications  Support remote collaboration activities: research, management, integration, training  Approach target research communities  Provide security measures for interactivity

Cracow Grid Workshp 2006 – Krakow 7 HEP Requirements in Brief SFI PF Local Event Processing Farms Back End Network SFOs mass storage CERN int.eu.grid infrastructure int.eu.grid PF Remote Processing Farms PF Packet Switched WAN: GEANT Switch lightpath PF 3000 events per second1.5MB each event1second to process one event When we need more power we should be able to harness grid

Cracow Grid Workshp 2006 – Krakow 8 HEP Use Case Requirments  To locate processing resources where processing tasks (PT) will be started. These tasks will be waiting to receive data and process the data once they are received.  Access to each of the CPU found should allow to deliver 1.5 MB of event data every second (the infrastructure should not propose more CPUs per site then the available access bandwidth will allow to use)  Event data transfer time should be minimal and should not interfere with processing resources allocated for processing  Sites with allocated CPUs should have recently updated data base (new updates at CERN every day or a few times per day)  The list of sites with current resources complying with the requirements should be presented to ATLAS operator and the final decision on selection of sites will be left up to him.  Operator should be able to pass approx events per second to grid infrastructure!

Cracow Grid Workshp 2006 – Krakow How to use the Grid faster?  Job submission for each event is to slow  We need interactive communication!  Pilot job idea One job to allocate a node and start PT One PT process many events Direct communication between PT and EFD Faster than job submission EFD provides event (1.5MB/event) PT responds with events analysis results (1Kb/event) Limited lifetime of PT to allow dynamic resource allocation

Cracow Grid Workshp 2006 – Krakow Proposed HEP Architecture SFI EFD Buffer PT Local PT Farm EFD Buffer PT Dispatcher Broker CE int.eu.grid HEP VO Infrastructure monitoring Application Monitoring Remote PTs proxyPT Events HEP VO Database WNs UI

Cracow Grid Workshp 2006 – Krakow 11 Development Status  HEP Application is developed independently to the project  We need to develop additional tools used to gridify application: at the moment we already designed solution and we started the implementation process very first and very simple demo will be ready in 3-4 months a bit more mature prototype should be ready before the review in may the development involves QoS issues

Cracow Grid Workshp 2006 – Krakow 12 Summary  Interactive grid environment helps us to fulfill rigorous time requirements  The intelligent distribution of event meets QoS requirements, exploits and optimize load of the system  The solution designed for HEP application is ready to be used for other applications requiring access to grid computational power in a similar way. Events wait in many queues Intelligent dispatcher has a view at the current status of computational resources occupied by previously submitted jobs, and it distributes events to best resources

Cracow Grid Workshp 2006 – Krakow 13 Thank you!