Claudio Grandi INFN Bologna Virtual Pools for Interactive Analysis and Software Development through an Integrated Cloud Environment Claudio Grandi (INFN.

Slides:



Advertisements
Similar presentations
CHEP 2000, Roberto Barbera Roberto Barbera (*) GENIUS: a Web Portal for the GRID Meeting Grid.it, Bologna, (*) work in collaboration.
Advertisements

Andrew McNab - Manchester HEP - 17 September 2002 Putting Existing Farms on the Testbed Manchester DZero/Atlas and BaBar farms are available via the Testbed.
LNL CMS M.Biasotto, Bologna, 29 aprile LNL Analysis Farm Massimo Biasotto - LNL.
Workload Management Workpackage Massimo Sgaravatto INFN Padova.
IT:Network:Applications VIRTUAL DESKTOP INFRASTRUCTURE.
Workload Management Massimo Sgaravatto INFN Padova.
Idle virtual machine detection in FermiCloud Giovanni Franzini September 21, 2012 Scientific Computing Division Grid and Cloud Computing Department.
The SAM-Grid Fabric Services Gabriele Garzoglio (for the SAM-Grid team) Computing Division Fermilab.
Cyberaide Virtual Appliance: On-demand Deploying Middleware for Cyberinfrastructure Tobias Kurze, Lizhe Wang, Gregor von Laszewski, Jie Tao, Marcel Kunze,
STRATEGIES INVOLVED IN REMOTE COMPUTATION
Nimbus & OpenNebula Young Suk Moon. Nimbus - Intro Open source toolkit Provides virtual workspace service (Infrastructure as a Service) A client uses.
Test Of Distributed Data Quality Monitoring Of CMS Tracker Dataset H->ZZ->2e2mu with PileUp - 10,000 events ( ~ 50,000 hits for events) The monitoring.
OSG Site Provide one or more of the following capabilities: – access to local computational resources using a batch queue – interactive access to local.
1 Evolution of OSG to support virtualization and multi-core applications (Perspective of a Condor Guy) Dan Bradley University of Wisconsin Workshop on.
Grid Workload Management & Condor Massimo Sgaravatto INFN Padova.
Group 1 : Grid Computing Laboratory of Information Technology Supervisors: Alexander Ujhinsky Nikolay Kutovskiy.
WNoDeS – Worker Nodes on Demand Service on EMI2 WNoDeS – Worker Nodes on Demand Service on EMI2 Local batch jobs can be run on both real and virtual execution.
Cracow Grid Workshop October 2009 Dipl.-Ing. (M.Sc.) Marcus Hilbrich Center for Information Services and High Performance.
Tool Integration with Data and Computation Grid GWE - “Grid Wizard Enterprise”
Giuseppe Codispoti INFN - Bologna Egee User ForumMarch 2th BOSS: the CMS interface for job summission, monitoring and bookkeeping W. Bacchi, P.
Architecture and ATLAS Western Tier 2 Wei Yang ATLAS Western Tier 2 User Forum meeting SLAC April
Fabric Monitoring at the INFN Tier1 Felice Rosso on behalf of INFN Tier1 Joint OSG & EGEE Operations WS, Culham (UK)
Evolution of a High Performance Computing and Monitoring system onto the GRID for High Energy Experiments T.L. Hsieh, S. Hou, P.K. Teng Academia Sinica,
NA61/NA49 virtualisation: status and plans Dag Toppe Larsen CERN
Proposal for a IS schema Massimo Sgaravatto INFN Padova.
GLIDEINWMS - PARAG MHASHILKAR Department Meeting, August 07, 2013.
Trusted Virtual Machine Images a step towards Cloud Computing for HEP? Tony Cass on behalf of the HEPiX Virtualisation Working Group October 19 th 2010.
Claudio Grandi INFN Bologna CMS Computing Model Evolution Claudio Grandi INFN Bologna On behalf of the CMS Collaboration.
Grid and Cloud Computing Alessandro Usai SWITCH Sergio Maffioletti Grid Computing Competence Centre - UZH/GC3
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE Site Architecture Resource Center Deployment Considerations MIMOS EGEE Tutorial.
| nectar.org.au NECTAR TRAINING Module 5 The Research Cloud Lifecycle.
VO Box Issues Summary of concerns expressed following publication of Jeff’s slides Ian Bird GDB, Bologna, 12 Oct 2005 (not necessarily the opinion of)
Korea Workshop May GAE CMS Analysis (Example) Michael Thomas (on behalf of the GAE group)
© 2006 Open Grid Forum INFN WNoDes and OCCI Michele Orru’, INFN-CNAF.
Workload management, virtualisation, clouds & multicore Andrew Lahiff.
Tool Integration with Data and Computation Grid “Grid Wizard 2”
Grid Workload Management (WP 1) Massimo Sgaravatto INFN Padova.
OpenStack Chances and Practice at IHEP Haibo, Li Computing Center, the Institute of High Energy Physics, CAS, China 2012/10/15.
INFN OCCI implementation on Grid Infrastructure Michele Orrù INFN-CNAF OGF27, 13/10/ M.Orrù (INFN-CNAF) INFN OCCI implementation on Grid Infrastructure.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Demonstration StratusLab First.
II EGEE conference Den Haag November, ROC-CIC status in Italy
WNoDeS – a Grid/Cloud Integration Framework Elisabetta Ronchieri (INFN-CNAF) for the WNoDeS Project
Instituto de Biocomputación y Física de Sistemas Complejos Cloud resources and BIFI activities in JRA2 Reunión JRU Española.
INFN/IGI contributions Federated Clouds Task Force F2F meeting November 24, 2011, Amsterdam.
Update on Computing/Cloud Marco Destefanis Università degli Studi di Torino 1 BESIII Ferrara, Italy October 21, 2014 Stefano Bagnasco, Flavio Astorino,
DGAS Distributed Grid Accounting System INFN Workshop /05/1009, Palau Giuseppe Patania Andrea Guarise 6/18/20161.
Trusted Virtual Machine Images the HEPiX Point of View Tony Cass October 21 st 2011.
WP5 – Infrastructure Operations Test and Production Infrastructures StratusLab kick-off meeting June 2010, Orsay, France GRNET.
EMI is partially funded by the European Commission under Grant Agreement RI Elisabetta Ronchieri, INFN CNAF Munich, 29 March 2012 WNoDeS Tutorial.
Building on virtualization capabilities for ExTENCI Carol Song and Preston Smith Rosen Center for Advanced Computing Purdue University ExTENCI Kickoff.
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Introduction Salma Saber Electronic.
Enabling Grids for E-sciencE Claudio Cherubino INFN DGAS (Distributed Grid Accounting System)
Claudio Grandi INFN Bologna Workshop congiunto CCR e INFNGrid 13 maggio 2009 Le strategie per l’analisi nell’esperimento CMS Claudio Grandi (INFN Bologna)
Dynamic Extension of the INFN Tier-1 on external resources
Workload Management Workpackage
Grid Computing: Running your Jobs around the World
SuperB – INFN-Bari Giacinto DONVITO.
First proposal for a modification of the GIS schema
Sviluppi in ambito WLCG Highlights
Blueprint of Persistent Infrastructure as a Service
Belle II Physics Analysis Center at TIFR
StratusLab Final Periodic Review
StratusLab Final Periodic Review
Distributed Job Submission in a Dynamic Virtual Environment
Architecture & System Overview
Artem Trunov and EKP team EPK – Uni Karlsruhe
Virtualization in the gLite Grid Middleware software process
CernVM Status Report Predrag Buncic (CERN/PH-SFT).
Support for ”interactive batch”
Module 01 ETICS Overview ETICS Online Tutorials
Presentation transcript:

Claudio Grandi INFN Bologna Virtual Pools for Interactive Analysis and Software Development through an Integrated Cloud Environment Claudio Grandi (INFN Bologna) A.Italiano and D.Salomoni (INFN-CNAF)

Claudio Grandi INFN Bologna Introduction Most sites that offer remote access also have local users that need: –interactive access –local access to the batch system –user interfaces to access the distributed infrastructure A classic model would be to statically allocate machines for the interactive access –These machines would have local access to the batch system and user interface software –In the worst case if special configurations are needed for the batch nodes used locally also the local batch farm nodes would need to be allocated statically We describe here a solution based on virtualization to dynamically allocate resources to different tasks regardless of the needed configuration 19 October 2010 CHEP10 - Taipei 2

Claudio Grandi INFN Bologna Classical model 19 October 2010 CHEP10 - Taipei /home /data SE bastion CE interactive login local submission grid submission desktop /opt interactive pool local batch farm grid worker nodes 3

Claudio Grandi INFN Bologna Dynamic model 19 October 2010 CHEP10 - Taipei bastion CE desktop /home /data SE /opt mounts as from machine profile Idle slots are not allocated to any specific task 4

Claudio Grandi INFN Bologna Description of the project Build on the experience of the WNoDeS (Worker Nodes on Demand Service) project –Instantiate virtual machines (VM) with the desired profile on demand Including VO specific customization, e.g. file system mounts, etc... –The batch system steers resources allocation –In production at the INFN-CNAF Tier-1 with 2000 VMs A specific client is installed on the local desktops and bastion hosts –Instantiate the VM with the desired number of cores and memory Throttle bandwidth to avoid abuse by individual users –Open an interactive shell on that VM When exiting the machine is destroyed Tests at INFN-CNAF on the INFN-Bologna Tier-3 19 October 2010 CHEP10 - Taipei 5

Claudio Grandi INFN Bologna INFN-Bologna Tier-3 Joint project of INFN and Università di Bologna –Supports local research groups including LHC experiments Integrated with the INFN-CNAF Tier-1 infrastructure –~ 50 dual quad-core boxes accessible through CNAF-LSF –150 TB on CNAF-GPFS –The boundaries of the Tier-1 and Tier-3 are virtual Each site can expand into the other according to policies in LSF The profile of the virtual machine defines to which site it belongs –Computing and Storage Elements are independent Differently from the Tier-1 also offers interactive services to local users Currently in a prototype phase 19 October 2010 CHEP10 - Taipei 6

Claudio Grandi INFN Bologna WNoDeS 19 October 2010 CHEP10 - Taipei WNoDeS Interactive Virtual Interactive Pools (VIP) Batch Job Execution Cloud Computing Grid Integration Grid jobs executed on virtual machines instantiated using the runtime environment required in the job JDL (CREAM CE) Batch job execution on real or virtual machines instantiated on demand On demand computing via OCCI API and AuthN + AuthZ layer Interactive access to virtual machines instantiated on demand 7

Claudio Grandi INFN Bologna Virtual Interactive Pools The VIP command line interface is build on top of the WNoDeS cloud computing interface The user requests the instantiation of a VM via the VIP CLI –The CLI allows to specify the requirements (RAM, CPU, bandwidth, mount points) –It is possible to wait until the login is possible or to get back a request ID that may be used to check the status of the request and eventually login on the virtualized resource The request goes through the authN and authZ layer of WNoDeS WNoDeS submits a fake batch job with the specified requirements The batch job is dispatched on the WNoDeS bait host –The bait host runs a process that “attracts” jobs On the bait host the batch job execution is immediately suspended and the process to instantiate the VM starts When the VM is ready and reachable via ssh, WNoDeS customizes the machine, e.g. it creates the mount points The VM is now ready to be used and the associated batch job is resumed At logout WNoDeS automatically kills the batch job and destroys the VM Wall clock time spent by the bait job is used for accounting purposes 19 October 2010 CHEP10 - Taipei 8

Claudio Grandi INFN Bologna Starting a session... X application running on the VM image name number of cores memory mount points 19 October 2010 CHEP10 - Taipei 9

Claudio Grandi INFN Bologna...all files I need are there... my home directory my files Experiment software AFS cms data /home fs 19 October 2010 CHEP10 - Taipei 10

Claudio Grandi INFN Bologna...it looks like what I asked for! Two cores 1500 GB RAM 19 October 2010 CHEP10 - Taipei 11

Claudio Grandi INFN Bologna Outlook Address the use case of a user needing to save the VM status –Currently each time the user starts an interactive session gets a fresh virtual machine The only persistent data is on the mounted file systems –The use case may be useful e.g. to test services –In general it requires root access –A snapshot of the VM is saved on the storage area before destroying it. The saved image may be requested via the VIP CLI Package the client for easy installation also on personal workstations Optimize the resource allocation time –Access to an already instantiated machine with default configuration Fast – only 1 or 2 machines available for each VO –Instantiation of a machine with custom configuration on reserved slots Less than 1’. Defined number of slots reserved for that scope for all VOs –Instantiation of a machine out of the standard batch pool 19 October 2010 CHEP10 - Taipei 12

Claudio Grandi INFN Bologna More info Web site: Mailing list: At CHEP 2010: –A. Italiano WNoDeS, a tool for integrated Grid/Cloud access and computing farm virtualization Tue 19 th, 16:45; Computing Fabrics and Networking Technologies –D.Salomoni An Authentication Gateway for integrated Grid and Cloud access Wed 20 th, 12:30; Grid and Cloud Middleware 19 October 2010 CHEP10 - Taipei 13