Presentation is loading. Please wait.

Presentation is loading. Please wait.

The CernVM Infrastructure Insights of a paradigmatic project Carlos Aguado Sanchez Jakob Blomer Predrag Buncic.

Published byClara Campbell Modified over 9 years ago

Similar presentations

Presentation on theme: "The CernVM Infrastructure Insights of a paradigmatic project Carlos Aguado Sanchez Jakob Blomer Predrag Buncic."— Presentation transcript:

1 The CernVM Infrastructure Insights of a paradigmatic project Carlos Aguado Sanchez Jakob Blomer Predrag Buncic

2 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Outline Objectives Requirements Support tools Service architecture Implementation CPU and storage provisioning Conclusions

3 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Outline Objectives Requirements Support tools Service architecture Implementation CPU and storage provisioning Conclusions

4 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Objectives CernVM: versatile virtual platform – Baseline virtual machine: CernVM Virtual Software Appliance Management of the baseline image for all experiments with possible further customization – Software Delivery Network: the CernVM File System (CVMFS) HTTP compliant

5 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Constraints This is a proof of concept: preproduction environment Oriented to create community Not a final production service, but worldwide Economic Modest budget for external support (tools) and HW (during all project long) – Netapps, SAN clusters, Barracuda, Inifiband, etc. – RightScale, VSA, etc.

6 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Outline Objectives Requirements Support tools Service architecture Implementation CPU and storage provisioning Conclusions

7 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Application model

8 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments CernVM App Store

9 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments CernVM App Store  Scalability  Delegated management  Availability  Performance

10 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Requirements Ability to provide: – Virtual Appliance Configuration Management – Uniform and persistent URL namespace With a large HTTP-repository behind – All services sandboxed as appliances Deployment and management interface

11 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Support tools Information systems for project management – CVS/SVN, AuthN/AuthZ, Wiki Technical Support services – Monitoring, backup, bootstrapping Log analysis

12 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Outline Objectives Requirements Support tools Service architecture Implementation CPU and storage provisioning Conclusions

13 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Service architecture

14 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments rPath philosophy: – JeOS – Transactional model for software deployment (Conary) – Simple user interface (rAA) service

15 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments rPath philosophy: – JeOS – Transactional model for software deployment (Conary) – Simple user interface (rAA) service Build types  Installable CD/DVD  Stub Image  Raw Filesystem Image  Netboot Image  Compressed Tar File  Demo CD/DVD (Live CD/DVD)  Raw Hard Disk Image  Vmware ® Virtual Appliance  Vmware ® ESX Server Virtual Appliance  Microsoft ® VHD Virtual Apliance  Xen Enterprise Virtual Appliance  Virtual Iron Virtual Appliance  Parallels Virtual Appliance  Amazon Machine Image  Update CD/DVD  Appliance Installable ISO

16 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments CDN: the URL scheme://hostname:port/path?query#fragment  scheme: http | https  hostname: cernvm.cern.ch cernvm-webfs.cern.ch rbuilder.cern.ch …

17 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Outline Objectives Requirements Support tools Service architecture Implementation CPU and storage provisioning Conclusions

18 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Service architecture

19 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments 1: Front end Gateway to back end services DNS Load-balanced cluster hosting public services: – Switch L7: combination of Squid/Apache ‘Router’ + webcache SSL engine URL mapper – LDAP directory (multimaster replica)

20 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Virtualization platforms – Management tools are key: provision (deployment and life cycle), resource allocation, integration VMWare Infrastructure VMotion HA: High Availability DRS: Distributed Resource Scheduling 2: CPU provisioning

21 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Several ways – Shared disk file systems Fibre Channel Protocol, iSCSI … On top: GFS, OCFS,GPFS… – Network file systems NFS, AFS, SMB, Lustre … Thin provisioning – Virtual SAN Appliance – Openfiler 3: Storage provisioning I

22 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments NexentaStor – 2-node cluster with active replication – LDAP: users, groups, netgroups – ZFS CVS-like semantics SW Raid NFS – System disk (vmdk) – Data vols. (NFS mounts) 3: Storage provisioning II

23 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments 4: Networking Two different physical networks – Public: dedicated CERN IP service (/24) Only the front end are opened in the central firewall – Private: access to storage and VM operations Aggregation of interfaces is used between CPU servers and the storage

24 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Network emulated within ESX: – VMs can easily move Connection to Nexentas via 2xGb 4:Networking

25 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Our MIT Front end rBuilder VMWare Nexenta

26 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments And some statistics With those 8 machines we currently run: – CVMFS (clients and web servers), rBuilder, LDAP, Zenoss, Boinc, djabberd, SFT-Web Traffic (April 2009): – +15GB from experiment repositories – +600 downloads of CernVM – +800 different IPs – +50GB total HTTP traffic – ~30% hits in the Squid cache

27 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Outline Objectives Requirements Support tools Service architecture Implementation CPU and storage provisioning Conclusions

28 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Conclusions Versatile and modular infrastructure – Easy deployment of new services – Simple addition of new resources – Secure and controlled environment Good performance for commodity HW Easy replicable by definition – Door opened to scale the service based on DNS- related techniques

29 PH-SFT 2/20/2016CernVM – A virtual mchine for LHC experiments Thanks! Mailing lists: cernvm-talk@cern.ch (open list for announcements and discussion) cernvm.support@cern.ch (end-user support for the CernVM project) Savannah Portal: Please submit bugs and feature requests to Savannah at http://savannah.cern.ch/projects/cernvm CernVM Home Page: http://cernvm.cern.ch rBuilder & Download Page: http://rbuilder.cern.ch CernVM Wiki: http://cernvm.cern.ch/project/trac/cernvm ATLAS Wiki: https://twiki.cern.ch/twiki/bin/view/Atlas/CernVM

Download ppt "The CernVM Infrastructure Insights of a paradigmatic project Carlos Aguado Sanchez Jakob Blomer Predrag Buncic."

Similar presentations

Cloud computing is used to describe a variety of computing concepts that involve a large number of computers connected through a real-time communication.

Cloud computing is used to describe a variety of computing concepts that involve a large number of computers connected through a real-time communication.
Microsoft Dynamics AX Technical Conference 2013

Microsoft Dynamics AX Technical Conference 2013
GLOBAL VIRTUAL CLUSTER DEPLOYMENT THROUGH A CONTENT DELIVERY NETWORK Pongsakorn U-chupala, Kohei Ichikawa (NAIST) Luca Clementi, Philip Papadopoulos (UCSD)

GLOBAL VIRTUAL CLUSTER DEPLOYMENT THROUGH A CONTENT DELIVERY NETWORK Pongsakorn U-chupala, Kohei Ichikawa (NAIST) Luca Clementi, Philip Papadopoulos (UCSD)
“It’s going to take a month to get a proof of concept going.” “I know VMM, but don’t know how it works with SPF and the Portal” “I know Azure, but.

“It’s going to take a month to get a proof of concept going.” “I know VMM, but don’t know how it works with SPF and the Portal” “I know Azure, but.
Highly Available Central Services An Intelligent Router Approach Thomas Finnern Thorsten Witt DESY/IT.

Highly Available Central Services An Intelligent Router Approach Thomas Finnern Thorsten Witt DESY/IT.
VMware Infrastructure Alex Dementsov Tao Yang Clarkson University Feb 28, 2007.

VMware Infrastructure Alex Dementsov Tao Yang Clarkson University Feb 28, 2007.
VMware Update 2009 Daniel Griggs Solutions Architect, Virtualization Servers & Storage Solutions Practice Dayton OH.

VMware Update 2009 Daniel Griggs Solutions Architect, Virtualization Servers & Storage Solutions Practice Dayton OH.
1 Bridging Clouds with CernVM: ATLAS/PanDA example Wenjing Wu 2010-8-27.

1 Bridging Clouds with CernVM: ATLAS/PanDA example Wenjing Wu
Copyright © 2005 VMware, Inc. All rights reserved. VMware Virtualization Phil Anthony Virtual Systems Engineer

Copyright © 2005 VMware, Inc. All rights reserved. VMware Virtualization Phil Anthony Virtual Systems Engineer
Implementing Failover Clustering with Hyper-V

Implementing Failover Clustering with Hyper-V
Understanding and Managing WebSphere V5

Understanding and Managing WebSphere V5
Virtual Desktop Infrastructure Solution Stack Cam Merrett – Demonstrator User device Connection Bandwidth Virtualisation Hardware Centralised desktops.

Virtual Desktop Infrastructure Solution Stack Cam Merrett – Demonstrator User device Connection Bandwidth Virtualisation Hardware Centralised desktops.
CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Next generation of virtual infrastructure with Hyper-V Michal Kwiatek, Juraj Sucik, Rafal.

CERN IT Department CH-1211 Genève 23 Switzerland t Next generation of virtual infrastructure with Hyper-V Michal Kwiatek, Juraj Sucik, Rafal.
Opensource for Cloud Deployments – Risk – Reward – Reality

Opensource for Cloud Deployments – Risk – Reward – Reality
Yury Kissin Infrastructure Consultant Storage improvements Dynamic Memory Hyper-V Replica VM Mobility New and Improved Networking Capabilities.

Yury Kissin Infrastructure Consultant Storage improvements Dynamic Memory Hyper-V Replica VM Mobility New and Improved Networking Capabilities.
Cloud Computing for the Enterprise November 18th, 2011. This work is licensed under a Creative Commons.

Cloud Computing for the Enterprise November 18th, This work is licensed under a Creative Commons.
 Cloud computing  Workflow  Workflow lifecycle  Workflow design  Workflow tools : xcp, eucalyptus, open nebula.

 Cloud computing  Workflow  Workflow lifecycle  Workflow design  Workflow tools : xcp, eucalyptus, open nebula.
Components of Windows Azure - more detail. Windows Azure Components Windows Azure PaaS ApplicationsWindows Azure Service Model Runtimes.NET 3.5/4, ASP.NET,

Components of Windows Azure - more detail. Windows Azure Components Windows Azure PaaS ApplicationsWindows Azure Service Model Runtimes.NET 3.5/4, ASP.NET,
Technology Overview. Agenda What’s New and Better in Windows Server 2003? Why Upgrade to Windows Server 2003 ?  From Windows NT 4.0  From Windows 2000.

Technology Overview. Agenda What’s New and Better in Windows Server 2003? Why Upgrade to Windows Server 2003 ?  From Windows NT 4.0  From Windows 2000.
1 port BOSS on Wenjing Wu (IHEP-CC) 2011-5-10.

1 port BOSS on Wenjing Wu (IHEP-CC)

Similar presentations

Ads by Google