VMDIRAC status Vanessa HAMAR CC-IN2P3

Acknowledgments This is the work mostly done by: Victor Mendez ( PIC ), Mathieu Puel ( CC/IN2P3 ) France-Grilles - LCG-France 28/11/13

France-Grilles - LCG-France Overview Introduction VMDIRAC Installation at CC Conclusions and future work France-Grilles - LCG-France 28/11/13

France-Grilles - LCG-France Introduction France-Grilles - LCG-France 28/11/13

France-Grilles - LCG-France Introduction France-Grilles - LCG-France 28/11/13

France-Grilles - LCG-France Introduction France-Grilles DIRAC service Hosted by the CC/IN2P3 Distributed administrator team 5 participating universities 18 VOs, ~100 registered users In production since May 2012 > 7 millions jobs France-Grilles - LCG-France 28/11/13

France-Grilles - LCG-France VMDIRAC France-Grilles - LCG-France 28/11/13

France-Grilles - LCG-France VMDIRAC VMDIRAC is an extension of DIRAC used to integrate Federated Clouds in a transparent manner to the user. VM scheduler initially developed for the Belle MC production system (VMDIRAC) Dynamic VM spawning taking the Task Queue state into account Discarding VMs automatically when no more needed The VM at boot time starts the “Pilot Job” This makes the instantiated VMs behave as any other WN with respect to the DIRAC WMS JobAgent and VMMonitorAgent are started inside the “Pilot Job” France-Grilles - LCG-France 28/11/13

France-Grilles - LCG-France DIRAC & Clouds WMS Task Queue Matcher Sandboxes Job Manager VM Scheduler Web API VM Cloud B Job Agent Cloud A Cloud C EC2 ONE OCCI En este caso el Payload no es DIRAC, por lo tanto es plano y verde. France-Grilles - LCG-France 28/11/13

France-Grilles - LCG-France Federated Cloud Test France-Grilles - LCG-France 28/11/13

France-Grilles - LCG-France Contextualization The contextualization mechanism allows to configure the VM to start the Pilot script at boot time Provides also the necessary configuration parameters to the pilot The contextualization mechanism is rather complicated Different for different cloud managers cernvm amiconfig Ad hoc images No contextualization, everything is put in the image for a particular endpoint ( software, certificates ) France-Grilles - LCG-France 28/11/13

France-Grilles - LCG-France VMDIRAC at CC France-Grilles - LCG-France 28/11/13

France-Grilles - LCG-France OpenStack DIRAC at CC Project diracsft 10 available instances - 10 available vCPUs - 15 Mb RAM Available OpenStack, libcloud , nova 1.1 python client Floating IPs available France-Grilles - LCG-France 28/11/13

Installation VMDIRAC at CC openstack/python-novaclient : Client Library for OpenStack Compute API. apache_libcloud-0.14.0 : standard Python library that abstracts away differences among multiple cloud provider API paramiko-1.11.0 : ssh2 protocol for python France-Grilles - LCG-France 28/11/13

Installation VMDIRAC at CC Workload Management Services : Virtual Machine Manager Virtual Machine Scheduler Virtual Machine Contextualization Database : VirtualMachineDB France-Grilles - LCG-France 28/11/13

Contextualization at CC Standard (golden) VM image Installing DIRAC software, as normal pilot job Security token, e.g. VM service certificate cernvm-batch-node 2.6-occi The IPs of the VMs are available to the VMDIRAC scheduler Polling the VMs to establish connection Use SSH access to contextualize the image Copy the software + certificate bundle via scp to the VM Start the JobAgent and VM Monitor Agent France-Grilles - LCG-France 28/11/13

France-Grilles - LCG-France Configuration vo.france-grilles.fr France-Grilles - LCG-France 28/11/13

France-Grilles - LCG-France Monitoring France-Grilles - LCG-France 28/11/13

Conclusions and Future Job Cloud access authorization X509, VOMS in addition to login/password To be installed at CC in January 2014 Add more “Clouds” to the configuration Test with real use cases Cloud usage accounting Incorporate into the DIRAC Accounting service France-Grilles - LCG-France 28/11/13

France-Grilles - LCG-France Questions ? France-Grilles - LCG-France 28/11/13