1. VMDIRAC status
Vanessa HAMAR
CC-IN2P3

2. Acknowledgments This is the work mostly done by:
Victor Mendez ( PIC ),
Mathieu Puel ( CC/IN2P3 )
France-Grilles - LCG-France
28/11/13

3. France-Grilles - LCG-France
Overview
Introduction
VMDIRAC
Installation at CC
Conclusions and future work
France-Grilles - LCG-France
28/11/13

4. France-Grilles - LCG-France
Introduction
France-Grilles - LCG-France
28/11/13

5. France-Grilles - LCG-France
Introduction
France-Grilles - LCG-France
28/11/13

6. France-Grilles - LCG-France
Introduction
France-Grilles DIRAC service
Hosted by the CC/IN2P3
Distributed administrator team
5 participating universities
18 VOs, ~100 registered users
In production since May
> 7 millions jobs
France-Grilles - LCG-France
28/11/13

7. France-Grilles - LCG-France
VMDIRAC
France-Grilles - LCG-France
28/11/13

8. France-Grilles - LCG-France
VMDIRAC
VMDIRAC is an extension of DIRAC used to integrate Federated Clouds in a transparent manner to the user.
VM scheduler initially developed for the Belle MC production system (VMDIRAC)
Dynamic VM spawning taking the Task Queue state into account
Discarding VMs automatically when no more needed
The VM at boot time starts the “Pilot Job”
This makes the instantiated VMs behave as any other WN with respect to the DIRAC WMS
JobAgent and VMMonitorAgent are started inside the “Pilot Job”
France-Grilles - LCG-France
28/11/13

9. France-Grilles - LCG-France
DIRAC & Clouds
WMS
Task Queue
Matcher
Sandboxes
Job Manager
VM Scheduler
Web
API VM
Cloud B
Job Agent
Cloud A
Cloud C
EC2
ONE
OCCI
En este caso el Payload no es DIRAC, por lo tanto es plano y verde.
France-Grilles - LCG-France
28/11/13

10. France-Grilles - LCG-France
Federated Cloud Test
France-Grilles - LCG-France
28/11/13

11. France-Grilles - LCG-France
Contextualization
The contextualization mechanism allows to configure the VM to start the Pilot script at boot time
Provides also the necessary configuration parameters to the pilot
The contextualization mechanism is rather complicated
Different for different cloud managers
cernvm
amiconfig
Ad hoc images
No contextualization, everything is put in the image for a particular endpoint ( software, certificates )
France-Grilles - LCG-France
28/11/13

12. France-Grilles - LCG-France
VMDIRAC at CC
France-Grilles - LCG-France
28/11/13

13. France-Grilles - LCG-France
OpenStack DIRAC at CC
Project diracsft
10 available instances - 10 available vCPUs - 15 Mb RAM Available
OpenStack, libcloud , nova 1.1 python client
Floating IPs available
France-Grilles - LCG-France
28/11/13

14. Installation VMDIRAC at CC
openstack/python-novaclient : Client Library for OpenStack Compute API.
apache_libcloud : standard Python library that abstracts away differences among multiple cloud provider API
paramiko : ssh2 protocol for python
France-Grilles - LCG-France
28/11/13

15. Installation VMDIRAC at CC
Workload Management Services :
Virtual Machine Manager
Virtual Machine Scheduler
Virtual Machine Contextualization
Database : VirtualMachineDB
France-Grilles - LCG-France
28/11/13

16. Contextualization at CC
Standard (golden) VM image
Installing DIRAC software, as normal pilot job
Security token, e.g. VM service certificate
cernvm-batch-node 2.6-occi
The IPs of the VMs are available to the VMDIRAC scheduler
Polling the VMs to establish connection
Use SSH access to contextualize the image
Copy the software + certificate bundle via scp to the VM
Start the JobAgent and VM Monitor Agent
France-Grilles - LCG-France
28/11/13

17. France-Grilles - LCG-France
Configuration
vo.france-grilles.fr
France-Grilles - LCG-France
28/11/13

18. France-Grilles - LCG-France
Monitoring
France-Grilles - LCG-France
28/11/13

19. Conclusions and Future Job
Cloud access authorization
X509, VOMS in addition to login/password
To be installed at CC in January 2014
Add more “Clouds” to the configuration
Test with real use cases
Cloud usage accounting
Incorporate into the DIRAC Accounting service
France-Grilles - LCG-France
28/11/13

20. France-Grilles - LCG-France
Questions ?
France-Grilles - LCG-France
28/11/13