GRNETDeIC konference 20121 20121112 Building a real-world public cloud from the ground up Vangelis Koukis Technical.

GRNETDeIC konference 20121 vkoukis@grnet.gr| 20121112 Building a real-world public cloud from the ground up Vangelis Koukis vkoukis@grnet.gr Technical Coordinator,  okeanos Project

Greek Research and Technology NetworkDeIC konference 20122 vkoukis@grnet.gr| 20121112 Outline   okeanos ?  Rationale  Design – Platform - Features  Unity - Automation  Opensource – Upcoming

Greek Research and Technology NetworkDeIC konference 20123 vkoukis@grnet.gr| 20121112 What is  okeanos? ‘okeanos’ is Greek for ‘ocean’. Oceans capture, store and deliver energy, oxygen and life around the planet.

GRNETDeIC konference 20124 vkoukis@grnet.gr| 20121112 Simplicity

GRNETDeIC konference 20125 vkoukis@grnet.gr| 20121112

GRNETDeIC konference 20126 vkoukis@grnet.gr| 20121112 Compute Network Storage Security Virtual Machines Virtual Ethernets Virtual Disks Virtual Firewalls

GRNETDeIC konference 20127 vkoukis@grnet.gr| 20121112 Flexibility

GRNETDeIC konference 20129 vkoukis@grnet.gr| 20121112 1x 2x5x 8x

Greek Research and Technology NetworkDeIC konference 201210 vkoukis@grnet.gr| 20121112  okeanos service  Goal: Production-quality IaaS  Beta in Dec, current Alpha: >1600 VMs / >1000 users  Target group: GRNET’s customers  direct: IT depts of connected institutions  indirect: university students, researchers in academia  Users manage resources over  a simple, elegant UI, or  a REST API, for full programmatic control

Greek Research and Technology NetworkDeIC konference 201211 vkoukis@grnet.gr| 20121112  okeanos features  Compute/Network Service: Cyclades  File Storage Service: Pithos+  Image Service: Plankton  Identity Service: Astakos  Volume Service: Archipelago

GRNETDeIC konference 201212 vkoukis@grnet.gr| 20121112 Rationale

Greek Research and Technology NetworkDeIC konference 201213 vkoukis@grnet.gr| 20121112 How it all started  Need for easy, secure access to GRNET’s datacenters  User friendliness, simplicity  Scalable to the thousands  #VMs, TBs, users (Pithos:  10k)  running within GRNET’s AAI Federation  Resell or build your own?  IaaS cloud provider, vendor, or own infrastructure?  It all depends on your needs

Greek Research and Technology NetworkDeIC konference 201214 vkoukis@grnet.gr| 20121112 Build on commercial IaaS?  Commercial IaaS  Amazon EC2 not an end-user service  Need to develop custom UI, AAI layers  Vendor lock-in  Unsuitable for IT depts persistent, long-term servers custom networking requirements  GRNET has invested heavily in its core network  > 8000km of dark fiber

Greek Research and Technology NetworkDeIC konference 201215 vkoukis@grnet.gr| 20121112 Bring vendor into datacenter?  Hypervisor lock-in  Is a turn-key solution suitable for a public cloud?  Building public clouds is an ongoing process  Manageable by GRNET’s operation  Integrated into the rest of the infrastructure  Scaling to thousands of users  Build on existing know-how  Gain know-how, build own IaaS  reuse for own services

Greek Research and Technology NetworkDeIC konference 201216 vkoukis@grnet.gr| 20121112 What about opensource?  OpenStack, Eucalyptus, OpenNebula  Need a mature opensource core to build around  Maturity, production-readiness?  proven in production environments, predictable  Extensibility?  Flexibility?  Upgradeability, maintainability?

GRNETDeIC konference 201217 vkoukis@grnet.gr| 20121112 Design

Greek Research and Technology NetworkDeIC konference 201218 vkoukis@grnet.gr| 20121112  okeanos design decisions  Reuse existing components  Build on Google Ganeti  target commodity hardware  release to the community as opensource

Greek Research and Technology NetworkDeIC konference 201219 vkoukis@grnet.gr| 20121112  okeanos design principles  No need to make the world  No need to support everything  Service developed and maintained by  10-15 people  Start from the architecture…  …then discover, combine, reuse the right components  And for everything that’s not already available  Do it yourself!

Greek Research and Technology NetworkDeIC konference 201221 vkoukis@grnet.gr| 20121112 Jigsaw puzzle  Synnefo  custom cloud management software to power  okeanos  Google Ganeti backend  VM cluster management: physical nodes, VMs, migrations  OpenStack APIs: Compute API v1.1, Object Storage API  with custom extensions whenever necessary  Then everything comes together  UI, Networking, Images, Storage, Monitoring, Identity management, Accounting, Billing, Clients, Helpdesk

Greek Research and Technology NetworkDeIC konference 201222 vkoukis@grnet.gr| 20121112 Why Ganeti?  No need to reinvent the wheel  Scalable, proven software infrastructure  Built with reliability and redundancy in mind  Combines open components (KVM, LVM, DRBD)  Well-maintained, readable code  VM cluster management in production is serious business  reliable VM control, VM migrations, resource allocation  handling node downtime, software upgrades

Greek Research and Technology NetworkDeIC konference 201223 vkoukis@grnet.gr| 20121112 Why Ganeti?  GRNET already had long experience with Ganeti  provides  280 VMs to NOCs through the ViMa service  involved in development, contributing patches upstream  Build on existing know-how for  okeanos  Common backend, common fixes  reuse of experience and operational procedures  simplified, less error-prone deployment

GRNETDeIC konference 201224 vkoukis@grnet.gr| 20121112 Platform

Greek Research and Technology NetworkDeIC konference 201225 vkoukis@grnet.gr| 20121112 Software Stack Multiple users, multiple resources Multiple VMs on cluster Single VM Synnefo Ganeti KVM REST API

Greek Research and Technology NetworkDeIC konference 201226 vkoukis@grnet.gr| 20121112 Platform Design user@home admin@home Web ClientCLI ClientWeb Client 2 GRNET datacenter Debian Virtual Hardware OpenStack Compute API v1.1 Direct Outof Band Access Synnefo cloud management software Google Ganeti KVM GRNET Proprietary

GRNETDeIC konference 201227 vkoukis@grnet.gr| 20121112 Features

Greek Research and Technology NetworkDeIC konference 201228 vkoukis@grnet.gr| 20121112 Virtual Machine Actions My_Windows_desktop Shutdown Reboot Start Console Destroy

Greek Research and Technology NetworkDeIC konference 201229 vkoukis@grnet.gr| 20121112 IaaS – Compute (1)  Virtual Machines  powered by KVM Linux and Windows guests, on Debian hosts  Google Ganeti for VM cluster management  accessible by the end-user over the Web or programmatically (OpenStack Compute v1.1)

Greek Research and Technology NetworkDeIC konference 201230 vkoukis@grnet.gr| 20121112 IaaS – Compute (2)  User has full control over own VMs  Create Select # CPUs, RAM, System Disk OS selection from pre-defined or custom Images popular Linux distros (Fedora, Debian, Ubuntu) Windows Server 2008 R2  Start, Shutdown, Reboot, Destroy  Out-of-Band console over VNC for troubleshooting

Greek Research and Technology NetworkDeIC konference 201231 vkoukis@grnet.gr| 20121112 IaaS – Compute (3)  REST API for VM management  OpenStack Compute v1.1 compatible  3rd party tools and client libraries  custom extensions for yet-unsupported functionality  Python & Django implementation  Full-featured UI in JS/jQuery  UI is just another API client  All UI operations happen over the API

Greek Research and Technology NetworkDeIC konference 201232 vkoukis@grnet.gr| 20121112 IaaS – Network (Virtual Ethernets) Internet Private Network 1 Private Network 2Private Network 3

Greek Research and Technology NetworkDeIC konference 201233 vkoukis@grnet.gr| 20121112 IaaS – Network - Functionality  Dual IPv4/IPv6 connectivity for each VM  Easy, platform-provided firewalling  Array of pre-configured firewall profiles  Or roll-your-own firewall inside VM  Multiple private, virtual L2 networks  Construct arbitrary network topologies  e.g., deploy VMs in multi-tier configurations  Exported all the way to the API and the UI

GRNETDeIC konference 201234 vkoukis@grnet.gr| 20121112 Unity

Greek Research and Technology NetworkDeIC konference 201235 vkoukis@grnet.gr| 20121112 Spawn Freeze Images Ubuntu my own Ubuntu

Greek Research and Technology NetworkDeIC konference 201236 vkoukis@grnet.gr| 20121112 Custom Images: snf-image  Untrusted images  Host cannot touch user-provided data  Resize fs, change hostname, change passwords, inject files  Split design  snf-image-host  snf-image-helper  All customization in helper VM

Greek Research and Technology NetworkDeIC konference 201237 vkoukis@grnet.gr| 20121112  OpenStack Object Storage API  Block storage  Content-based addressing for blocks  Every file is a collection of blocks  Web-based, command-line, and native clients  Synchronization, deduplication  An integral part of  okeanos  User files, Image registry for VM Images  Goal: use common backend with Archipelago

Greek Research and Technology NetworkDeIC konference 201238 vkoukis@grnet.gr| 20121112 Spawn Freeze Images Ubuntu my own Ubuntu

Greek Research and Technology NetworkDeIC konference 201239 vkoukis@grnet.gr| 20121112 Clone Snapshot Images  Storage Ubuntu root Ubuntu + user data

Greek Research and Technology NetworkDeIC konference 201240 vkoukis@grnet.gr| 20121112 Images – Golden Image golden Debian

Greek Research and Technology NetworkDeIC konference 201241 vkoukis@grnet.gr| 20121112 IaaS – Storage

Greek Research and Technology NetworkDeIC konference 201242 vkoukis@grnet.gr| 20121112 Archipelago RADOS Object Storage nodes IaaS – Storage Maps Volume Composer object I/O Monitor nodes Storage

Greek Research and Technology NetworkDeIC konference 201243 vkoukis@grnet.gr| 20121112 RADOS Storage nodes IaaS – Storage Maps Volume Composer object I/O Monitor nodes

Greek Research and Technology NetworkDeIC konference 201244 vkoukis@grnet.gr| 20121112 IaaS – Storage (1)  First-phase deployment  System-provided and custom user Images  Redundant storage based on DRBD  VMs survive physical node downtime or failure  Currently under testing  Reliable distributed storage over RADOS  Combined with custom software for snapshotting, cloning  Dynamic virtual storage volumes

Greek Research and Technology NetworkDeIC konference 201245 vkoukis@grnet.gr| 20121112 IaaS – Storage (2)  Multi-tier storage architecture  Dedicated Storage Nodes (SSD, SAS, and SATA storage)  OSDs, e.g., for RADOS  Custom storage layer: Archipelago  manages snapshots, creates clones over block pools  OS Images held as snapshots  VMs created as clones of snapshots

GRNETDeIC konference 201246 vkoukis@grnet.gr| 20121112 Integration

Greek Research and Technology NetworkDeIC konference 201247 vkoukis@grnet.gr| 20121112

Greek Research and Technology NetworkDeIC konference 201251 vkoukis@grnet.gr| 20121112 Support services  Identity: Astakos  Provides the user base for  okeanos  Once authenticated, the user retrieves a common auth token for programmatic access

GRNETDeIC konference 201252 vkoukis@grnet.gr| 20121112 Automation

Greek Research and Technology NetworkDeIC konference 201253 vkoukis@grnet.gr| 20121112./kamaki $./kamaki Usage: kamaki [options] … --api=API API can be either openstack or synnefo --url=URL API URL --token=TOKEN use token TOKEN … Commands: flavor info get flavor details flavor list list flavors … image create create image image delete delete image $./kamaki server shutdown 101 --url=http://localhost:8000/api/v1.1 --token=1234527db2…

Greek Research and Technology NetworkDeIC konference 201254 vkoukis@grnet.gr| 20121112./kamaki $ ipython In [1]: from kamaki.client import Client In [2]: c = Client('http://localhost:8000/api/v1.1', "1234527db2…") In [3]: c.list_flavors() … In [4]: i = c.list_images() In [5]: i[5] {u'created': u'2011-06-09T00:00:00+00:00', u'id': 7, u'metadata': {u'values': {u'OS': u'windows', u'size': u'11000'}}, u'name': u'Windows', u'progress': 100, u'status': u'ACTIVE', u'updated': u'2011-09-12T14:47:12+00:00'} In [6]: c.create_server('mywin1', 3, 5)

GRNETDeIC konference 201255 vkoukis@grnet.gr| 20121112 Sights

Greek Research and Technology NetworkDeIC konference 201256 vkoukis@grnet.gr| 20121112 Live Demo  Prepare and upload Image from local template VM  Spawn compute cluster to run MPI app  Make local modifications and repeat  … What if it was over a 3G connection?  Time needed to upload 1GB Image file?  Time needed to prepare and spawn virtual nodes?

GRNETDeIC konference 201257 vkoukis@grnet.gr| 20121112 Internals

Greek Research and Technology NetworkDeIC konference 201258 vkoukis@grnet.gr| 20121112 Ganeti Master Queue Deployment DB Ganeti node KVM … Web Server REST API Logic RAPI SQL snf-gnt-eventd snf-gnt-hook snf-dispatcher API Server apiaai SQL ui

Greek Research and Technology NetworkDeIC konference 201259 vkoukis@grnet.gr| 20121112 Upcoming goals  Credit-based resource allocation  Abstract away the Ganeti backend, replace with backend connector behind the MQ  Release to community as reference implementation of OpenStack Compute v1.1  Support live modification of VMs in Ganeti  Snapshots, clones in storage layer  Dramatic decrease in VM initialization time  Support workloads with 100s of ephemeral VMs e.g. for scientific computation, MPI jobs

GRNETDeIC konference 2012116 vkoukis@grnet.gr| 20121112 Upcoming

Greek Research and Technology NetworkDeIC konference 2012117 vkoukis@grnet.gr| 20121112 Current and Upcoming features  Now: Alpha2  Common user base, custom user images on Pithos+  short-term: Synnefo v0.12, Beta  Ultra-lightweight VMs on Archipelago with RADOS backend  medium-term  Volumes: clonable / snapshottable / attachable disks  Network and storage hotplugging  Upcoming beta in fully populated datacenter

Greek Research and Technology NetworkDeIC konference 2012118 vkoukis@grnet.gr| 20121112 Opensource  Synnefo: Cyclades / Pithos+ / Astakos  https://code.grnet.gr/projects/synnefo  https://code.grnet.gr/projects/pithos  https://code.grnet.gr/projects/astakos  kamaki  https://code.grnet.gr/projects/kamaki pip install or apt-get install everything!

http://okeanos.io

Greek Research and Technology NetworkDeIC konference 2012120 vkoukis@grnet.gr| 20121112 Thank You! Questions?

Greek Research and Technology NetworkDeIC konference 2012121 vkoukis@grnet.gr| 20121112 Asynchronous design  DB contains All state needed to handle API queries  no need to reach the backend  Ganeti GetInstanceInfo() is a proper job, too slow  Two distinct paths, effect and update  Effect changes to VMs  when servicing API requests to modify VM state  issue commands to Ganeti backend, over RAPI  ACK reception of request to user  Update DB, when interesting things happen  user or admin initiated  Queue notifications to Message Queue, over AMQP

Greek Research and Technology NetworkDeIC konference 2012122 vkoukis@grnet.gr| 20121112 Ganeti Master Queue Synnefo deployment DB Ganeti node KVM … Web Server REST API Logic RAPI SQL snf-gnt-eventd snf-gnt-hook snf-dispatcher API Server apiaai SQL ui

Greek Research and Technology NetworkDeIC konference 2012123 vkoukis@grnet.gr| 20121112 The “effect” Path  Reception of API request to modify VM state (e.g., PUT /servers over HTTP)  API enforces access rights and policy  Ganeti knows no cloud users or access rights  Need to translate from Openstack Compute to backend ops (e.g., CreateInstance())  Asynchronous request processing  Return HTTP 202 Accepted  it’s up to the API client to poll for completion

Greek Research and Technology NetworkDeIC konference 2012124 vkoukis@grnet.gr| 20121112 Ganeti Master Queue Synnefo deployment DB Ganeti node KVM … Web Server REST API Logic RAPI SQL snf-gnt-eventd snf-gnt-hook snf-dispatcher API Server apiaai SQL ui

Greek Research and Technology NetworkDeIC konference 2012125 vkoukis@grnet.gr| 20121112 The “update” path  May run at any time  Completely decoupled from “effect” path  Design goal:  Ganeti admins free to bypass frontend  Synnefo adapts  Synnefo logic triggered on backend events  Ganeti operation progressing in the queue  Synnefo hook running inside Ganeti Hooks run at various phases in a VM’s lifecycle

Greek Research and Technology NetworkDeIC konference 2012126 vkoukis@grnet.gr| 20121112 Ganeti Master Queue Synnefo deployment DB Ganeti node KVM … Web Server REST API Logic RAPI SQL snf-gnt-eventd snf-dispatcher ui API Server apiaai SQL snf-gnt-hook

Greek Research and Technology NetworkDeIC konference 2012127 vkoukis@grnet.gr| 20121112 The Ganeti event daemon  Ganeti master manages job queue  Jobs pass Queued, Waiting, Running, end up in Canceled, Success, Error.  Need a way for Synnefo to monitor job progress  Synnefo-specific solution: Ganeti event daemon  Passively monitor the Ganeti job queue  Notifications over AMQP on job progress  Synnefo logic listens to Message Queue, updates DB  inotify()- based mechanism, no code changes to Ganeti

Greek Research and Technology NetworkDeIC konference 2012128 vkoukis@grnet.gr| 20121112 The Synnefo hook in Ganeti  Different phases in a VM’s lifecycle  { pre, post } – { add, start, stop, reboot, modify }  Run Synnefo-specific hook in post-*  Pushes VM configuration notifications to MQ  e.g., NIC setup

Greek Research and Technology NetworkDeIC konference 2012129 vkoukis@grnet.gr| 20121112 IaaS – Network - Implementation  Custom modifications to Ganeti  IP pool management for the public network  Custom-written DHCP server over NFQUEUE  Custom interface handling scripts  Enforce VM networking configuration  Private Networks  Alpha: pre-provisioned bridges to 802.1Q VLANs  Later on: MAC-prefix based filtering

Greek Research and Technology NetworkDeIC konference 2012130 vkoukis@grnet.gr| 20121112 Ganeti Master Queue Synnefo deployment DB Ganeti node KVM … Web Server REST API Logic RAPI SQL snf-gnt-eventd snf-dispatcher ui API Server apiaai SQL snf-gnt-hook

Greek Research and Technology NetworkDeIC konference 2012131 vkoukis@grnet.gr| 20121112 Reconciliation with Ganeti  What if the MQ is down, and messages are lost?  Ganeti is the Single Source of Truth for VM state  Reconcile DB state asynchronously  On success notification for a Ganeti GetInstanceInfo() op  Triggered periodically, e.g., using cron  or even by the administrator, running gnt-instance info manually

GRNETDeIC konference 20121 20121112 Building a real-world public cloud from the ground up Vangelis Koukis Technical.

Similar presentations

Presentation on theme: "GRNETDeIC konference 20121 20121112 Building a real-world public cloud from the ground up Vangelis Koukis Technical."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

GRNETDeIC konference 20121 20121112 Building a real-world public cloud from the ground up Vangelis Koukis Technical.

Similar presentations

Presentation on theme: "GRNETDeIC konference 20121 20121112 Building a real-world public cloud from the ground up Vangelis Koukis Technical."— Presentation transcript:

Similar presentations

About project

Feedback