Ops Manager API, Puppet and OpenStack – Fully automated orchestration from scratch! MongoDB World 2016

Nice to Meet You! Naama Bamberger Einav Rivni (nabamber@cisco.com) (erivni@cisco.com) Nice to Meet You! Naama Bamberger (nabamber@cisco.com)

Serving 12M households & 50M devices and 14 Deployments Serving 12M households & 50M devices 2 shards, 2 data-centers * ~14 live deployments. All of them are in customer site (on-prem) * Biggest deployment has 2 shards, handling 12M households and 50M devices and entitlements, across 2 data centers * Started with user management system for two ways TV system, storing household information, devices, entitlements and preferences. Storing user’s booking and recordings * Now many application ported to mongo db to store catalog, SWDL management, offer management and many more.

Our Goal –

Our Goal – Create a fully automated process for MongoDB deployment on a production OpenStack environment

Don’t we have MongoDB automation already?

Called OpsManager?

Well, sort of

We can deploy a cluster automatically once: OpsManager is installed and configured VMs (or physical machines) exist for cluster Agents installed on all nodes Agents are registered to OpsManager

And we’ll try to automate all the ‘once’ steps

Desired deployment model Ops-Manager Ops-Manager Backup DB - Backup MongoD Mongo Config MongoS

Environment, tools and technologies

Virtualization “Virtualization refers to technologies designed to provide a layer of abstraction between computer hardware systems and the software running on them” (John K. Waters)

Openstack Set of software tools for IaaS Began in 2010 by NASA and Rackspace Opensource Project Current Version: Mitaka

Main Services

Compute: Nova Nova

Networking: Neutron Neutron

Storage: Cinder Cinder

Dashboard: Horizon Horizon

And many more

HEAT overview

HEAT – Openstack’s orchestration engine

OS::Neutron::SecurityGroup OS::Heat::ResourceGroup OS::Nova::Server HEAT Resources OS::Neutron::SecurityGroup OS::Heat::ResourceGroup OS::Nova::Server OS::Cinder::Volume

OS::Neutron:: SecurityGroup for ops-manager

OS::Neutron:: SecurityGroup for agents

OS::Heat:: ResourceGroup

OS::Nova::Server

OS::Cinder::Volume

Mongo orchestration – installation steps Ops-Manager MongoD Mongo Config MongoS Ops-Manager Backup Deployer MongoD Mongo Config MongoS DB - Backup MongoD Mongo Config

Step 1: Create Ops-manager servers

Ops manager – deployment model DB - Backup Ops-Manager Backup Ops-Manager 27017 27017 27017 27018 27018 27018

Step 2: Install Ops-manager software Ops-Manager Backup DB - Backup

Step 3: Create User Ops-Manager Ops-Manager Backup DB - Backup

Step 4: Save Keys Ops-Manager Ops-Manager Backup DB - Backup

Step 5: Create hosts for mongod/mongos/config instances Ops-Manager Ops-Manager Backup DB - Backup

Agents deployment model cinder cinder cinder agent

Step 6: Mount Volumes for instances reserved for mongod Ops-Manager agent agent agent Ops-Manager Backup agent agent agent DB - Backup agent agent

Step 7: Get keys Ops-Manager agent agent agent Ops-Manager Backup DB - Backup agent agent

Step 8: Install agent and configure to point to ops-manager Ops-Manager Backup agent agent agent DB - Backup agent agent

Step 9: Create deployer server Agents Ops-Manager agent agent agent Ops-Manager Backup agent agent agent DB - Backup agent agent

Step 10: Deployer executes REST requests on ops-manager, to create cluster MongoD Mongo Config MongoS Ops-Manager Backup Deployer MongoD Mongo Config MongoS DB - Backup MongoD Mongo Config

That’s all nice, but life is a bit more complicated

Saving API keys and Group info Problem: No way to retrieve key once created, though we need it to configure all agents

Saving API keys and Group info Solution: Save info as Json file, on ops-manager static file location (delete once installation is complete!) agentApiKey: "2af35366951865e509520b2d3458b5cc"

Isolated Mode Problem: No access to the outside world from the openstack environment By default, ops-manager downloads all mongodb software from the mongodb site

Isolated mode Solution: - Configure Ops-Manager to work in an isolated-mode. - Create a version Manifest identifying the allowed versions. - Download and save the required tar files on the ops-Manager machine

Allowing Mongo versions Store version manifest json file in Ops-Manager.

First User Creation

Demo

MongoDB Clusters Across Data Centers

Luckily, OpsManager can help us with that Steps: On primary site: Deploy fully working cluster as shown above Then, on secondary: Deploy agents Create deployer From deployer – access OpsManager on primary site to extend cluster

Desired Model DC2 DC1 Ops-Manager Ops-Manager Backup DB - Backup MongoD Mongo Config MongoS MongoD Mongo Config MongoS Ops-Manager Backup MongoD Mongo Config MongoS MongoD Mongo Config MongoS DB - Backup MongoD Mongo Config

In case of a real disaster DC1 DC2 Ops-Manager MongoD Mongo Config MongoS MongoD Mongo Config MongoS Ops-Manager Backup MongoD Mongo Config MongoS MongoD Mongo Config MongoS DB - Backup MongoD Mongo Config

Arbitar addition DC2 DC1 DC3 Ops-Manager Ops-Manager Backup MongoD Mongo Config MongoS MongoD Mongo Config MongoS Ops-Manager Backup MongoD Mongo Config MongoS MongoD Mongo Config MongoS DB - Backup MongoD Mongo Config DC3 Arbitar

OpsManager is a great tool Takeaways OpsManager is a great tool Orchestration is more than calling ‘create cluster’ With some effort, we can automate it all