1. Ops Manager API, Puppet and OpenStack – Fully automated orchestration from scratch!
MongoDB World 2016

2. Nice to Meet You! Naama Bamberger Einav Rivni (nabamber@cisco.com)
Nice to Meet You!
Naama Bamberger

3. Serving 12M households & 50M devices
and 14
Deployments
Serving 12M households & 50M devices
2 shards, 2 data-centers
* ~14 live deployments. All of them are in customer site (on-prem)
* Biggest deployment has 2 shards, handling 12M households and 50M devices and entitlements, across 2 data centers
* Started with user management system for two ways TV system, storing household information, devices, entitlements and preferences. Storing user’s booking and recordings
* Now many application ported to mongo db to store catalog, SWDL management, offer management and many more.

4. Our Goal –

5. Our Goal –
Create a fully automated process for MongoDB deployment on a production OpenStack environment

7. Don’t we have MongoDB automation already?

8. Called OpsManager?

9. Well, sort of

10. We can deploy a cluster automatically once:
OpsManager is installed and configured
VMs (or physical machines) exist for cluster
Agents installed on all nodes
Agents are registered to OpsManager

11. And we’ll try to automate all the ‘once’ steps

12. Desired deployment model
Ops-Manager
Ops-Manager Backup
DB - Backup
MongoD
Mongo Config
MongoS

13. Environment, tools and technologies

15. Virtualization
“Virtualization refers to technologies designed to provide a layer of abstraction between computer hardware systems and the software running on them”
(John K. Waters)

16. Openstack Set of software tools for IaaS
Began in 2010 by NASA and Rackspace
Opensource Project
Current Version: Mitaka

17. Main Services

18. Compute: Nova
Nova

19. Networking: Neutron
Neutron

20. Storage: Cinder
Cinder

21. Dashboard: Horizon
Horizon

22. And many more

23. HEAT overview

24. HEAT – Openstack’s orchestration engine

25. OS::Neutron::SecurityGroup OS::Heat::ResourceGroup OS::Nova::Server
HEAT Resources
OS::Neutron::SecurityGroup
OS::Heat::ResourceGroup
OS::Nova::Server
OS::Cinder::Volume

26. OS::Neutron:: SecurityGroup for ops-manager

27. OS::Neutron:: SecurityGroup for agents

28. OS::Heat:: ResourceGroup

29. OS::Nova::Server

30. OS::Cinder::Volume

31. Mongo orchestration – installation steps
Ops-Manager
MongoD
Mongo Config
MongoS
Ops-Manager Backup
Deployer
MongoD
Mongo Config
MongoS
DB - Backup
MongoD
Mongo Config

32. Step 1: Create Ops-manager servers

33. Ops manager – deployment model
DB - Backup
Ops-Manager Backup
Ops-Manager
27017
27017
27017
27018
27018
27018

34. Step 2: Install Ops-manager software
Ops-Manager Backup
DB - Backup

35. Step 3: Create User
Ops-Manager
Ops-Manager Backup
DB - Backup

36. Step 4: Save Keys
Ops-Manager
Ops-Manager Backup
DB - Backup

37. Step 5: Create hosts for mongod/mongos/config instances
Ops-Manager
Ops-Manager Backup
DB - Backup

38. Agents deployment model
cinder
cinder
cinder
agent

39. Step 6: Mount Volumes for instances reserved for mongod
Ops-Manager
agent
agent
agent
Ops-Manager Backup
agent
agent
agent
DB - Backup
agent
agent

40. Step 7: Get keys Ops-Manager agent agent agent Ops-Manager Backup
DB - Backup
agent
agent

41. Step 8: Install agent and configure to point to ops-manager
Ops-Manager Backup
agent
agent
agent
DB - Backup
agent
agent

42. Step 9: Create deployer server
Agents
Ops-Manager
agent
agent
agent
Ops-Manager Backup
agent
agent
agent
DB - Backup
agent
agent

43. Step 10: Deployer executes REST requests on ops-manager, to create cluster
MongoD
Mongo Config
MongoS
Ops-Manager Backup
Deployer
MongoD
Mongo Config
MongoS
DB - Backup
MongoD
Mongo Config

44. That’s all nice, but life is a bit more complicated

45. Saving API keys and Group info
Problem:
No way to retrieve key once created, though we need it to configure all agents

46. Saving API keys and Group info
Solution:
Save info as Json file, on ops-manager static file location
(delete once installation is complete!)
agentApiKey: "2af e509520b2d3458b5cc"

47. Isolated Mode Problem:
No access to the outside world from the openstack environment
By default, ops-manager downloads all mongodb
software from the mongodb site

48. Isolated mode Solution:
- Configure Ops-Manager to work in an isolated-mode.
- Create a version Manifest identifying the allowed versions.
- Download and save the required tar files on the ops-Manager machine

49. Allowing Mongo versions
Store version manifest json file in Ops-Manager.

50. First User Creation

51. Demo

52. MongoDB Clusters Across Data Centers

53. Luckily, OpsManager can help us with that
Steps:
On primary site: Deploy fully working cluster as shown above
Then, on secondary:
Deploy agents
Create deployer
From deployer – access OpsManager on primary site to extend cluster

54. Desired Model DC2 DC1 Ops-Manager Ops-Manager Backup DB - Backup
MongoD
Mongo Config
MongoS
MongoD
Mongo Config
MongoS
Ops-Manager Backup
MongoD
Mongo Config
MongoS
MongoD
Mongo Config
MongoS
DB - Backup
MongoD
Mongo Config

55. In case of a real disaster
DC1
DC2
Ops-Manager
MongoD
Mongo Config
MongoS
MongoD
Mongo Config
MongoS
Ops-Manager Backup
MongoD
Mongo Config
MongoS
MongoD
Mongo Config
MongoS
DB - Backup
MongoD
Mongo Config

56. Arbitar addition DC2 DC1 DC3 Ops-Manager Ops-Manager Backup
MongoD
Mongo Config
MongoS
MongoD
Mongo Config
MongoS
Ops-Manager Backup
MongoD
Mongo Config
MongoS
MongoD
Mongo Config
MongoS
DB - Backup
MongoD
Mongo Config
DC3
Arbitar

57. OpsManager is a great tool
Takeaways
OpsManager is a great tool
Orchestration is more than calling ‘create cluster’
With some effort, we can automate it all