1. The Life of a MongoDB GitHub Commit
Evergreen
The Life of a MongoDB GitHub Commit
This is a talk about how we do continuous integration at MongoDB. We’ll be a talking about Evergreen, our homegrown CI system, which we developed because of the unique testing needs that MongoDB has. Here’s some context:
Describe MongoDB, - whether or not you have any view of the database will not affect your view of this talk. ]
GENERAL PURPOSE OPEN SOURCE DISTRIBUTED DATABASE

2. This Talk Will Provide an overview of CI at MongoDB
Journey through the Evergreen internals.
Share some useful lessons we’ve learned along the way.
Not a talk about our tests

3. Continuous Integration
Travis
Jenkins
TeamCity
Buildbot

4. Testing MongoDB is not a typical CI use case

5. MongoDB queries storage backups monitoring authentication aggregation
Does a lot of things.
queries
storage
backups
monitoring
authentication
aggregation
replication
sharding
MongoDB has a variety of features that interact in many ways.

6. 20+ hours in serial MongoDB Has a lot of tests
MongoDB has a variety of features that interact in many ways.

7. MongoDB Runs in a lot of places. Linux Windows Solaris OSX
We support MongoDB on a many different platforms, from Linux to Windows, and architectures like x86 and PowerPC

8. Where can we run those tests?

9. We needed something with...
Dynamic Host Allocation AND Static Hardware
Multiplatform Support
Powerful Navigation
Open Source Licensing

10. Our in-house continuous integration system
Evergreen
Our in-house continuous integration system

11. High Level
Elizabeth Programmer commits a change.

12. High Level

13. How do the pieces fit together?

14. Server Web User Interface Logs & Results Commits Tasks Agents
Task Runner
Agents
Repo Tracker
Scheduler
Host Initializer

15. The life of a commit to the MongoDB codebase
We need to see how we get from (commit) to (screenshot of a finished version)

16. A commit is born
An engineer pushes a commit to mongodb/mongo master on GitHub. (TODO make this s

17. Picked up by Repotracker
a312b7e
d5a2826
Repotracker sits, ever diligent, checking for new commits every once in a while. When it finds a commit it picks it up.

18. Project Configuration
tasks:
-name: jsCore
commands:
- func: "do setup"
- func: "run tests"
vars:
resmoke_args: --suites=core --storageEngine=mmapv1
run_multiple_jobs: true
buildvariants:
- name: linux-64
display_name: Linux
run_on:
- rhel55
- rhel55-test
Repo Tracker
d5a2826
evergreen.yml

19. Create tasks for the commit
Repo Tracker
a312b7e
d5a2826
24ac5dd

20. Tasks Sharding Tests on Solaris Auth Tests on RHEL 6.2 Enterprise
Compile on Amazon Linux!

21. Schedule the tasks
Scheduler

22. Windows 10
Solaris
RHEL 6.2
OSX 10.10
Amazon Linux

23. Host Initializer

24. Agent
Task Runner
TODO: Add a slide on Go.

26. Run the Task
Agent: Amazon Linux #1234
Task: sharding

27. name: sharding
depends_on:
name: compile
commands:
func: “do stuff”
func: “run tests”
name: replication
func: “different”
Sharding
Sharding

28. Logs and Results Sent Back
Notifications
sharding_test1
results.json
Aggregate Statistics

29. results.json
sharding_test1

31. Server Web User Interface Logs & Results Commits Tasks Agents
Task Runner
Agents
Repo Tracker
Scheduler
Host Initializer

32. Scheduler Transition to the scheduler internals.
*Scheduler is responsible for creating the task queue and ensuring that there are enough hosts to run all the tasks in a reasonable amount of time.

33. Server Web User Interface Logs & Results Commits Tasks Agents
Task Runner
Agents
Repo Tracker
Scheduler
Host Initializer

34. Schedule the tasks Scheduler After the repo tracker creates the tasks…
Scheduler find all unscheduled tasks and adds them to priority queues that are platform specific.

35. Allocate New Hosts Host Initializer Scheduler
After the task queues are created, the scheduler determines how many additional hosts need to be spun up based on how many hosts exist and how many tasks there are.

36. Scaling Load! Hosts when we need them; not when we don’t.
Amazon EC2 Spending $
Graph of EC2 Usage - amazong
Days in the Month

37. Windows 10 Solaris RHEL 6.2 OSX 10.10 Amazon Linux
Multiplexing all of the tasks that are being created by the repotracker, for different commits and maybe even for different projects and putting them into tasks queues that are solely based on the build variants.

38. Windows 10 Solaris RHEL 6.2 OSX 10.10 Amazon Linux Prioities
Dependenceis

39. Windows 10 Solaris RHEL 6.2 OSX 10.10 Amazon Linux
Now that we know the reasons for leveraging elastic computing to run our tests, lets take a look at how the scheudler figures out how many tests to run.
I am going to talk about Amazon EC2 Linux

40. Task Queue 5 15 15 30 Amazon Linux Task Queue
MENTION: HOW WE CALCULATE ESTIMATED TASK DURATION

41. EC2 Hosts 60 Amazon Linux EC2 Host
MENTION: We are using the Amazon billing scheme

42. Goals Minimize Time in Task Queue (TQT) Minimize Idle Host Time 🤑
Go into depth of what Kyle said about why we created our own. We need to be conservative.
We want to Save Developer Time - meaning we want tasks to be scheduled and run as fast as possible. We don’t want the task queue to be too long.
We want to save money, meaning we want to spin up the minimum amount of hosts possible.
Note that Developer time in itself varies in many ways
These two things contradict each other.

43. Example 30 5 15 Amazon Linux Tasks
We have a set of tasks and a set of hosts and we need to figure out the optimal ordering of these tasks on hosts.

44. One Task Per Host 30 5 15
Amazon Linux EC2 Hosts (4) 🤑🤑🤑🤑 🙂🙂 45 30 60 15 5
0 TQT
0 TQT
0 TQT
0 TQT

45. With n tasks... Amazon Linux EC2 Hosts (n) 🤑🤑🤑🤑🤑🤑🤑🤑🤑🤑🤑🤑🤑🤑🤑🤑 5 55 15 45🙂 5 55 🙂 15 45 🙂 30 30 🙂 15 45 🙂 5 55 🙂 15 45 🙂 30 30 🙂 15 45 🙂 5 55 🙂 15 45 🙂 30 30 🙂 15 45 🙂 5 55 🙂
TODO: Make these proportional 15 45 🙂 30 30 15 45

46. Fit Tasks On As Few Hosts as Possible30 5 15
Amazon Linux EC2 Hosts (1) 🤑🤑 45 30 5 15 🙂🙂 😕 ☹️
0 TQT
30 TQT
45 TQT
😭 😭
60 TQT
For example...

47. Job Shop Scheduling Minimize Makespan
For example...

48. Makespan = Total Length Of Schedule
For example...

49. One Task Per Host 30 5 15
Amazon Linux EC2 Hosts (4) 🤑🤑🤑🤑 🙂🙂 45 30 60 15 5
0 TQT
0 TQT
0 TQT
0 TQT

50. Fit Tasks On As Few Hosts as Possible30 5 15
Amazon Linux EC2 Hosts (1) 🤑🤑 45 30 5 15 🙂🙂 😕 ☹️
0 TQT
30 TQT
45 TQT
😭 😭
60 TQT
For example...

51. Job Shop Scheduling Minimize Makespan
NP HARD!
Job Shop Scheduling
Minimize Makespan
NP hard - means that a problem can be reduced in polynomial-time into an NP problem. The Job Shop Scheduling is a known NP Hard problem. We use the Flexible Flow Job Shop problem, which is the same problem, except that there is a strict order of operations (depen

52. NP Hard = Approximations

53. Duration Based Estimation30 5 15
Estimated Task Duration
Maximum Time in Task Queue

54. Duration Based Estimation
Amazon Linux Tasks 5 15 15 30
Amazon Linux EC2 Hosts (2) 60 60
We have a set of tasks and a set of hosts and we need to figure out the optimal ordering of these tasks on hosts.
Bring back the point from Kyle’s talk about how we want to not be waiting a week and not have control.
Developer Wait time (TODO CREATE A NAME)
Total Task Time = 65 = 2 Hosts

55. Duration Based Estimation
Amazon Linux Tasks 5 15
Amazon Linux EC2 Hosts (2) 30 30 15 45
We have a set of tasks and a set of hosts and we need to figure out the optimal ordering of these tasks on hosts.
Bring back the point from Kyle’s talk about how we want to not be waiting a week and not have control.
Developer Wait time (TODO CREATE A NAME)

56. Duration Based Estimation
Amazon Linux Tasks 5 15
Amazon Linux EC2 Hosts (2) 30 30 15 45
We have a set of tasks and a set of hosts and we need to figure out the optimal ordering of these tasks on hosts.
Bring back the point from Kyle’s talk about how we want to not be waiting a week and not have control.
Developer Wait time (TODO CREATE A NAME)

57. Duration Based Estimation
Amazon Linux Tasks 5
Amazon Linux EC2 Hosts (2) 30 30 15 15 30
We have a set of tasks and a set of hosts and we need to figure out the optimal ordering of these tasks on hosts.
Bring back the point from Kyle’s talk about how we want to not be waiting a week and not have control.
Developer Wait time (TODO CREATE A NAME)

58. Duration Based Estimation
Amazon Linux Tasks 5
Amazon Linux EC2 Hosts (2) 30 30 15 15 30
We have a set of tasks and a set of hosts and we need to figure out the optimal ordering of these tasks on hosts.
Bring back the point from Kyle’s talk about how we want to not be waiting a week and not have control.
Developer Wait time (TODO CREATE A NAME)

59. Duration Based Estimation
Amazon Linux Tasks
Amazon Linux EC2 Hosts (2) 30 5 25 15 15 30
We have a set of tasks and a set of hosts and we need to figure out the optimal ordering of these tasks on hosts.
Bring back the point from Kyle’s talk about how we want to not be waiting a week and not have control.
Developer Wait time (TODO CREATE A NAME)

60. Duration Based Estimation
Amazon Linux Tasks 5
Amazon Linux EC2 Hosts (2) 30 5 5 20 15 15 10 20
We have a set of tasks and a set of hosts and we need to figure out the optimal ordering of these tasks on hosts.
Bring back the point from Kyle’s talk about how we want to not be waiting a week and not have control.
Developer Wait time (TODO CREATE A NAME)

61. Duration Based Estimation
Amazon Linux Tasks
Amazon Linux EC2 Hosts (2) 30 5 5 5 15 15 15 10 20
We have a set of tasks and a set of hosts and we need to figure out the optimal ordering of these tasks on hosts.
Bring back the point from Kyle’s talk about how we want to not be waiting a week and not have control.
Developer Wait time (TODO CREATE A NAME)

62. Success!
With that we are able to create a good approximation for a very difficult problem.

63. Future Work
Collecting more scheduler statistics to be able to optimize the scheduler
It would be cool to maybe use stochastic optimizations to make the scheduler even better.
As it is simple solution works well

64. Thank You! github.com/evergreen-ci/evergreen evergreen.mongodb.com