1. A Marriage of Open Source and Custom William Deck
Monitoring HTCondor
A Marriage of Open Source and Custom
William Deck

2. Goals
Help those looking to augment/roll out their own cluster Monitoring
Possibly a different perspective on monitoring
Get some of this work out in the open

3. Who am I? Originally a mainframe developer
Moved to SIG January of 2015
First task: Monitor the “cluster”

4. Our Cluster Mixed Cluster (Several) Storage ~1700 cores Lustre: 2.8 PB
1100 Windows
600 Linux (SLES11/SLES12)
Storage
Lustre: 2.8 PB
GPFS: 8.4 PB
independent clusters across the world

5. Our Workload ~100K jobs/week Mixed OS Workloads (Majority Windows)
Daily Workflows (1 – 10K)
One Off Workflows (+100K)
Enough about me!

6. What is the Problem? Many moving parts
Multiple groups using condor differently
Single submit files
DAGs
Someone found out about condor_run
Monitor 1 – 100k jobs
Hard for users to self-support
Correlate job failures to infrastructure problems
cluster begets problems
goal is to keep the error/retry rate down while giving users the ability to track down errors

7. Monitor Evolution HTCondor’s Job Monitor CycleComputing Solution
Custom Scripts parsing condor_* commands
Python bindings with Elastic, Grafana, and Conmon

8. First Solution – Tales of Condor
Modelled after CycleComputing monitoring
Parsing condor_status, condor_history output
Put information into csv files
Simple webpage displayed
Total jobs vs running jobs
Total jobs/User vs running jobs/User
Coorelation between condor monitoring and infrastructure

9. First Solution: Revolutionary command grep
For everything else we used grep
Painful and tedious
Good news!
Really good at grep
ssh and grep condor logs

10. Enter: Monmon
Custom webpage designed around our workflow creation scripts
Parses nodestatus/HTCondor log files
Specific for one group’s workflows
Useful for the user to drill down and share with others

11. Enter Elastic, Grafana, and Conmon
Elastic (ELK) (
Elasticsearch - Search/analytics engine
Logstash - Ingest of data
Kibana - Frontend
Grafana - (
extension of Kibana (3.0)
Multiple backends (graphite, ES, influxdb ...)

12. Enter Elastic, Grafana, and Conmon
Poll periodically using python bindings
Insert into ES
Augmented Job/Host ClassAd
Custom subset of ClassAd
Use Grafana frontend
Tales of Condor 2.0
More Complex Dashboards
Extended Monmon -> Conmon

13. ES Job Ad Example Searchable Job ClassAd Common Uses condor_history
Used Resources
search: kibana frontend, curl, python
complete history of all schedds completed jobs
Used/requested resources
cluster utilization
possiblities are endless
ES led to Grafana

14. Grafana: One Stop Shop

15. Grafana Dashboards
Easily create dashboards from ES and performance metrics
Single Pane of Glass for Condor and Infrastructure

16. Tales of Condor 2.0

17. Overall Cluster Health
This dashboard has every layer of our stack from
NetApp/Storage
lustre (infrastructure)
exec
Job stats
schedds
workstations
application results,
A rousing game of find the bottleneck

19. Grafana: Error Rates Find blackhole exec Postscript vs. ExitCode
Errors by:
User
DAG
Is it User or infrastructure
Good to keep track of postcheck vs programmatic failures

20. Conmon Uses Flask ES Queries Display job classad Workflow overview
Grid View
DAG/Job Logs
Workflow Analysis

21. Conmon: Home (DAGs)

22. Conmon: Workflow (DAGs)

23. Conmon: Grid (DAGs)

25. Workflow Analysis
Information from ES
Easy to see long legs

26. Individual Jobs

27. Conmon: Benefits Loading Condor information from ES
Can handle multiple submission/workflows types (DAGs, submit)
User can click through jobs
Search/Filter
Shareable
Consistent views

28. Questions?
Late to the party. Based on Todd’s talk it looks like a lot of what I did a while back is in condor.