declarative distributed debugging Kuang Chen, Gunho Lee, Byung-Gon Chun

Motivation Debugging distributed systems is hard Logs are scattered and numerous Log content can vary –Application-specific logs: aka printf statements –Tracing: X-Trace Needed: a more formal and efficient way to use logs in debugging.

State of the Art Centralized log processing Master “controller” node collects logs for processing Techniques for filtering/aggregation before transmission to the controller –Logs size and # of nodes can be huge –Debugging look for needles in the haystack –Many logging mechanisms are ad-hoc (e.g. printf statements) and focus on how instead of what.

New Idea Towards more efficient and formal distributed debugging –Step one: declarative querying of logs Queries are sent out. Logs do not move –Specify queries and dataflow declaratively –Process queries in a distributed manner “Declarative” - a separation of what from how –What: a language for specifying distributed debugging queries and data specification for tracing the causality of events –How: recursive query engine (e.g. P2) and tracing infrastructure (e.g. X-Trace)

Risks Query language - (OverLog, NDLog, DPLog …coincidence?) –Sufficient for expressing logging and tracing queries? Implementation –P2 has limited in-memory persistence –Need datacenter specific (potentially real-time) X-Trace data e.g. Hadoop X-Trace data Success metrics –Working prototype e.g. creates causality graph of debugging use case –Show declarative query specification is concise / easy

6 The Plan DueTaskDescriptionDeliver 10/12Literature review Understand related system Read papers Play with P2 and X-Trace Initial Proposal 10/19Design brainstormingFind debugging use casesRequirement document 10/26Initial design Base implementation Integrate X-Trace (or its data) into P2Base system Revised Proposal 11/02Further designImplement basic use case Define advanced use case Design document 11/09Basic queryDeployment!Running example 11/16Advanced query Design refinement Implement advanced use caseAdvanced example 11/23AnalysisEvaluate results Make poster Project Poster 11/30SummarizeDiscussion Write report Final Report