1 Reliable Adaptive Distributed Systems Armando Fox, Michael Jordan, Randy H. Katz, David Patterson, George Necula, Ion Stoica, Doug Tygar

2 Motivations and Observations Existing systems: –Fragile, easily broken, poor reliability/security –Overwhelming configuration complexity, infrequently correctly performed, yielding lack of dependability, significant vulnerabilities Magic bullets? –Statistical learning theory: foundation for algorithms that observe/predict future behaviors –Verification technology—check for correct behavior, reveal vulnerabilities, automatically generate behaviors with desirable properties –Programmable network elements—“active” code inserted into network, provide observation/enforcement points without access to user end systems

3 The Team and Approach Team: –Statistical Learning Theory (SLT) (Michael Jordan) –Network Services/Protocols (Armando Fox, Randy Katz, David Patterson, Ion Stoica) –Verification Methods for program, network, security behaviors (George Necula, Ion Stoica, Doug Tygar). Approach: comprehensive distributed system architecture –SLT building block/practical components for distributed system observation, coordination, inference, correction, and evolution of behaviors –Network behaviors and how they reveal operation of higher-level network applications –Key enabler: embed observational and inference means at strategic points in the network, avoiding modification of end hosts or apps (aka “knowledge plane”) –Apps: web services, intrusion detection, storage access, etc. Improvements: –Dependability enhanced by monitoring network state, rapidly detecting behavioral changes (e.g., failures), configuring new resources in response –Security enhanced thru more rapid discovery of and response to attacks

4 Old System Block the in-coming attack Contain outgoing attacks Patch here once Too difficult to apply patches here Old System Old System Old System Network is patched Block things rather than upgrade software! Not the end systems Network is smart Not the end systems Fragile to change, upgrade Secure the edge network Redundancy Isolation/Containment Heterogeneity

5 Make-A-Difference Technologies Statistical Learning Theory (Michael Jordan) –Toolbox for the design and analysis of adaptive systems –New and scaled-up algorithms for classification, diagnosis, prediction, novelty detection, outlier detection, quantile estimation, density estimation, feature selection, variable selection, response surface optimization, sequential decision-making »kernel machine: functional analysis plus convex optimization, yielding generalized inner product to measure similarities among data point pairs »novelty detection/quantile estimation problems: given cloud of data in feature space, place boundary so as to guarantee only small fraction falls outside (second-order cone program— convex optimization with efficient solution methods) –Challenge: make these algorithms work “on-line” while embedding them within network and distributed systems architectures

6 Make-A-Difference Technologies New Approach to Run-Time Error Handling (George Necula) –Program-level error-handling is difficult »Repetitive, hard to maintain and specify »Standard run-time approach is abort or ignore! –Necula’s approach »Add explicit support at the programming language level »Make compensations and interface obligations First-class citizens (like objects) With Static and Dynamic checking »Handles tricky features (e.g., loops, the heap) »Still provides strong guarantees Like sagas or compensating transactions Standard transactions not appropriate for this use

7 Implementation Platform based on Programmable Networks Interconnection Fabric Input Ports Output Ports Buffers Tag Mem CP AP Action Processor CP Classification Processor Rules & Programs

8 Short Statements Patterson/Fox: Dependability of Distributed Systems Stoica: Trust in Distributed Systems through Protocol Verification Tygar: Security in Distributed Systems