1 RADS Conceptual Architecture Commodity Internet & IP networks Edge Network Distributed Middleware Client SLT Services Distributed Middleware Server Router Edge Network PNE Prototype Applications: E-voting, Messaging, , etc. Operator User Application- Specific Overlay Network Programming Abstractions For Roll-back (Necula Crash-Only Middleware & Servers, System O&C Infrastructur e (Fox) Protocols Enabling Fast Detection & Route Recovery, Network O&C Infrastructure (Katz, Stoica) Online Statistical Learning Algorithms (Jordan) Benchmarks, Tools for Human Operators (Patterson) Reduction to practice of online SLT and observe/analyze/act infrastructure Reusable embeddable components SLT Services

2 Apps and Science Messaging (Randy’s scenario) Voting systems Online medical records system “Volunteer coordination” for disaster response

3 What are “SLT Services”? “SLT clients” are client or server apps, middleware or OS layer, machine hardware, programmable network elements,... Monitoring hooks for SLT clients Control hooks for SLT clients Database(s) for aggregating SLT client data Plug-ins for online and offline analysis

4 Macroscopic behaviors Application diversity “Fail over” to another whole infrastructure –Completely separate app architecture (client, server, middleware, –Free: provisioning across different services (eg messaging) Use VM/appliance based migration for the servers

5 Reflections from 9/11 (from Douglas Yoshida, MD, Bellevue Hosp & NYU Med Ctr) In a crisis, patients needing medical attention brought to closest hospital, not most appropriate hospital (absent better information) Baseline EMS comms in NYC: no direct contact between ED’s and ambulances; sometimes doctors would scramble to “clear out” ER’s, then wait for hours for patients to arrive Cell phone and landline failure impeded communication between hospitals –“Needed separate inter-hospital radio comms with direct link to onsite command center”

6 More reflections Families flooding hospitals trying to find out about their loved ones –No other way to get the info out –Creates potential security nightmare for hospital (“If terrorists had wanted to attack hospitals, it would have been easy”) Lack of info leads to frustration and “disaster voyeurism” –Med students and attendings flocked down to Ground Zero because they were frustrated at not being able to help w/in their own hospital –Too many doctors around each stretcher; poor allocation/distribution of resources

7 Multiple communication channels Closed: inter-hospital Semi-closed: hospital/command site/firefighters etc Open/unidirectional: communication to public about condition of victims (can be largely unidirectional) Open/bidirectional: volunteer coordination

8 Recovery & policy DB External monitors Mdlware monitors OS monitors PNE monitors App-specific monitors Fault injection Client app Server app Middleware OS Overlay/PNE’s Internet Fault injection Fault injection Fault injection Anomaly det Novelty det Cluster- ing Results fusion Policy selection

9 EJB J2EE Application From JBoss to JAGR Application Server (JBoss) Persistence tier Fault Injector Recovery Agent Internal Monitors Recovery Map External Monitors Servlet/JSP Container Http Server Stall Proxy Client Requests l Before deployment, use controlled faults to build Recovery Map l Builds fault propagation map, based on observed failures l Restart single EJBs, redeploy apps, or restart whole app-server l E2EMon: detects app-specific, end-to-end failures in requests (also app-generic using character histograms) l Stalls user requests during recovery

10 EJB J2EE Application From JBoss to JAGR Application Server (JBoss) Persistence tier Fault Injector Recovery Agent Internal Monitors Recovery Map External Monitors Servlet/JSP Container Http Server Stall Proxy Client Requests l Before deployment, use controlled faults to build Recovery Map l Builds fault propagation map, based on observed failures l Restart single EJBs, redeploy apps, or restart whole app-server l ExcMon: detects Java exceptions in the application & app server l PPMon: detects “anomalous” behaviors l E2EMon: detects app-specific, end-to-end failures in requests (also app-generic using character histograms) l Stalls user requests during recovery