High-Performance Simulations of Complex Networked Systems for Capturing Feedback and Fidelity Kalyan S. Perumalla, Ph.D. Senior Research Staff Member Oak Ridge National Laboratory Adjunct Professor Georgia Institute of Technology Kalyan S. Perumalla, Ph.D. Senior Research Staff Member Oak Ridge National Laboratory Adjunct Professor Georgia Institute of Technology SIAM PP Conference, Seattle Feb 26, 2010

2Managed by UT-Battelle for the U.S. Department of Energy SIAM PP10 Presentation – Perumalla (ORNL) Main Thesis Model Feedback and Fidelity Feedback is appearing as essential in recent models Feedback is appearing as essential in recent models – Across a range of domains and applications Feedback seems to necessitate higher fidelity Feedback seems to necessitate higher fidelity – To integrate phenomena with network dynamics Implications to Computation Computation needs overall are dramatically increased Computation needs overall are dramatically increased – To simulate new models – E.g., O(N) to O(N 2 ), O(N 2 ) to O(N 3 ) High performance computing seems necessary High performance computing seems necessary – For fast simulation of feedback- heavy, high-fidelity models

3Managed by UT-Battelle for the U.S. Department of Energy SIAM PP10 Presentation – Perumalla (ORNL) Static Networks Abstraction of Elements N : Network properties N : Network properties – Connectivity structure – E.g., Internet, roads, contacts P(t) : Phenomena of focus P(t) : Phenomena of focus – E.g., entity movement, device state, cognitive attributes Static Network Relation N N P(t) SnSn

4Managed by UT-Battelle for the U.S. Department of Energy SIAM PP10 Presentation – Perumalla (ORNL) Dynamic Network without Feedback Abstraction of Elements N(t) : Network properties N(t) : Network properties – Structure, behavior – E.g., Traffic signals, packet store- forward, electric frequency P(t) : Phenomena of focus P(t) : Phenomena of focus – Similar to previous (static) – Or slightly more refined/complex S n (t) : State of network S n (t) : State of network – Needed for P(t) dynamics – E.g., Dynamic-Network Relation N(t) P(t) S n (t)

5Managed by UT-Battelle for the U.S. Department of Energy SIAM PP10 Presentation – Perumalla (ORNL) Feedback in Networked Systems Abstraction of Elements N(t) : Network dynamics N(t) : Network dynamics – Structure, behavior, reactions, coupling P(t) : Phenomena of focus P(t) : Phenomena of focus – More tightly integrated operation – E.g., S n (t) : State of network S n (t) : State of network – Needed for P(t) dynamics S p (t) : State of phenomenon S p (t) : State of phenomenon – Feedback of P(t) into N(t) Nature of Feedback N(t) P(t) S n (t) S p (t)

6Managed by UT-Battelle for the U.S. Department of Energy SIAM PP10 Presentation – Perumalla (ORNL) Feedback-Heavy Networks: Illustrations Simulation Domains Cyber security Simulations Cyber security Simulations Transportation Simulations Transportation Simulations Epidemiological Simulations Epidemiological Simulations Electric Grid Simulations Electric Grid Simulations Several other, emerging domains Several other, emerging domains Feedback examples E.g., bandwidth-limited worms E.g., bandwidth-limited worms E.g., dynamic re-routing E.g., dynamic re-routing E.g., quarantines and curfews E.g., quarantines and curfews E.g., smart grid devices E.g., smart grid devices E.g., online-pricing, peer-to- peer networks E.g., online-pricing, peer-to- peer networks

7Managed by UT-Battelle for the U.S. Department of Energy SIAM PP10 Presentation – Perumalla (ORNL) Example: Disease or Worm Propagation SI Model Typical dynamics model Typical dynamics model – Multiple variants exist (e.g., SIR) Excellent fit Excellent fit – But post-facto (!) – Plot collected data Difficult as predictive model Difficult as predictive model – Great amount of detail buried in α Feedback for accuracy Feedback for accuracy – Interaction topology – Resource limitations – E.g., bandwidth-limited worm P(t)

8Managed by UT-Battelle for the U.S. Department of Energy SIAM PP10 Presentation – Perumalla (ORNL) Transportation - Example Dynamic, semi-automated rerouting Dynamic decisions Dynamic decisions – Individual-level – Semi-automated – Fairly tight feedback Vastly higher computational load Vastly higher computational load – Shortest path per individual – Hard to aggregate – Poorly understood (analytically) Fundamental in nature Fundamental in nature – Needs duplication of M real computing units on fewer simulation computers units Evacuation time estimation or Emission control analysis

9Managed by UT-Battelle for the U.S. Department of Energy SIAM PP10 Presentation – Perumalla (ORNL) Transportation – Our Approach SCATTER – A Systems Approach to Scalable Vehicular Mobility Models, Winter Simulation Conference 06 – Kinetic and Non-Kinetic Aspects…, Europen Modeling and Simulation Symposium 06 Large-scale parallel discrete event simulation Efficient, cache-based on-demand shortest path computations SCATTER – A Systems Approach to Scalable Vehicular Mobility Models, Winter Simulation Conference 06 – Kinetic and Non-Kinetic Aspects…, Europen Modeling and Simulation Symposium 06 Large-scale parallel discrete event simulation Efficient, cache-based on-demand shortest path computations

10Managed by UT-Battelle for the U.S. Department of Energy SIAM PP10 Presentation – Perumalla (ORNL) Electric Grid Smart Grid Simulation-based analysis indispensible Simulation-based analysis indispensible Feedback absolutely essential Feedback absolutely essential Fidelity of grid model must be elevated Fidelity of grid model must be elevated Computational load is dramatically increased Computational load is dramatically increased Smart Devices and Appliances

11Managed by UT-Battelle for the U.S. Department of Energy SIAM PP10 Presentation – Perumalla (ORNL) Electric Smart Grid – Our Approach Discrete Event Formulation Detect threshold crossings at devices Detect threshold crossings at devices Evolve state of phenomenon P (electric) Evolve state of phenomenon P (electric) – Conventional equations – Coupled differential equations Incorporate network changes at threshold crossings Incorporate network changes at threshold crossings Very heavy computation Very heavy computation – Detection of crossings – Revision of network Conventional (static): O(n 2 ) New (dynamic): O(n3) n is large ~ 10 4 buses

12Managed by UT-Battelle for the U.S. Department of Energy SIAM PP10 Presentation – Perumalla (ORNL) Epidemiology – Feedback Example Network structure dictates and is defined by evolution Network structure dictates and is defined by evolution Interventions change network structure, in turn evolution Interventions change network structure, in turn evolution Evolution triggers dynamic interventions Evolution triggers dynamic interventions Analytical approaches yet to be refined Analytical approaches yet to be refined Asymptotic behaviors are relatively well understood Asymptotic behaviors are relatively well understood Transients are poorly understood, hard to predict well Transients are poorly understood, hard to predict well Can be an extremely challenging simulation problem Can be an extremely challenging simulation problem

13Managed by UT-Battelle for the U.S. Department of Energy SIAM PP10 Presentation – Perumalla (ORNL) Epidemiology – Our Approach Optimistic parallel simulation Optimistic parallel simulation – Cray XT5, Blue Gene P Reverse computation for efficient simulation Reverse computation for efficient simulation – New discrete event formulation – Support dynamic decisions – Quarantines, curfews, closures, vaccinations Switching to High Gear… Switching to High Gear… – Distributed Simulations and Real- time Applications 09 Additional recent work to appear soon Additional recent work to appear soon Optimistic parallel simulation Optimistic parallel simulation – Cray XT5, Blue Gene P Reverse computation for efficient simulation Reverse computation for efficient simulation – New discrete event formulation – Support dynamic decisions – Quarantines, curfews, closures, vaccinations Switching to High Gear… Switching to High Gear… – Distributed Simulations and Real- time Applications 09 Additional recent work to appear soon Additional recent work to appear soon Image from psc.edu

14Managed by UT-Battelle for the U.S. Department of Energy SIAM PP10 Presentation – Perumalla (ORNL) Recent results

15Managed by UT-Battelle for the U.S. Department of Energy SIAM PP10 Presentation – Perumalla (ORNL) Cyber-infrastructure Bandwidth-limited worms Bandwidth-limited worms E.g., Slammer worm of early 2000s E.g., Slammer worm of early 2000s Throttling and spread are tightly inter-twined Throttling and spread are tightly inter-twined

16Managed by UT-Battelle for the U.S. Department of Energy SIAM PP10 Presentation – Perumalla (ORNL) Cyberinfrastructure: Packet-Level Simulations Number of packet transmissions simulated per sec Campus network scenario (500 packets/flow, TCP) Scales well to large number of processors, up to 4 million nodes 106 million pkt hops/sec

17Managed by UT-Battelle for the U.S. Department of Energy SIAM PP10 Presentation – Perumalla (ORNL) Summary Feedback is a major differentiator – Compared to previous models and simulations Feedback is a commonality we see in a variety of domains In all, computation is dramatically increased High Performance Computing seems essential – In absence of additional insights into the models – Or, alternatively, to gain additional insights into the models Feedback is a major differentiator – Compared to previous models and simulations Feedback is a commonality we see in a variety of domains In all, computation is dramatically increased High Performance Computing seems essential – In absence of additional insights into the models – Or, alternatively, to gain additional insights into the models