Vanderbilt Consulting / Amy K. C.S. Vanderbilt, Ph.D. Owner – Principal Consultant Vanderbilt Consulting Predictability of Dynamic Network Behavior – Unanswered Questions & Possible Directions Visualisation Network-of-Experts Supporting NATO Research Task Group IST-059/RTG-025 November 6-8, 2007 Predictability of Dynamic Network Behavior – Unanswered Questions & Possible Directions Visualisation Network-of-Experts Supporting NATO Research Task Group IST-059/RTG-025 November 6-8, 2007

Vanderbilt Consulting / Predictability of Dynamic Network Behavior BACKGROUND  Predicting the behavior of various types of networks to various influence factors is a hot topic  BUT is it is not moving forward with the speed expected  WHY??  There are a number of unanswered questions slowing the progress of this research  These questions must be answered before efficient, effective research can be carried out to fruitful end  Let’s examine a few of these Objective: To start a conversation that ends in a definition of how to answer these questions

Vanderbilt Consulting / Predictability of Dynamic Network Behavior QUESTIONS Q1 - How do we best define, in a useful way, a measure of predictability for networks of various types, over time, given various influence factors? Q2 - Under what conditions and to what degree do networks of various types and attributes behave in a predictable way to various influence factors? Q3 - Can network prediction tools and algorithms be sufficiently tested within simulated or modeled networks thereby avoiding human subject testing? Are such models sufficiently accurate? Q4 - How can we measure the difference between results given by models and what would be found in the real world? Q5 - What role does visualization play in measuring and understanding the degree of predictability (or lack thereof)? Q6 – how can “clean” data sets be produced for various types of networks to provide testbeds with realistic traffic and other elements?

Vanderbilt Consulting / Predictability of Dynamic Network Behavior DEFINING PREDICTABILITY Q1 - How do we best define, in a useful way, a measure of predictability for networks of various types, over time, given various influence factors?  Predictability is a measure of the upper limit of how accurate a given algorithm COULD be in predicting the response of a network to various influence factors  It will certainly be different for each type of network and for each influence factor  To discover the upper bound of predictability, we might measure the chaos inherent in each type of network  But if we are testing on simulations/models, the results may be biased since these models are - by their programming - less chaotic than real life.  Can we accurately measure the chaos inherent in a real network?

Vanderbilt Consulting / Predictability of Dynamic Network Behavior PREDICATBILITY OF NETWORKS Q2 - Under what conditions and to what degree do networks of various types and attributes behave in a predictable way to various influence factors?  If we knew which types of networks would react with what degree of predictability to various influence factors:  Significant time, effort and money could be saved by targeting program funds towards prediction programs that have a chance of higher accuracy  Efforts can be accelerated by avoiding programs that will not result in accurate predictions  Prediction results can be taken with the right size grain of salt – too often results are treated as gospel when error rates should be acknowledged and taken into account  This question might be answered via experimentation on simulated networks - Is that good enough?

Vanderbilt Consulting / Predictability of Dynamic Network Behavior TESTING ON SIMULATIONS Q3 - Can network prediction tools and algorithms be sufficiently tested within simulated or modeled networks thereby avoiding human subject testing? Are such models sufficiently accurate?  Evidence suggests that simulations are sufficient for several types of networks:  Computer networks  Some social networks including disease propagation applications  Can this question be put to bed once and for all? How?

Vanderbilt Consulting / Predictability of Dynamic Network Behavior SIMULATION ERROR Q4 - How can we measure the difference between results given by models and what would be found in the real world?  To increase confidence in results gathered on simulated networks, we need a measure of the error  How can we measure the error in results gathered from experimentation on simulations when those results are then transferred to the “real world”

Vanderbilt Consulting / Predictability of Dynamic Network Behavior ROLE OF VISUALIZATION Q5 - What role does visualization play in measuring and understanding the degree of predictability (or lack thereof)?  And what kinds of visualization will be most useful?  Visualization may be a simple and effective way to see the effect of various influence factors on networks of all types  Visualization may also bias the results since we as humans tend to “see” what we want to see

Vanderbilt Consulting / Predictability of Dynamic Network Behavior TESTBED DATA SETS Q6 – how can “clean” data sets be produced for various types of networks to provide testbeds with realistic traffic and other elements?  The DARPA Intrusion Detection Experiment data is a good although outdated example (closed computer network traffic)  How can we develop similar test bed data sets for other network types?  One way may be to accept real world networks (social and otherwise) where a certain sub-network is modeled in detail based on historical (and hopefully unbiased) data.  Such testbed data sets may be the first step towards answering the rest of these questions

Vanderbilt Consulting / Predictability of Dynamic Network Behavior STRAWMAN SOLUTION Develop testbed data sets for major network types (based on framework categorizations) Develop a measure of chaos for each network type Make these data sets and predictability measures available to the research community Carry out comparative experiments between real world networks and these simulated networks to measure error

Vanderbilt Consulting / Predictability of Dynamic Network Behavior OPEN FLOOR QUESTIONS AND THOUGHTS? Amy K. C.S. Vanderbilt, Ph.D. Owner – Principal Consultant Vanderbilt Consulting