Choosing an Accurate Network Model using Domain Analysis Almudena Konrad, Mills College Ben Y. Zhao, UC Santa Barbara Anthony Joseph, UC Berkeley The First International Workshop on Performance Modelling in Wired, Wireless, Mobile Networking and Computing (PMW2MNC, July 2005)

Modeling Network Measurements Model-driven traces as experimental tool –Real measurements costly to perform –Produce models by extracting statistics from measurements –Use model to generate traces w/ “similar” characteristics New networks (wireless) difficult to model –Traditional models require stationary statistics –Gilbert model, Higher order DTMC, Hidden Markov Models (HMM) –Artifacts disrupt stationarity of wireless traces Bursty error or delays due to signal interference or loss –Result: traditional models generate inaccurate results Our solution: –Data preconditioning: preprocess traces before extracting model –Result: MTA, MMTA accurately model wireless traces

A New Challenge Many mathematical models are available –Gilbert, Higher order DTMC, HMM, MTA, MMTA… –Each optimized for certain characteristics –Each with different computational and memory costs Different networks exhibit different characteristics –IP networks, wireless (802.11), sensor nets The challenge –Match up network type with more accurate, lowest cost model –No tools or techniques to do this Our solution: –Domain analysis

Outline Motivation Data preconditioning and its models –The Markov-based Trace Analysis model (MTA) –The Multiple states MTA model (MMTA) Quantifying modeling accuracy Domain Analysis Conclusion

Modeling using Data Preconditioning Collect network characteristic trace –Binary traces show occurrence of an event, e.g. lost or delayed frame Identify data patterns or “states” (regions with stationary behavior) Precondition the data to fit traditional models –Calculate probability distribution for each state –Determine transition probabilities among states Collected Trace Subtrace 1 Subtrace 2 Subtrace 3

Preconditioning Model (MTA) MTA: Markov-based Trace Analysis Algorithm –Two states: lossy & error-free states –Create two subtraces  lossy & error-free subtraces –Algorithm to optimize the change-of-state variable, C For a given trace, executes stationary test for large range of values C Choose the highest C that yields a stationary lossy states –Model subtraces as Exponential distributions C … … … … Lossy Error-free C Trace: Lossy sub-trace: Error-free sub-trace: … …00... … 0000… States:

Preconditioning Model (MMTA) MMTA: Multiple states MTA Algorithm –Allows for multiple states in the data –Identifies states in original trace –Concatenates similar states to create subtraces –Uses a DTMC to model subtraces & transition between states

Network Path Traces A selection of end to end & wireless layer path traces … … –End-to-end IP traces (IP-1) –End-to-end WLAN delay trace (WLAN-D) –Wireless WLAN trace (WLAN-E) –Wireless GSM trace (GSM-E) Trace collected at the Radio Link Protocol (RLP) Traces can be decomposed into two stationary states –Lossy & error-free states Characterize traces with three parameters (Lexp, EFexp, Lden) –Exponential length distributions of lossy and error-free states –Lossy error density

Quantifying Accuracy of Models Previously used metrics –FER and simulated transmission time –No actual correlation to error distribution How accurate is a particular model? –Synthetic traces comparison ( Lexp, EFexp, Lden) –Compute correlation coefficient between error-burst distributions Relationship between cc and accuracy: CC > 0.96 indicates an accurate model Reference trace: (Lexp, EFexp, Lden)= (0.006, 0.1, 1)

Example of Model Accuracy: GSM Error Burst Distributions GSM, MTA & MMTA experience similar error burst characteristics Gilbert, 3 rd order and HMM don’t reproduce large error burst CC values Gilbert: 0.74 HMM : 0.89 MTA : 0.99 MMTA : 0.99

Choosing the Right Network Model: Domain Analysis Preconditioning models also work well for traditional networks –There could be a “bursty” nature to congestion loss in IP networks –Key is the presence of “bursty” behavior Need a way to choose optimal model for given network –Solution: domain analysis Domain analysis –Create Domain Analysis Plots (DAPs) for wide range of parameters –Collect empirical packet trace: T= … … 1: corrupted/delayed packet, 0: normal packet –Calculate parameters (Lexp, EFexp, Lden) for T –Use DAPs to lookup optimal model

Domain Analysis Plots (DAPs) Generate reference traces (T 1..T x ) –Range of values (Lexp, EFexp, Lden) For each reference trace T i create –mathematical models –artificial traces from models Plot error and error-free distribution Calculate CC between reference and artificial distributions Optimal model for Ti => models that yield highest CC value network model artificial network metric trace trace analysis algorithm real network metric trace

DAPs for (Lexp,EFexp) > 0.1 DAP 1 Lden=0.2 Gilbert region CC values (Gilbert = 0.99) MMTA = 0.98 MMTA region CC values (MMTA =0.97) Gilbert = 0.96 DAP 2 Lden = 0.4 Gilbert region CC values (Gilbert = 0.99) MTA = 0.97 MTA region CC values (MTA = 0.98) MMTA region CC values (M3=0.97)

DAP for (Lexp,EFexp) > 0.1 DAP 3 Lden=0.7 MMTA region CC values (MMTA=0.98) MTA=0.97 MTA region CC values (MTA=0.99) MMTA=0.99

Conclusions The challenge –How to choose the most accurate, least cost model for each network type The solution –Key metric for modeling accuracy by error burst distributions –Domain analysis plots show optimal model for different networks Using domain analysis –Take your own traces for network X –Calculate the three parameter values –Use DAP to determine optimal model for your trace

Thank you Questions?