1 Introduction to Simulation Chapters 24

Overview Simulation: Key Questions Introduction to Simulation Common Mistakes in Simulation Other Causes of Simulation Analysis Failure Checklist for Simulations Terminology Types of Models

Simulation: Key Questions What are the common mistakes in simulation and why most simulations fail? What language should be used for developing a simulation model? What are different types of simulations? How to schedule events in a simulation? How to verify and validate a model? How to determine that the simulation has reached a steady state? How long to run a simulation?

Simulation: Key Questions (Cont) How to generate uniform random numbers? How to verify that a given random number generator is good? How to select seeds for random number generators? How to generate random variables with a given distribution? What distributions should be used and when?

Introduction 5 System to be characterized may not be available During design or procurement stage Still want to predict performance Or, may have system but want to evaluate wide-range of workloads  Simulation The best advice to those about to embark on a very large simulation is often the same as Punch’s famous advice to those about to marry: ‘Don’t!’ – Bratley, Fox and Schrage (1986)

Simulation Failure simulation models often fail; that is, they produce no useful results or misleading results. This is either because the model developers are proficient in software development but lack statistical background or because they are proficient in statistical techniques but are not aware of good software development techniques

Common Mistakes in Simulation (1 of 4) 7 1. Inappropriate level of detail Level of detail often potentially unlimited But more detail requires more time to develop And often to run! Can introduce more bugs, making more inaccurate not less! Often, more detailed viewed as “better” but may not be the case More detail requires more knowledge of input parameters Getting input parameters wrong may lead to more inaccuracy (Ex: disk service times exponential vs. simulating sector and arm movement) Start with less detail, study sensitivities and introduce detail in high impact areas

Common Mistakes in Simulation (2 of 4) 8 2. Improper language Choice of language can have significant impact on time to develop Special-purpose languages can make implementation, verification and analysis easier C++Sim, JavaSim 3. Unverified models Simulations generally large computer programs Unless special steps taken, bugs or errors

Common Mistakes in Simulation (3 of 4) 9 4. Invalid models No errors, but does not represent real system Need to validate models by analytic, measurement or intuition 5. Improperly handled initial conditions Often, initial trajectory not representative of steady state Including can lead to inaccurate results Typically want to discard, but need method to do so effectively

Common Mistakes in Simulation (4 of 4) Too short simulation runs Attempt to save time Makes even more dependent upon initial conditions Correct length depends upon the accuracy desired (confidence intervals) 7. Poor random number generators “Home grown” are often not random enough Best to use well-known one 8. Improper Selection of Seeds:

More Causes of Failure 11 Large software Quotations above apply to software development projects, including simulations If large simulation efforts not managed properly, can fail 1. Inadequate time estimate Need time for validation and verification Time needed can often grow as more details added Any given program, when running, is obsolete. If a program is useful, it will have to be changed. Program complexity grows until it exceeds the capacity of the programmer who must maintain it. - Datamation 1968

More Causes of Failure No achievable goal Common example is “model X” But there are many levels of detail for X Goals: Specific, Measurable, Achievable, Repeatable, Through (SMART) The goals should be clearly written down and agreed to by the analysts and the end users of the results before beginning the model development Project without goals continues indefinitely

More Causes of Failure 3. Incomplete mix of essential skills A simulation project requires at least the following four areas of skills: (a) Project Leadership: The ability to motivate, lead, and manage the members of the simulation team. (b) Modeling and Statistics: The ability to identify the key characteristics of the system and model them at the required level of detail. (c) Programming: The ability to write a readable and verifiable computer program that implements the model correctly. (d) Knowledge of the Modeled System: The ability to understand the system, explain it to the modeling team, and interpret the modeling results in terms of their impact on the system

Other Causes of Simulation Analysis Failure 4. Inadequate Level of User Participation: You need to have meeting with the user to make sure the system is not changing. 5. Obsolete or Nonexistent Documentation: Do the documentation during the simulation 6. Inability to Manage the Development of a Large Complex Computer: Program Need software engineering tools to manage it 7. Mysterious Results: Most mysterious results are due to bugs in the simulation program, invalid model assumptions, or lack of understanding of the real system.

Simulation Checklist (1 of 2) 15

Simulation Checklist (2 of 2) 16

Terminology (1 of 7) 17 Introduce terms using an example of simulating CPU scheduling Study various scheduling techniques given job characteristics, ignoring disks, display… State variables Variables whose values define current state of system Saving can allow simulation to be stopped and restarted later by restoring all state variables Ex: In the CPU scheduling simulation, the state variable is the length of the job queue. Ex: in M/M/1 simulation, we will write down the queue length

Terminology (2 of 7) 18 Event A change in system state is called event Ex: In the CPU scheduling simulation, Three events: arrival of job, beginning of new execution, departure of job Continuous-time and discrete-time models If state defined at all times  continuous If state defined only at instants  discrete Ex: class that meets M-F 2-3 is discrete since not defined other times Jobs Time Students Time

Terminology (3 of 7) 19 Continuous-state and discrete-state models If uncountably infinite  continuous Ex: time spent by students on hw If countable  discrete Ex: jobs in CPU queue Note, continuous time does not necessarily imply continuous state and vice-versa All combinations possible Jobs Time Term

Types of Models (Cont)

Terminology (4 of 7) 21 Deterministic and probabilistic models If output predicted with certainty  deterministic If output different for different repetitions  probabilistic Ex: Simulation is usually probabilistic if you change the seed. It is better to repeat it. Otherwise it is deterministic Output Input Output Input (Deterministic)(Probabilistic) (vertical lines)

Terminology (5 of 7) 22 Static and dynamic models If Time is not a variable  static If state changes with time  dynamic Ex: CPU scheduler is dynamic, while matter-to-energy model E=mc 2 is static Linear and nonlinear models If Output is linear combination of input  linear Otherwise  nonlinear Output Input (Linear) Output Input (Non-Linear)

Terminology (6 of 7) 23 Open and closed models Input is external and independent  open Closed model has no external input Ex: if same jobs leave and re-enter queue then closed, while if new jobs enter system then open cpu open cpu closed

Terminology (7 of 7) 24 Stable and unstable Model output settles down and reaches steady state  stable Model output always changes  unstable How about Sinusoidal Model? Output Time (Unstable) Output Time (Stable)

Computer System Models Continuous time Discrete state Probabilistic Dynamic Nonlinear Open or closed Stable or unstable

HW

Selecting a Language for Simulation

Simulation Languages

General Purpose Language Analyst's familiarity Easy availability Quick startup: You know the programming language Time for routines for event handling, random number generation Other Issues: Efficiency, flexibility, and portability Recommendation: Learn at least one simulation language.

Extensions of a General Purpose Language

Simulation Packages Examples: Opnet, NS2 Input dialog Library of data structures, routines, and algorithms Big time savings Problem: Inflexible

Types of Simulation Languages

Discrete Event Simulation Example NS - (1 of 4) 33 NS-2, network simulator Government funded initially, Open source Wildly popular for IP network simulations

Discrete Event Simulation Example NS - (2 of 4) 34 (Event scheduler is core of simulator)

Discrete Event Simulation Example NS - (3 of 4) 35 # Open the NAM trace file set nf [open out.nam w] $ns namtrace-all $nf # Define a 'finish' procedure proc finish {} { global ns nf $ns flush-trace # Close the trace file close $nf #Execute NAM on file exec nam out.nam & exit 0 } # Setup a FTP set ftp [new Application/FTP] $ftp attach-agent $tcp $ftp set type_ FTP # Initial schedule events $ns at 0.1 "$cbr start" $ns at 1.0 "$ftp start" $ns at 4.0 "$ftp stop" $ns at 4.5 "$cbr stop" # Finish after 5 sec (sim time) $ns at 5.0 "finish" # Run the simulation $ns run

Discrete Event Simulation Example NS - (4 of 4) 36 Output in text file, can be processed with Unix command line tools (Objects and script can have custom output, too)