IE 594 : Research Methodology – Discrete Event Simulation David S. Kim Spring 2009

Research Methodology - Simulation Simulation as a research tool Research in simulation Focus here is on simulation of discrete event dynamic systems

Simulation as a Research Tool What is the role of simulation in the research? 1. Used as the tool to understand general system dynamics and generate insights? 2. Comparison or validation of an approximation or heuristic? What is the role of simulation in the buffer allocation paper (Conway paper)?

Simulation as a Research Tool – Case 1 Why simulation? The system of interest has a performance function f that transforms controllable system parameters into system performance. The function f is unknown or computationally intractable.  Markov chain that approximate the throughput function have too many states  Analytical models do not exist. What is f in the buffer allocation paper (Conway et al.)? What are the input parameters?

Simulation as a Research Tool – Case 2 Why simulation? A heuristic or analytical approximation has been developed to model some system performance measure. The development of the approximation requires simplifying assumptions/approximations. The conjecture is that the analytical model is still a reasonable representation of the real system. Simulation is being used to support or refute this conjecture. What are the simplifying assumptions/ approximations used in the Nagarajan paper?

Simulation as a Research Tool What is the real system being simulated? Is there evidence or research precedent indicating that the simplified model being simulated characterizes real systems? Is this generally known or do references need to be cited? What’s being simulated in the buffer paper?

Simulation as a Research Tool Are the assumptions applied in the simulation clearly stated? Distributions used. Operational protocols, e.g., blocking, etc. Correlation? Can you simulate the same system? Steady State vs. Terminating Number of runs Length of runs  Some models take a long time to “settle down” Warm-up

Simulation as a Research Tool Verification & validation Mainly applies to studying a real system or a detailed representation How was this conducted?  Results compared to an existing system?  Comparisons made to existing analytical results?  Extreme cases tested?

Simulation as a Research Tool Experimental design How was the “parameter space” explored? Experimental design? Random systems? Worse case systems? Standard problem library? The importance of this depends on the way the simulation was used If simulating to understand a system and gain insight, these issues become more important.

Simulation as a Research Tool Output analysis Were proper statistical procedures applied to the output?  e.g., confidence intervals What is the variance around the average results?

Research in Simulation Research into the various aspects of simulation mechanisms or conducting simulation studies Generating random numbers Generating random variates Use of distributions in simulations Dealing with extremely large simulation models Optimization of systems using simulation etc., … See general topics for the WSC

Research in Simulation Simulation optimization The system of interest has a performance function f that transforms controllable system parameters into system performance. The function f is unknown or computationally intractable. Simulation is used to approximate f How can simulation be used to optimize f subject to constraints and costs?

Research in Simulation Examples of simulation optimization research Perturbation analysis Use of global optimization heuristics (e.g., genetic algorithms) Vergara paper.

Research in Simulation Simulation modeling methodology How can different systems be modeled? Distributed/parallel simulation Methods for verification/validation Short cuts/approximations Where is the Sharma paper ?

Conclusions Many of the items discussed today are things to look for when evaluating research and when conducting research. Often the importance of each item will depend on the circumstances, the existing body of research, and the objectives of the research.