Automating the Analysis of Simulation Output Data Katy Hoad Stewart Robinson, Ruth Davies, Mark Elder Funded by EPSRC and SIMUL8.

Slides:

Advertisements

Similar presentations

Statistical Process Control for Short-Runs

Advertisements

Introduction to parameter optimization

Automating Software Module Testing for FAA Certification Usha Santhanam The Boeing Company.

Modellierung großer Netze in der Logistik SFB 559 Initial Transient Period Detection Using Parallel Replications F. Bause, M. Eickhoff LS Informatik IV,

Automating the Analysis of Simulation Output Data Katy Hoad Stewart Robinson, Ruth Davies, Mark Elder Funded by EPSRC and SIMUL8.

Alternate Software Development Methodologies

Output analyses for single system

Dynamic Systems Development Method (DSDM)

Output Data Analysis. How to analyze simulation data? simulation –computer based statistical sampling experiment –estimates are just particular realizations.

Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded.

The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC 07 A 3 year, EPSRC funded project in collaboration with SIMUL8.

Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School Simulation Workshop - April 2008 The AutoSimOA.

Automated Analysis of Simulation Output Data and the AutoSimOA Project

Automating the Analysis of Simulation Output Data Katy Hoad, Stewart Robinson, Ruth Davies SSIG Meeting, 24th October 2007

Classification of Discrete Event Simulation Models and Output Data: Creating a Sufficient Model Set. Katy Hoad Stewart Robinson,

Lecture 9 Output Analysis for a Single Model. 2  Output analysis is the examination of data generated by a simulation.  Its purpose is to predict the.

The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.

Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.

© January 27, 2006 Dr. Lynn Fuller, Motorola Professor Simulation Page 1 Rochester Institute of Technology Microelectronic Engineering ROCHESTER INSTITUTE.

EEM332 Design of Experiments En. Mohd Nazri Mahmud

Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies ORGS Meeting, 4th October 2007

SIMULATION. Simulation Definition of Simulation Simulation Methodology Proposing a New Experiment Considerations When Using Computer Models Types of Simulations.

1 Simulation Modeling and Analysis Output Analysis.

191 Drawing Statistical Inference from Simulation Runs......the "fun" stuff!

Automating The Selection of a Simulation Warm-up Period Stewart Robinson, Katy Hoad, Ruth Davies Warwick Business School Cardiff University - October 2008.

Monté Carlo Simulation MGS 3100 – Chapter 9. Simulation Defined A computer-based model used to run experiments on a real system.  Typically done on a.

AutoSimOA : A Framework for Automated Analysis of Simulation Output Stewart Robinson Katy Hoad, Ruth Davies Funded by.

School of Education Archive Analysis: On EEF Trials Adetayo Kasim, ZhiMin Xiao, and Steve Higgins.

Analysis of Simulation Results Andy Wang CIS Computer Systems Performance Analysis.

Steady-State Statistical Analysis By Dr. Jason Merrick.

Verification & Validation

Macros, Navigation Form, PivotTables, and Pivot Charts Access – Lesson 6.

Measurement Uncertainty Information in

Chapter © 2015 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or.

26 June 2008 DG REGIO Evaluation Network Meeting Ex-post Evaluation of Cohesion Policy Programmes co-financed by the European Fund for Regional.

Online aggregation Joseph M. Hellerstein University of California, Berkley Peter J. Haas IBM Research Division Helen J. Wang University of California,

Evaluation of User Interface Design 4. Predictive Evaluation continued Different kinds of predictive evaluation: 1.Inspection methods 2.Usage simulations.

Delivering Integrated, Sustainable, Water Resources Solutions Monte Carlo Simulation Robert C. Patev North Atlantic Division – Regional Technical Specialist.

Guide to Using Minitab 14 For Basic Statistical Applications To Accompany Business Statistics: A Decision Making Approach, 8th Ed. Chapter 8: Estimating.

STEADY-STATE SYSTEM SIMULATION(2). REVIEW OF THE BASICS Initial Transient a.k.a. Warm-Up Period.

McGraw-Hill/Irwin © 2006 The McGraw-Hill Companies, Inc., All Rights Reserved. 1.

AP Statistics Section 10.2: Estimating a Population Mean.

Chapter 3 System Performance and Models Introduction A system is the part of the real world under study. Composed of a set of entities interacting.

1 OUTPUT ANALYSIS FOR SIMULATIONS. 2 Introduction Analysis of One System Terminating vs. Steady-State Simulations Analysis of Terminating Simulations.

Analysing Miss O’Grady. Analysing Analysing is the interpretation of the data. It involves examining the data and giving meaning to it. When data has.

McGraw-Hill/Irwin Copyright © 2009 by The McGraw-Hill Companies, Inc. All rights reserved.

Monte-Carlo based Expertise A powerful Tool for System Evaluation & Optimization  Introduction  Features  System Performance.

Output Analysis for Simulation

Item Parameter Estimation: Does WinBUGS Do Better Than BILOG-MG?

Importing record using DOIs Catherine Jones & Robert Darby eScience Centre, Science & Technology Facilities Council.

ANALYSIS PHASE OF BUSINESS SYSTEM DEVELOPMENT METHODOLOGY.

Evaluation of statistical methods for meta-analysis Julian Higgins School of Social and Community Medicine University of Bristol, UK 1 Cochrane Methods.

Modern Approaches of Systems Development By: Hanouf AL-Monawer Sara Mohammed.

TAUCHI PHILOSOPHY SUBMITTED BY: RAKESH KUMAR ME

Automating the Single Crystal X-Ray Diffraction Experiment – Mark Light – School of Chemistry - University of Southampton – ECM22 – Budapest 2004 Automating.

SECTION 6 DESIGN STUDY. What’s in this section: –Design Variables –Design Studies Overview –Specifying an Objective –Execution Display Settings –Output.

Computer Simulation Henry C. Co Technology and Operations Management,

Engineering (Richard D. Braatz and Umberto Ravaioli)

Sample size calculation

Guide to Using Minitab 14 For Basic Statistical Applications

National e-Infrastructure Vision

Software Process Models

Manufacturing system design (MSD)

Engineering Systems Design

Monte Carlo Simulation Managing uncertainty in complex environments.

(or why should we learn this stuff?)

The Quality Control Function

MECH 3550 : Simulation & Visualization

AP Statistics Chapter 12 Notes.

Presentation transcript:

Automating the Analysis of Simulation Output Data Katy Hoad Stewart Robinson, Ruth Davies, Mark Elder Funded by EPSRC and SIMUL8 Corporation

The Problem Prevalence of simulation software: ‘easy- to-develop’ models and use by non- experts. Simulation software generally have very limited facilities for directing/advising user how to run the model to get accurate estimates of performance. With a lack of the necessary skills and support, it is highly likely that simulation users are using their models poorly.

3 Main Decisions: How long a warm-up is needed? How many replications should be run? How long a run length is needed?

Enter Analyser Warm-up analyser Warm-up period specified Trial Calculator Recommend number of replications Run-length calculator Recommend run-length Replication s or one long run? Replications One long run EXIT Analyser AUTOMATIC SIMULATION OUTPUT ANALYSER

Warm-up Analyser MSER-5 most promising method for automation –Performs robustly and effectively for the majority of data sets tested. –Not model or data type specific. –No estimation of parameters needed. –Can function without user intervention. –Quick to run. –Fairly simple to understand.

MSER-5 warm-up method Truncation Point Test Statistic Batch Means MSER-5 test statistic Rejection zone Estimated warm-up period Estimated truncation point, Lsol Output data (batched means values)

Heuristic framework around MSER-5 Includes: Iterative procedure for procuring more data when required. ‘Failsafe’ mechanism - to deal with possibility of data not in steady state; insufficient data provided when highly auto-correlated. Graphical feedback to user.

Trial calculator

Precision≤ 5% Precision> 5% Precision ≤ 5% f(kLimit) Nsol 2 Nsol 2 + f(kLimit ) Nsol 1 95% confidence limits Cumulative mean, Confidence Interval Method with ‘look-ahead’

Run length calculator Batch Means Method. Want a robust automatable method that estimates the minimum run length needed to achieve a required precision in the output point estimator. Currently investigating literature and testing methods.

Enter Analyser Warm-up analyser Warm-up period specified Trial Calculator Recommend number of replications Run-length calculator Recommend run-length Replication s or one long run? Replications One long run EXIT Analyser AUTOMATIC SIMULATION OUTPUT ANALYSER IMPLEMENTED IN SIMUL8 BEING IMPLEMENTED IN SIMUL8

ACKNOWLEDGMENTS This work is part of the Automating Simulation Output Analysis (AutoSimOA) project ( that is funded by the UK Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in collaboration with SIMUL8 Corporation, who are also providing sponsorship for the project. Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School SW08