Copyright © 2010 Lumina Decision Systems, Inc. Monte Carlo Simulation Analytica User Group Modeling Uncertainty Series #3 13 May 2010 Lonnie Chrisman, Ph.D. Lumina Decision Systems

Copyright © 2010 Lumina Decision Systems, Inc. Course Syllabus (tentative) Over the coming weeks: What is uncertainty? Probability. Probability Distributions Monte Carlo Sampling (today) Measures of Risk and Utility Common parametric distributions Assessment of Uncertainty Risk analysis for portfolios (risk management) Hypothesis testing

Copyright © 2010 Lumina Decision Systems, Inc. Today’s Outline LogNormal exercise (review) Another way to represent uncertainty The representative sample Computing result uncertainty The Run index and Sample(..) Sample error Latin Hypercube Viewing uncertainty Evaluation modes

Copyright © 2010 Lumina Decision Systems, Inc. Modeling Exercise A mining company obtains rights to extract a gold deposit during a one-week window next year, before a construction project starts on the site. Extracting the deposit will cost $900K. The size of the deposit: LogNormal(Mean:1K,Stddev:300) oz. The price of gold next year: LogNormal(Mean:$1K, stddev:$500) What is the expected value of these mining rights? Compare to result ignoring uncertainty. Hint: The price of gold next year becomes known before the decision to proceed with extraction.

Copyright © 2010 Lumina Decision Systems, Inc. Another representation of Uncertainty: A Representative Sample 10 possible prices for gold next year (in $ per oz): $411/oz $548 $650 $746 $843 $949 $1,073 $1,230 $1,459 $1,945 This “sample” is a way to represent our uncertainty about the quantity. It captures the range of possibilities. ~$100/oz spacing ~$500/oz spacing

Copyright © 2010 Lumina Decision Systems, Inc. How uncertain is a computed result? Profit := f(Price) Compute each “scenario” separately. The result is a representative sample for Profit. This sample becomes our representation for the uncertainty in the result. This method works when computing any function! PriceProfit $411f(411) $548f(548) $650f(650) $746f(746) $843f(843) $949f(949) $1073f(1073) $1230f(1230) $1459f(1459) $1945f(1945) { The model’s result }

Copyright © 2010 Lumina Decision Systems, Inc. Mean(profit)=394K Multiple Uncertain Inputs bad EstimatedComputed Price ($/oz) Deposit sizeExtract?Profit $ $ $ $ $ $ K $ K $ K $ K $ M Each line here is a separate “scenario”. Oops: The computed results here are not representative. Why?

Copyright © 2010 Lumina Decision Systems, Inc. Multiple Uncertain Inputs bad EstimatedComputed Price ($/oz) Deposit sizeExtract?Profit $ K $ M $ $ K $ $ $ $ $ K $ K Each line here is a separate “scenario”. Shuffle each sample separately. (They are independent) Mean(profit)=232K

Copyright © 2010 Lumina Decision Systems, Inc. Analytica’s Internal representation of Uncertainty bad EstimatedComputed Price ($/oz) Deposit sizeExtract?Profit $ K $ M $ $ K $ $ $ $ $ K $ K Mean(profit)=231K Run index Analytica represents (and computes) uncertainty using samples indexed by Run.

Copyright © 2010 Lumina Decision Systems, Inc. Analytica Generates the Sample for you We usually encode uncertainty assessments using distribution functions (for convenience). Analytica generates sample from the distribution and uses these for computations. We can, however, supply the sample directly if we want (e.g., if we have measurements). Leak rate Definition: LogNormal(mean:10K,stddev:8K) Sample(Leak_rate): Array(Run,[3.5K,18.3K,12.1K,…])

Copyright © 2010 Lumina Decision Systems, Inc. Exercise Try this in Analytica: Mean( Normal(0,1) * Normal(0,1) ) Is the computed result correct? Why not? (use SampleSize=100) Theoretical answer = 0.0 Computed result: e.g., (your result will vary, due to randomness in sample generation)

Copyright © 2010 Lumina Decision Systems, Inc. Sample Error A sample is an approximate representation of the analytic distribution. Computations based on the this sample end up with some error as a result of the approximation. This error is called “Sample Error”. Sample error reduces with larger sample size.

Copyright © 2010 Lumina Decision Systems, Inc. Notice: Sample Error: Precision of computed mean You compute the mean for an uncertain result. You want to be 95% sure that: Guaranteed when: Where σ = SDeviation(y) is estimated first by using a small sample. Reference: Appendix A, Analytica Users Guide

Copyright © 2010 Lumina Decision Systems, Inc. Precision required for Mean(Normal(0,1) * Normal(0,1)) Computed mean should be within 0.05 of the correct mean. (Δ=0.05) How many samples do we need? StdDev  1.2 sampleSize > (4.8/0.05) 2 = 9216

Copyright © 2010 Lumina Decision Systems, Inc. Pure Monte Carlo Randomness (20 points sampled) Poor Coverage Clusters of over-coverage

Copyright © 2010 Lumina Decision Systems, Inc. Pure Monte Carlo Randomness (20 points sampled) Poor Coverage Clusters of over-coverage

Copyright © 2010 Lumina Decision Systems, Inc. Pure Monte Carlo Randomness (20 points sampled) Poor Coverage Clusters of over-coverage

Copyright © 2010 Lumina Decision Systems, Inc. Latin Hypercube Sampling Every 5% of area has one point. Vertical green lines every 10%. Two points always in each 10% region.

Copyright © 2010 Lumina Decision Systems, Inc. Latin Hypercube Sampling CDF of same sample Vertical green lines every 10%. Two points always in each 10% region.

Copyright © 2010 Lumina Decision Systems, Inc. Sample Error for Latin Hypercube Latin Hypercube often obtains smaller sampling error than Monte Carlo at the same sample size. For some smooth functions with a single uncertain scalar variable: (i.e., quadratically faster than Monte Carlo) For non-smooth functions, it can be worse than Monte Carlo in some cases, but these are rare. With many uncertain inputs, convergence rates are very similar to Monte Carlo.

Copyright © 2010 Lumina Decision Systems, Inc. Exercise Compute π by sampling. Use SampleSize=100 Compare precision for: Pure Monte Carlo Median Latin Hypercube Random Latin Hypercube Here’s how: x,y ~ Uniform(-1,1) Probability(x^2+y^2<1) is π /4 Area of circle = π Area of square = 4 x y

Copyright © 2010 Lumina Decision Systems, Inc. Uncertainty Views All uncertainty views are computed from the sample. Mean Value Statistics Bands PDF CDF Sample

Copyright © 2010 Lumina Decision Systems, Inc. Sample Statistics Given the 10-point sample: [2.0, 2.5, 2.9, 3.2, 3.5, 3.9, 4.3, 4.8, 5.5, 7.0] What is the median? ( )/2 = 3.7 What is the sample mean? ( )/10=3.96 What is the sample variance? ( ) 2 +( ) ( ) 2 ) / 9 = 2.26 What is the sample standard deviation? Sqrt(2.26) = 1.5

Copyright © 2010 Lumina Decision Systems, Inc. Fractiles (percentiles, Probability Bands) Given the 10-point sample: [2.0, 2.5, 2.9, 3.2, 3.5, 3.9, 4.3, 4.8, 5.5, 7.0] What is the 25% fractile? Answer: 2.9 What is the 60% fractile? Answer: 4.1 0% 10% 20% 30%40%50%60%70%80%90% 100%

Copyright © 2010 Lumina Decision Systems, Inc. Tricks for Smoothing Probability Density Plots Sample Size Samples per PDF step  1.6 * 2.5^logten(sampleSize) Equal P vs. Equal X Line style Histo vs. line Manual axis scaling (when extremes present)

Copyright © 2010 Lumina Decision Systems, Inc. Evaluation Modes Analytica has two evaluation modes: Mid mode Sample mode Mid result view uses Mid-mode Uncertainty functions return the Median. All other result views use Sample-mode Uncertainty functions return a sample.

Copyright © 2010 Lumina Decision Systems, Inc. Summary A Sample is a way of representing an arbitrary distribution of uncertainty. Enables uncertainty analysis for arbitrary computations (Monte Carlo). Analytica’s Run index associates scenarios between the samples of different variables. All result uncertainty views are derived from the computed sample.