1. Reminders HW 3 Posted HW 1 Graded and Posted Grading appeal process

2. MGTSC 352 Lecture 6: Forecasting Wrap-up of Forecasting Holdout strategy Debugging Forecasting Models Monte Carlo Simulation Playing Roulette with Excel Bard Outside example

3. 95% Prediction Interval Technically correct formula; –Forecast + Bias + 2 x Std Error Heuristic for use in this class; –Forecast  2  SE

4. Steps in a Forecasting Project -1:Collect data 0:Plot the data (helps detect patterns) 1:Decide which models to use –level – SA, SMA, WMA, ES –level + trend – SLR, DES –level + trend + seas. – TES, SLR w SI,... 2:Use models 3:Compare and select (one or more) 4:Generate forecast and range (prediction interval) More on selection Pg. 39

5. Appropriate model... linear Nonlinear (ex. power) S-curve (ex. any CDF)

6. Calgary EMS Data Trend? Seasonality? Number of calls / month

7. Checking for (Yearly) Seasonality Number of calls / month

8. Weekly or Hourly Seasonality Avg. # of calls / hr., 2004

9. How to select a model? Look at performance measures –BIAS, MAD, MAPE, MSE Use holdout strategy Example: 4 years of data Use first 3 years to fit model(s) Forecast for Year 4 and check the fit(s) Select model(s) Refit model(s) adding Year 4 data If you have more than one good model... COMBINE FORECASTS Pg. 41

10. Example: Building Materials, Garden Equipment, and Supply Dealers

11. TES vs. SLR w SI (Both optimized to minimize SE) TESSLR w SI BIAS$130BIAS$5 MAD$628MAD$733 MAPE2.9%MAPE3.3% SE$843SE$1,016 Which method would you choose?

12. One possibility: Combining Forecasts TESSLR w SI weight+ (1 - weight) Minimize SE of the combined forecast to find the best weight

13. Holdout Strategy 1.Ignore part of the data (the “holdout data”) 2.Build models using the rest of the data 3.Optimize parameters 4.Forecast for the holdout data 5.Calculate perf. measures for holdout data 6.Choose model that performs best on holdout data 7.Refit parameters of best model, using all data

14. TES vs. SLR w/ SI …in holdout period holdout period

15. TES vs. SLR w SI … … in holdout period TESSLR w SI BIAS$1,266BIAS$2,956 MAD$1,514MAD$2,956 MAPE4.9%MAPE10.6% SE$1,824SE$3,356 Now which method would you choose?

16. Holdout Strategy Recap Performance during holdout period: a.k.a. “out of sample” performance In other words: how well does the method perform when forecasting data it hasn’t “seen” yet? Question: Why is SE during holdout period worse than SE during “training period”?

17. Do we have to implement these models from scratch? Forecasting software survey –http://lionhrtpub.com/orms/surveys/FSS/FSS.html General statistics program –Minitab, NCSS, SAS, Systat Dedicated forecast software –AutoBox, Forecast Pro (MGTSC 405)

18. Do Spreadsheet Models Have Errors? Field audits of real-world spreadsheets: 94% had errors http://panko.cba.hawaii.edu/ssr/Mypapers/whatknow.htm http://panko.cba.hawaii.edu/ssr/Mypapers/whatknow.htm What are the consequences of spreadsheet errors? –Incorrect financial statements –Bad publicity, loss of investor confidence –Lawsuits –Loss of election –See http://www.eusprig.org/stories.htm for morehttp://www.eusprig.org/stories.htm

19. Debugging – Finding Your Mistakes Before entering a formula: –Pause and predict the result After entering a formula: –Double-click to see where numbers are coming from Try simple test values: 0, 1 Graph your results ctrl+~ – use to look for breaks in patterns To Excel

20. Playing roulette with Excel To Excel …

21. Game 1 Spin the spinner once Payoff = (spinner outcome)  ($1 Million) Q1: What would you pay to play this game? Q2: Suppose the game were played 10,000 times. What do you think the payoff distribution will look like?

22. Game 2 Spin the spinner twice Payoff = ($1 Million) x (spinner outcome 1 + spinner outcome 2)/2 Q1: What would you pay to play this game? Q2: Suppose the game were played 10,000 times. What do you think the payoff distribution will look like?

23. Game 1 payoff distribution orneither ???

24. Game 2 payoff distribution orneither ???

25. Using Excel to get the right answer Simulate one spin: =RAND() Repeat 10,000 times Plot histogram –To Excel

26. Excel Details Using Data tables to replicate a simulation Enter replication numbers (1, …, n) in leftmost column Enter formulas for outputs in top row Highlight table Data  Table … –Column input cell: any empty cell Pg. 43

27. More Excel Details “Freezing” simulated values: –Copy the values –Paste special …  values Frequency distributions: (see also pg. 134) –Generate sample –Enter “bins” values –Highlight range where frequencies should be calculated –=FREQUENCY(sample, bins) –“Ctrl + shift + enter” instead of just “enter.”

28. Bard Outside The Bard Outside theatre group puts on plays by Shakespeare 20 times every summer in a 200-seat outdoor theatre. Data: –Attendance and weather (rain / no rain) for last five seasons (5 x 20 = 100 shows) –Revenue = $10 per customer –Cost = $1,600 per show Question: how much would profit increase if the number of seats were increased?

29. Data Analysis What’s the probability of rain? What is the mean and standard deviation of demand when it rains? How about when it doesn’t rain? How can we simulate demand? To Excel …

30. Simulating Profit per show Simulate weather Simulate demand Make sure 0 ≤ demand ≤ capacity Calculate revenue Subtract cost Replicate! Remember: freeze tables of simulation results

31. Final results