1. Session 6: A guide to choosing forecast models Demand Forecasting and Planning in Crisis 30-31 July, Shanghai Joseph Ogrodowczyk, Ph.D.

2. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 2 A guide to choosing forecasting models Session agenda  Judgment modeling: Using expert knowledge to provide forecasts  Mixed methods: Constructing forecasting frameworks from expert knowledge  Quantitative modeling: Using statistical techniques to provide forecasts  Criteria for combining or adjusting forecasts

3. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 3 A guide to choosing forecasting models Review  The last session explained some short run forecasting methods These techniques are used to quickly obtain forecasts  How do we produce more accurate forecasts for longer time horizons?  We need more sophisticated models What are the different types of forecasting models? What are the criteria for choosing which model to use in a particular instance?

4. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 4 A guide to choosing forecasting models Judgment: Experts are providing forecasts Mixed: Applying forecasting guidelines to statistically produced forecasts Quantitative: Using statistical techniques to generate forecasts (Armstrong and Green 2009)

5. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 5 A guide to choosing forecasting models Judgment model types: Experts (in forecasting or a product line) are producing the forecasts via …  Unaided judgment  Expert forecasting  Decomposition  Conjoint analysis  Intentions / Expectations  Role playing / Simulated interaction  Structured analogies

6. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 6 A guide to choosing forecasting models Judgment models  Unaided judgment: Forecasts made without the use of formal forecasting methods Conditions of use  Large changes are not expected  Forecasts are not used for policy analysis  Highly predictable/repetitive atmosphere Advantages:  Quick to produce forecasts  Inexpensive if only a few forecasters are needed  Accuracy can be improved when forecaster obtains rapid feedback

7. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 7 A guide to choosing forecasting models Judgment models  Expert forecasting: Experts are asked to provide forecasts Conditions of use can vary  Easy access to experts  Motivated and knowledgeable experts  Need for confidentiality  Low dispersal of knowledge  Limited time for forecast production

8. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 8 A guide to choosing forecasting models Judgment models  Expert forecasting Nominal group technique: A one-round survey for forecasts in which experts possess similar information Estimate-talk-estimate: A three-round survey where experts possess different information. Between forecast estimations, the participants are asked to have a discussion Delphi method: At least two survey rounds with results of the previous round summarized for participants Prediction markets: Incentive-based arrangements that use markets to aggregate, in the form of prices, information that is dispersed among participants.

9. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 9 A guide to choosing forecasting models Judgment models  Decomposition: Breaking down the estimation task into a set of components to produce a target forecast Method is to ask experts to forecast parts of the whole and then aggregate for the whole forecast Conditions of use can vary  Forecasting a highly complex system  Forecasting in an unfamiliar metric/market/customer  Higher confidence in component forecasts than in the target forecast Decomposition can be additive or multiplicative

10. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 10 A guide to choosing forecasting models Judgment models  Conjoint analysis: Survey method based on characteristics of a product Decomposition used pieces of the target forecast. Conjoint uses characteristics of the forecasted item Sometimes used to construct a forecast manually but often used as the basis for a regression type analysis (mixed method) Conditions of use:  Large changes in demand expected  To be used for policy analysis  Survey users of the forecasted item instead of experts with market / forecasting knowledge

11. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 11 A guide to choosing forecasting models Judgment models  Conjoint analysis Popular in marketing  Consumers are asked to rank / assign a value trade-offs between multi-dimensional alternatives  Automobiles, soft drinks, computers, checking accounts, hotel accommodations, etc. Results can be plausible  Major drawback is the potential hypothetical nature of the survey  Consumers are not deciding between actual goods and services

12. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 12 A guide to choosing forecasting models Judgment models  Intentions / expectations: Survey method based on intended future behavior Conjoint asked consumers for preferences on specific item characteristics. Intentions asked consumers for anticipated future behavior Conditions of use  Large changes in demand are expected  Relatively little conflicts among forecasters  Forecasts are not used for policy analysis (a need to choose between different courses of action) Disadvantages  Research shows that intentions are biased as measures for prediction  Research is inconclusive about how best to measure intentions

13. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 13 A guide to choosing forecasting models Judgment models  Role playing / Simulated interaction: Customers are asked to act out prospective interactions in a realistic manner  Conditions of use Important as a tool in forecasting within conflicts (threats of striking workers, jury reactions, assessing outcomes from different strategies) Small number of parties interacting Need to predict in situations involving large changes  Advantages Decisions are often difficult to forecast if they are the result of a series of actions Role playing can be used to simulate the actions and reactions between the parties

14. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 14 A guide to choosing forecasting models Judgment models  Structured analogies: Surveying experts to compare analogous situations and using the outcomes of the analogies as the forecast for target  Conditions of use Large changes are expected Difference among forecasters Similar cases exist  Methodology Describe the target situation Select experts Identify and describe analogies  Ask the experts to describe as many analogies as they can without considering the extent of the similarity to the target situation

15. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 15 A guide to choosing forecasting models Judgment models  Structured analogies (Green and Armstrong 2007)  Methodology Rate similarity  Ask the experts to list similarities and differences between their analogies and the target situation, and then to rate the similarity of each analogy to the target  Ask them to match their analogies’ outcomes with target outcomes Derive forecasts  Set up the rule system (criteria) for choosing the analogy to use for the target forecast before interviewing experts  Many rules are reasonable  For example, one could select the analogy that the expert rated as most similar to the target and adopt the outcome implied by that analogy as the forecast

16. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 16 A guide to choosing forecasting models

17. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 17 A guide to choosing forecasting models Mixed modeling : Forecasting using statistical models created from expert rules  The development of statistical techniques is guided by expert’s rules Rules: Criteria, inputs, and other variables that experts use in producing forecasts using judgment methods Mixed model types  Quantitative analogies  Expert systems  Rule-based forecasting  Bootstrapping

18. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 18 A guide to choosing forecasting models Mixed model types  Quantitative analogies: Data from analogous situations are used as input to derive the target forecast Sample rule: When data are not available, use data from a similar situation  Conditions of use Not a good knowledge of the relationships in the data Cross sectional data (Data across multiple units for the same time period) Not used for policy analysis  Advantages Statistical forecasting methodologies can be used on input data Allows for increased observations for a data-poor series such as new products

19. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 19 A guide to choosing forecasting models Mixed model types  Expert systems: Statistical models are designed to represent the rules used by experts in the forecasting process Rules are based on knowledge of target area Models are based directly on the rules Information on rules can be found in research papers, surveys, and interviews Expert systems should be easy to use, incorporate the best available knowledge, and reveal the reasoning behind the recommendations they make

20. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 20 A guide to choosing forecasting models Mixed model types  Rule-based: A system to develop and apply weights for combining extrapolations Time series extrapolation: Univariate time series forecasting methods Rules result in each extrapolation method being assigned a weight based on trends, seasonality, and historical data The compilation forecast is the sum of the weighted extrapolation methods Knowledge for rules can be obtained through expert judgment, empirical research, and theory Guidelines for rules  Give separate consideration to level and trend  Use different models for short- and long-run forecasts  Damp the trend as the forecast horizon increases

21. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 21 A guide to choosing forecasting models Mixed model types  Bootstrapping: Translating experts’ rules into a quantitative model Difference from expert system  Based on inference about experts rules  Requires repeated sampling  Specific model design used in the translating (regression) Model is produced by regressing the forecasts produced upon the information that the expert used Guidelines for bootstrapping  Use experts who differ  Use simple analysis to represent behavior  Note: Quantifying the variables used by the experts can greatly affect the validity of the model

22. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 22 A guide to choosing forecasting models

23. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 23 A guide to choosing forecasting models Quantitative modeling: Using statistical techniques to provide forecasts  Forecasts are produced by statistical models modeling the behavior generating the data series (historical data or external variables)  Experts suggest appropriate variables  Stepwise regression, state-space models, and Bayesian techniques are statistical tools that can be applied either to univariate or multivariate models Quantitative model types  Extrapolation/neural networks  Statistical regression  Segmentation  Index

24. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 24 A guide to choosing forecasting models Quantitative model types  Extrapolation: Statistical models using only historical information to produce forecasts Univariate times series, Holt-Winters (exponential smoothing), Box-Jenkins and ARIMA (ARMA), autoregressive, linear trend (trend using only time), simple (single) regression From session 4, Naïve model and moving average models (without and with confidence intervals) Assumes that all the necessary information for forecasting is contained in the historical data Can also be used for cross-sectional data  To estimate the probability of a new hire lasting more than a year, analyze the percent of the previous 50 applicants lasting more than a year

25. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 25 A guide to choosing forecasting models Quantitative model types  Neural net: Models using complex, interdependent, variable relationships to produce forecasts Inspired by the behavior of biological neurons Often a “black box” for understanding the relationships  Conditions of use Best for quarterly or monthly data Discontinuous series Several-period lag between forecasting and forecasted periods  Advantages Does not need to fully understand the relationship of explanatory variables Estimates nonlinear functions well

26. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 26 A guide to choosing forecasting models Quantitative model types  Statistical regression (Econometrics): Using statistical methods to estimate the relationships of variables based on theory, prior studies, and expert knowledge (Augmented) Dickey-Fuller, vector autoregressive (VAR), error correction models (ECM), multiple regression Parameter estimates (elasticities, measures of influence on dependent variable by independent variable) can be obtained through using least squares or maximum likelihood Models use theory and expert knowledge to select the explanatory variables Dependent variable is the forecasted item and independent (explanatory, causal) variables are those which “explain” the behavior of the dependent variable

27. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 27 A guide to choosing forecasting models Quantitative model types  Segmentation: Forecasting a heterogeneous whole through forecasting of parts of the whole separately When the dependent variable responds in different ways to the independent variables  Forecasts for the parts will be created from separate econometric models because of the difference in causal effects Airline tickets: Business class and recreational coach class customers respond differently to price changes  Better to forecast each type of passenger and then aggregate Similar, but not the same, as bottom-up forecasting  Bottom-up down not necessarily need multiple model types

28. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 28 A guide to choosing forecasting models Quantitative model types  Index: Forecasting the value of the dependent variable by adding values of the independent variables Improper linear models. Unit-weight is a special case when variables are weighted evenly Explanatory variables can be subjective and assigned a 1 or 0 depending on if they are present or absent, respectively Explanatory variables can be quantitative data that have been normalized (units mathematically removed) Weights can be chosen by experts Values of the dependent variable can be used to forecast the probability of an event Example: Factors contributing to drug use in adolescents  Grades, parent relationship, self esteem, etc. (Bry et al. 1982)

29. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 29 A guide to choosing forecasting models

30. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 30 A guide to choosing forecasting models Criteria for combining or adjusting forecasts  Two main questions on combining Are several methods producing useful forecasts? If so, how can they be combined?  Two main questions on adjusting Is there a need to adjust the forecast because of omitted data? If so, how should the forecasts be adjusted?  At what level in the hierarchy, over what time horizon, and by how much (percents or quantities)?

31. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 31 A guide to choosing forecasting models Criteria for combining or adjusting forecasts  Best practices for forecasting include a business process for answering the questions on combining and adjusting. Tool is the demand management of the S&OP process  Tips for combining forecasts Use different data for different models Use equal weight unless there is strong evidence Use expert knowledge to vary the weights  Collect historical data on weight accuracy

32. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 32 A guide to choosing forecasting models Criteria for combining or adjusting forecasts  Tips for adjusting forecasts Adjusting forecasts can be necessary when…  Recent events are not fully reflected in the data  Experts possess reliable knowledge about future events  Key variables were omitted from the models To gain consensus for adjustment level, time horizon, and degree…  Construct scenarios with representative events  Ask experts to provide explanations of outcomes  Be careful to avoid “boomerang” effect

33. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 33 A guide to choosing forecasting models

34. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 34 A guide to choosing forecasting models

35. Session 6 Joseph Ogrodowczyk, Ph.D. Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 35 A guide to choosing forecasting models References  Armstrong, J. Scott and Kesten Green. 2009. Selection Tree for Forecasting Methods. Forecasting Principles (April) http://www.forecastingprinciples.com (accessed May 2009).  Bry, Brenna H., P. McKeon, and R.J. Pandina. 1982. Extent of drug use as a function of a number of risk factors. Journal of Abnormal Psychology 9: 273-279., in Armstrong J. Scott, ed. 2001. Principles of Forecasting: A handbook for researchers and practitioners. Norwell, Mass.: Kluwer Academic Publishers.  Green, K.C. and J.S. Armstrong. 2007. Structured analogies for forecasting. International Journal of Forecasting 23: 365-376., in Armstrong J. Scott, ed. 2001. Principles of Forecasting: A handbook for researchers and practitioners. Norwell, Mass.: Kluwer Academic Publishers.