Tracking Signal A Measure of Forecast Accuracy Tyler Hedin Brigham Young University Business Management 361, Section1 Dr. Tom Foster December 14, 2006 Prepared by: Tyler Hedin

Agenda Tracking Signal Defined Application, Advantages & Disadvantages Tracking Signal and Forecasting Application, Advantages & Disadvantages How it works Step by step formula Company XYZ Example Exercise Summary Readings list Useful websites Appendix A The objective of this presentation is to provide a basic explanation of a tracking signal as an accuracy measure for forecasting and describe its application through a realistic example and interactive exercise. Prior knowledge of forecasting methods and forecasting in general is assumed.

What is Tracking Signal? A measure that indicates whether the forecast average is keeping pace with any genuine upward or downward changes in demand* *definition taken from the glossary of Operations Management for Competitive Advantage, see Readings List for reference A tracking signal is a tool used to measure how well an individual forecasting method is performing.

Tracking Signal and Forecasting Continuous control indicator Monitor effectiveness of forecasting method Provide control limits A tracking signal is used as a continuous indicator of the forecasting method in order to determine if the established method needs modification. Its objective is to enhance the effectiveness of the forecasting effort by providing control limits. These limits are set by the organization and will be discussed later. A tracking signal that goes outside of established limits is an indicator that the forecasting method being used should be modified. Graphic obtained from www.claybennett.com

Application Evaluates forecasting method Indicator of change in demand patterns Used in conjunction with anything dependent on future demand Sales Inventory Because a tracking signal requires that actual data be compared with previously made forecasts, it is not normally used in deciding between forecasting methods, but rather in evaluating forecasting methods that have already been established. By nature, a forecast is based on historic patterns and a tracking signal is a common method for determining whether or not a pattern in demand has changed. A tracking signal can monitor any forecasts that have been made by an organization regarding sales, inventory, or anything pertaining to the organization’s future demand.

Advantages Unbiased Versatile Can be used with any type of forecasting method (time series, regression line, etc.) Some measures of forecast accuracy, such as MAD, measure the average size of the forecast error and do not take into account whether the forecast is too high or too low. An important component of a tracking signal is a continuous sum of the differences between forecasted and actual values, so as to measure whether a forecast is consistently too high or too low. If a forecast is consistently producing values that are too high or too low, a tracking signal will designate the forecasting method to be out of control, similar to the function of quality control charts. A tracking signal is also very versatile in that it can be used with a variety of forecasting methods. The only required numbers are an actual demand value and its corresponding forecasted value.

Disadvantages Could wrongfully flag perfect forecasts Unlikely Small differences in the same direction could cause signal to go outside of control limits The nature of a tracking signal assumes the existence of error. Though unlikely, a tracking signal could flag a forecasting method that is producing values that are very close to actual demand levels. As will be shown later, a tracking signal is calculated by dividing the most recent sum of errors by the most recent estimate of the mean absolute deviation (MAD). If forecasts are close to actual values, the mean absolute deviation would approach zero while the sum of the errors remained unchanged, launching the tracking signal towards infinity. Comparable to quality control charts, small deviations in the same direction over time could falsely indicate that the forecasting method is out of control. A closer look at forecast values and actual data following a tracking signal outside of designated limits would rectify this problem.

How it Works – Forecast Error Difference between actual demand and forecast The first step in calculating a tracking signal is to find the forecast error. The forecast error is found by simply subtracting the forecasted value from the actual demand value. Forecasts that are below the actual demand values will have a positive forecast error and forecasts that are above the actual demand values will have a negative forecast error. Week Actual Demand Forecasted Demand Forecast Error 1 21 19 2 25 22 3 24 -2

How it Works – Absolute Values Express the forecast errors as absolute values Week Actual Demand Forecasted Demand Forecast Error Absolute Value 1 21 19 2 25 22 3 24 -2 After finding the forecast errors, express these errors in there absolute values. These values will be used in a later step to calculate the mean absolute deviation (MAD).

How it Works – Running Sum Keep a continuous running sum of the forecast errors Do not add absolute values A running sum of the forecast errors is kept with each forecasted value. This running sum is cumulative and is found by summing the value of the forecast errors from the first period to the most recent period. You should make sure to not sum the absolute values found in the previous step. Week Actual Demand Forecasted Demand Forecast Error Absolute Value Running Sum 1 21 19 2 25 22 3 5 24 -2

How it Works - MAD Divide the summed absolute values by the number of periods to calculate MAD. Week Actual Demand Forecasted Demand Forecast Error Absolute Value Running Sum MAD 1 21 19 2 2.00 25 22 3 5 2.50 24 -2 2.33 The next step is to calculate the mean absolute deviation. Calculating MAD is done by dividing the sum of the absolute values found in the second step by the number of periods in the data set. This calculation must be done on a period by period basis in order for a tracking signal to be created. For each row, sum the absolute values for each period before it and divide that number by the period indicated by that row. For example, to calculate the MAD for row number three in the table above, one would need to sum the absolute values from the third row and each row that came before it (2+3+2) and divide that number by the total number of periods to that point (3). The MAD is a measure of the average error size for the forecasts and indicates the average deviation from actual demand. Because the MAD is calculated with absolute values, it does not take into account the direction of those deviations (i.e. whether they are above or below actual demand).

The Equation Tracking signal is mathematically defined as the sum of the forecast errors divided by the mean absolute deviation Tracking signal =  (Dt – Ft) MAD Once the MAD is calculated for each period, you are able to find the tracking signal. A tracking signal is found by dividing the most recent sum of forecast errors by the most recent estimate of MAD.

How it Works – Tracking Signal Divide the running sum of forecast errors by the corresponding MAD value As an example, the tracking signal for week 3 in the table above is found by dividing the running sum of errors as it stands in week 3 (3) by the estimate of MAD in week 3 (2.33). This value comes out to be 1.29. Week Actual Demand Forecasted Demand Forecast Error Absolute Value Running Sum MAD Tracking Signal 1 21 19 2 2.00 1.00 25 22 3 5 2.50 24 -2 2.33 1.29

What Do These Values Mean? Ratio of cumulative error to average deviation 0.8 σ ~ 1.25 MAD Limits are usually between 2 to 5 standard deviations Because a tracking signal is a ratio of the cumulative error produced by individual forecasts to the average deviation of those forecasts from their actual values, it typically “tracks” forecasts to see if they stay within a similar pattern to the corresponding demand. Companies often use the relationship between MAD and the standard deviation to establish limits for the tracking signal. Because 1 standard deviation is approximately equivalent to 1.25 MAD, a common boundary of 3 standard deviations is used (or 3.75 MAD). With the error being measured in MAD units, one can statistically determine the probability that the forecasts errors are attributable to random variation. Typically, a tracking signal that stays between 3 standard deviations is considered enough room for error. If the tracking signal is outside three standard deviations, the probability of random variation is low. For the purposes of this tutorial, limits for the tracking signal will be set at -4 and 4.

Example 1 Company XYZ has implemented a linear regression method to forecast sales. Actual sales for the months of January 2005 through January 2006 are given in Table 1 along with their corresponding forecasts. This hypothetical example will illustrate a tracking signal using the step by step method described earlier in the tutorial.

Table 1 SALES (in thousands) MONTH PERIOD DEMAND FORECAST January-05 1   SALES (in thousands) MONTH PERIOD DEMAND FORECAST January-05 1 $37 37.35 February-05 2 $40 38.97 March-05 3 $41 40.60 April-05 4 42.23 May-05 5 $45 43.85 June-05 6 $50 45.48 July-05 7 $43 47.10 August-05 8 $47 48.73 September-05 9 $56 50.36 October-05 10 $52 51.98 November-05 11 $55 53.61 December-05 12 $54 55.23 January-06 13 56.86 The forecasts in Table 1 were found using a linear regression line with the equation y=35.72+1.62x. The units for the sales values are in thousands of dollars.

Example 1 Company XYZ would like to employ a tracking signal to measure the performance of its forecasting method. This example assumes that a tracking signal was not put in place by Company XYZ while actual data was being collected. A tracking signal is being added later in order to better illustrate how a tracking signal is calculated.

Table 2 TRACKING SIGNAL -1.00 0.99 1.82 -2.37 -1.84 0.72 -1.08 -1.86 0.50 0.56 1.18 0.68 -0.16 MONTH PERIOD DEMAND FORECAST ERROR ABS DVN RUNNING SUM MAD TRACKING SIGNAL January-05 1 37 37.35 -0.35 0.35 -1.00 February-05 2 40 38.97 1.03 0.68 0.69 0.99 March-05 3 41 40.60 0.40 1.08 0.59 1.82 April-05 4 42.23 -5.23 5.23 -4.15 1.75 -2.37 May-05 5 45 43.85 1.15 -3.00 1.63 -1.84 June-05 6 50 45.48 4.52 1.52 2.11 0.72 July-05 7 43 47.10 -4.10 4.10 -2.58 2.40 -1.08 August-05 8 47 48.73 -1.73 1.73 -4.31 2.31 -1.86 September-05 9 56 50.36 5.64 1.33 2.68 0.50 October-05 10 52 51.98 0.02 1.35 2.42 0.56 November-05 11 55 53.61 1.39 2.74 2.32 1.18 December-05 12 54 55.23 -1.23 1.23 1.51 2.23 January-06 13 56.86 1.86 2.20 -0.16 All of the computations that were necessary in finding the values in Table 2 were performed in Microsoft Excel. As you can see, the tracking signal for this example stays well within the limits of -4 and 4 and should indicate that the forecasting method being used by Company XYZ is following demand patterns closely enough for the time being.

Exercise Your employer, Jones & Associates, has been using a linear regression method to forecast sales for 2006. After nine months have passed and actual sales data have been collected, your boss asks you to develop a tracking signal to measure the accuracy of the forecasts. The data for actual sales and forecasted sales is in Table 3. This exercise is meant to be performed by the reader. If feasible, the reader should perform necessary calculations in a Microsoft Excel worksheet. The solution to the exercise is given in Appendix A at the end of the tutorial.

Table 3 SALES MONTH PERIOD DEMAND FORECAST January-06 1 $3,769 3664.18   SALES MONTH PERIOD DEMAND FORECAST January-06 1 $3,769 3664.18 February-06 2 $3,912 3953.92 March-06 3 $4,212 4243.65 April-06 4 $4,861 4533.39 May-06 5 $4,672 4823.13 June-06 6 $4,937 5112.87 July-06 7 $5,346 5402.61 August-06 8 $5,783 5692.35 September-06 9 $6,021 5982.08 The data in Table 3 is meant to be used to establish a tracking signal following the steps outlined earlier in the tutorial.

Summary A tracking signal statistically determines if a forecasting method is out-of-control. As long as tracking signal stays within 3 standard deviations, probability of forecast error caused by random variation is high Used by companies to track changes in demand patterns Calculated by dividing the most recent sum of forecast errors by the most recent estimate of MAD A tracking signal outside of established limits indicates that a forecasting method should be modified. Compatible with any forecasting method

Readings List Chase, R. B. et al. (2004). Operations Management for Competitive Advantage 10th edition. McGraw-Hill Higher Education. Duncan, Robert M. (1992). Quality Forecasting Drives Quality Inventory at GE. Industrial Engineer, January edition. Hanke, J.E. & Wichern, D. W. (2004). Business Forecasting. Prentice Hall. Lawrence, F. B. (1999). Closing the logistics loop: A tutorial. Production & Inventory Management Journal, 40(1).

Useful Websites http://www.bestforecastingsoftware.com http://www.IdeaWins.com http://www.lehigh.edu/~rhs2/IBE098/forecating.ppt http://is.ba.ttu.edu/faculty/ch11.ppt http://www.microsoft.com/dynamics/intro/default.mspx Information on forecasting and tracking signal can be found in most textbooks dealing with operations management or supply chain management. Also, many websites offer forecasting software, such as www.IdeaWins.com and www.microsoft.com/dynamics.

Appendix A – Solution to Exercise MONTH PERIOD DEMAND FORECAST ERROR ABS DVN RUNNING SUM MAD TRACKING SIGNAL January-05 1 37 37.35 -0.35 0.35 -1.00 February-05 2 40 38.97 1.03 0.68 0.69 0.99 March-05 3 41 40.60 0.40 1.08 0.59 1.82 April-05 4 42.23 -5.23 5.23 -4.15 1.75 -2.37 May-05 5 45 43.85 1.15 -3.00 1.63 -1.84 June-05 6 50 45.48 4.52 1.52 2.11 0.72 July-05 7 43 47.10 -4.10 4.10 -2.58 2.40 -1.08 August-05 8 47 48.73 -1.73 1.73 -4.31 2.31 -1.86 September-05 9 56 50.36 5.64 1.33 2.68 0.50 October-05 10 52 51.98 0.02 1.35 2.42 0.56 November-05 11 55 53.61 1.39 2.74 2.32 1.18 December-05 12 54 55.23 -1.23 1.23 1.51 2.23 January-06 13 56.86 1.86 2.20 -0.16   The tracking signal is well within the limits of -4 and 4, signifying that the linear regression forecast model is correctly predicting Jones & Associates’ pattern of sales demand, within reason.