1. INTERPRETING AND COMMUNICATING CHECKOFF PROGRAM EVALUATION RESULTS Dr. Gary W. Williams Professor, Agricultural Economics Director, Texas Agribusiness Market Research Center (TAMRC) Dr. John P. Nichols Professor, Agricultural Economics Interim Head, Department of Agricultural Economics Texas A&M University College Station, Texas “Quantitative Evaluation of Advertising and Promotion Programs” Commodity Promotion Evaluation Workshop for Researchers and Managers 2006 Spring NEC-63 Meeting Monterey, California March 9, 2006

2. The Research Isn’t Done … … Until the Results Are Communicated!

3. Why Do Checkoff Groups Want Their Programs Evaluated? To demonstrate their effectiveness in expanding markets and enhancing profit. To assist in the design and adjustment of the program’s long-run strategic plan To provide the necessary economic ammunition to respond to legal challenges Simply to satisfy the legislative requirements To serve the information needs of contributors, industry, and other stakeholders

4. What Do Evaluation Studies Typically Provide? Measures of Program “Effectiveness” Retail Sales and Price Impacts Benefits to Producers and Other Program Contributors Other Measure of Effectiveness: - Market Share Impacts - Export Competitiveness Impacts - Production and Industry Growth Impacts

5. What Do Evaluation Studies Typically Provide? Aggregate Measures of Effectiveness Benefit-Cost Ratio (BCR): Marginal Rate of Return (MRR): Internal Rate of Return (IRR): The total benefits The total benefits of the program divided by the total costs of the program divided by the total costs of promotion. of promotion. The percent The percent change in program benefits from a 1% change in advertising expenditures. The return The return that a firm earns if it expands or invests in itself rather than investing that money elsewhere.

6. Communication Dilemma Results of Checkoff Program Evaluations Are Often: Not Well Understood Considered “Unbelievable”

7. … instead of the additional sales from each additional dollar in advertising … instead of the additional sales from each additional dollar in advertising … instead of how … instead of how responsive demand is to advertising responsive demand is to advertising Economic Jargon Often Gets in the Way “Advertising Elasticity” RESULTS NOT UNDERSTOOD “Marginal Effects of Advertising” … instead of how your advertising affects your competitors’ markets and how their advertising affects your market … instead of how your advertising affects your competitors’ markets and how their advertising affects your market “Spillover Effects of Advertising” … instead of more easily understood concepts like “profit” and “revenue” “Producer Surplus” “Economic Rent”

8. Results Not Straight-Forward in Their Interpretation Estimated AMEs reported in different forms and calculated in different ways RESULTS NOT UNDERSTOOD -Retail BCR: increase in retail sales revenue per dollar invested in advertising -Industry Profit or Producer Surplus BCR: increase in industry profit or producer surplus per dollar invested in advertising -Simultaneous BCR: increase in returns per $ of advertising assuming demand as well as supply, price and other relevant market variables can change as a result of advertising -Static BCR: increase in returns per $ of advertising assuming nothing (including price) changes except demand as a result of advertising -Discounted BCR: any of previous BCRs in which the returns are discounted to present value to account for the time value of money

9. Results Not Straight-Forward in Their Interpretation Estimated AMEs reported in different forms and calculated in different ways RESULTS NOT UNDERSTOOD -Retail BCR: increase in retail sales revenue per dollar invested in advertising -Industry Profit or Producer Surplus BCR: increase in industry profit or producer surplus per dollar invested in advertising -Simultaneous BCR: increase in returns per $ of advertising assuming demand as well as supply, price and other relevant market variables can change as a result of advertising -Static BCR: increase in returns per $ of advertising assuming nothing (including price) changes except demand as a result of advertising -Discounted BCR: any of previous BCRs in which the returns are discounted to present value to account for the time value of money

10. Results Not Straight-Forward in Their Interpretation Estimated AMEs reported in different forms and calculated in different ways RESULTS NOT UNDERSTOOD -Static BCR: increase in returns per $ of advertising assuming nothing (including price) changes except demand as a result of advertising -Simultaneous BCR: increase in returns per $ of advertising assuming demand as well as supply, price and other relevant market variables can change as a result of advertising -Discounted BCR: any of previous BCRs in which the returns are discounted to present value to account for the time value of money -Retail BCR: increase in retail sales revenue per dollar invested in advertising -Industry Profit or Producer Surplus BCR: increase in industry profit or producer surplus per dollar invested in advertising

11. Results Not Straight-Forward in Their Interpretation Estimated AMEs reported in different forms and calculated in different ways RESULTS NOT UNDERSTOOD -Simultaneous BCR: increase in returns per $ of advertising assuming demand as well as supply, price and other relevant market variables can change as a result of advertising of advertising assuming demand as well as supply, price and other relevant market variables can change as a result of advertising -Retail BCR: increase in retail sales revenue per dollar invested in advertising -Industry Profit or Producer Surplus BCR: increase in industry profit or producer surplus per dollar invested in advertising -Static BCR: increase in returns per $ of advertising assuming nothing (including price) changes except demand as a result of advertising -Discounted BCR: any of previous BCRs in which the returns are discounted to present value to account for the time value of money

12. Results Not Straight-Forward in Their Interpretation Estimated AMEs reported in different forms and calculated in different ways RESULTS NOT UNDERSTOOD -Retail BCR: increase in retail sales revenue per dollar invested in advertising -Industry Profit or Producer Surplus BCR: increase in industry profit or producer surplus per dollar invested in advertising -Static BCR: increase in returns per $ of advertising assuming nothing (including price) changes except demand as a result of advertising -Discounted BCR: any of previous BCRs in which the returns are discounted to present value to account for the time value of money -Simultaneous BCR: increase in returns per $ of advertising assuming demand as well as supply, price and other relevant market variables can change as a result of advertising

13. Results Not Straight-Forward in Their Interpretation No Clear Criteria for Interpreting AMEs RESULTS NOT UNDERSTOOD -BCR > 1 typically interpreted to mean that the returns to advertising > the costs -But how high must the BCR be in order to justify a conclusion that benefits of the program sufficiently exceed the costs to warrant continuation of the program? -Typical BCRs range from 2:1 to 10:1. So how do we interpret a BCR of 50:1 or 100:1? -And how do we interpret a BCR at the bottom of or below the typical range? Negative?

14. Results Do Not Answer Fundamental Questions of Checkoff Contributors Is the Advertising a “Good” Investment for Those Who Pay for the Program? RESULTS NOT UNDERSTOOD -Could they do better by keeping checkoff money and investing in other common investment alternatives? -Economists: A fundamental concern for program evaluations is the opportunity cost of the checkoff funds. -Non-Economists: If contributors think they could do better by investing checkoff funds in financial instruments and earn a greater return, then they will be unconvinced by an estimated BCR that is positive and “high.”

15. Results Do Not Answer Fundamental Questions of Checkoff Contributors Whether or Not and How Contributors Personally Benefit from Their Contributions? RESULTS NOT UNDERSTOOD -Contributors expect that the money they contribute to a Checkoff Program will be spent in such a way that they will individually be better off than they would have been without contributing. -The AMEs we typically provide do little to communicate how different groups of contributors with different economic characteristics benefit from checkoff program activities.

16. Costs Are Observable But Benefits Are Not Contributors See the Effects of Checkoff Assessments on their Bottom Lines But Have No Way of Knowing What Portion of Their Revenues Are Directly Attributable to Those Assessments RESULTS NOT “BELIEVABLE” -Evaluations specifically intended to measure the portion of industry revenues that can be attributed to checkoff promotion programs. -Compare actual sales or revenues to nebulous concepts like “what might have been earned in the absence of the program.” -We tell producers that the $5 or $6 of revenue earned for every $ spent on advertising is actually in their pockets but they just don’t know it.

17. Costs Are Observable But Benefits Are Not Contributors See the Effects of Checkoff Assessments on their Bottom Lines But Have No Way of Knowing What Portion of Their Revenues Are Directly Attributable to Those Assessments RESULTS NOT “BELIEVABLE” -Evaluations specifically intended to measure the portion of industry revenues that can be attributed to checkoff promotion programs. -Compare actual sales or revenues to nebulous concepts like “what might have been earned in the absence of the program.” -We tell producers that the $5 or $6 of revenue earned for every $ spent on advertising is actually in their pockets but they just don’t know it. This Concept Has Proved Difficult to Communicate.

18. Large BCRs Often Taken As Implying Large Market Impacts Checkoff Programs Are Only One of Many Factors That Impact Markets RESULTS NOT “BELIEVABLE” -Total checkoff assessment collections compared to farm sales are usually quite small (less than 0.5% in most cases). -Large BCRs can be calculated by dividing a small return by an even smaller cost. -So even if the BCR is large, the actual market impacts may actually be quite small and easily overwhelmed by the effects of the many other market forces. -Nevertheless, “large” positive BCRs are often taken to mean that the program has a large impact on market demand, prices, profitability.

19. Checkoff Boards Tend to Oversell Actual and Potential Program Impacts Contributors Come to Expect Large Impacts When the Impacts Are Usually Quite Modest RESULTS NOT “BELIEVABLE” -When they do not see the large effects they have been told to expect, they disbelieve the research results. -Contributors must be educated about the true potential of their checkoff programs. -Sell checkoff programs as tools to contribute to higher demand, prices, and profits in good years and to reduce downside pressure in bad years rather than as a panacea for the financial problems faced by contributors.

20. Focus on Inappropriate Measures of “Effectiveness” Market Price a Key Market Indicator But a Poor Indicator of Program Effectiveness RESULTS NOT “BELIEVABLE” -Price effects of advertising depend on how responsive supply is to price changes. -In industry with no supply controls and highly elastic (price responsive) supply, most benefits of advertising come from a sales increase rather than from a price increase. of advertising come from a sales increase rather than from a price increase. -In industry with inelastic (price unresponsive) supply, most advertising benefits come from price increase rather than from sales increase. -BUT long-run supply response can dampen price effects of advertising over many years.

21. Focus on Inappropriate Measures of “Effectiveness” The Appropriate Measure of “Effectiveness” Depends on Objectives of Promotion Program RESULTS NOT “BELIEVABLE” -We tend to report BCRs and other AMEs because we assume that the driving objective of all promotion programs is to enhance profitability. -Strategic Plans often reveal other objectives or sub-objectives of more immediate interest and relevance.

22. ALB VISION ALB STRATEGIC PLAN “A unified, thriving American Lamb Industry that concentrates its resources around a prioritized and measurable plan that fosters the opportunity for profitability of all Contributors.” 1.Increase American Lamb demand by 19% by 2010 2.Protect & enhance the business climate for American lamb 3.Leverage the ALB budget 4.Expand markets and market share for American lamb 5.Establish open and interactive communications with all segments of the industry 6.Gather research data and evaluate programs

23. “A unified, thriving American Lamb Industry that concentrates its resources around a prioritized and measurable plan that fosters the opportunity for profitability of all Contributors.” 4.Expand markets and market share for American lamb 1.Increase American Lamb demand by 10% by 2010 2.Protect & enhance the business climate for American lamb 3.Leverage the ALB budget 5.Establish open and interactive communications with all segments of the industry 6.Gather research data and evaluate programs ALB VISION ALB STRATEGIC PLAN

24. Recognize that Communication is a Two-Way Street For Researchers to Communicate Better, Then Checkoff Groups Need to Better Communicate What They Want from an Evaluation IMPROVING COMMUNICATION -What do you want to know about your programs? -What metrics do you focus on? -What implies program success to you? Increased Sales? Higher Price? Industry Growth? Increased Exports? Reduced Imports?

25. Recognize that Communication is a Two-Way Street Also, Researchers Need to Know Why a Program Evaluation is Wanted IMPROVING COMMUNICATION -Legislative reporting requirements? -Assist in program management? -Strategic Plans provide insights into appropriate evaluation objectives and measures. Let the researchers design the study objectives and measures of effectiveness. Researchers need to know what that means to you so they can provide measures to meet your needs.

26. Inform Boards About What Can and Cannot Be Done They Usually Do Not Understand Extensive Data Requirements IMPROVING COMMUNICATION -Consumer attitude and awareness studies typically done by advertising firms, market research firms, and business school researchers. They Need to Understand the Types of Evaluations Typically Not Done By Agricultural Economists -The typical checkoff program evaluation attempts to measure directly the effects of advertising on demand without concern for attitudes and awareness. PROMOTION RETAILSALES/PRICE

27. Provide reasonable estimates of the returns to specific checkoff program activities. Provide reasonable estimates of the returns to specific checkoff program activities. Provide reasonable estimates of the returns to checkoff program activities in the aggregate. Provide reasonable estimates of the returns to checkoff program activities in the aggregate. Inform Boards About What Can and Cannot Be Done IMPROVING COMMUNICATION They Also Need to Understand What Our Tools Will Allow US to Do and NOT Do Effectively -CAN DO EFFECTIVELY: -CANNOT DO EFFECTIVELY: ? FirmPerformance

28. Inform Boards About What Can and Cannot Be Done IMPROVING COMMUNICATION They Also Need to Understand What Our Tools Will Allow US to Do and NOT Do Effectively -CAN DO EFFECTIVELY: Provide reasonable estimates of the returns to checkoff program activities in the aggregate. -CANNOT DO EFFECTIVELY: Provide reasonable estimates of the returns to specific checkoff program activities. AllFirmActivities FirmPerformance

29. Other Ways to Improve Communication IMPROVING COMMUNICATION Reduce economic jargon in reports Standardize and categorize measures of program effectiveness Develop measures that more readily convey the success or failure of checkoff programs in meeting their objectives Be careful not to imply that large estimated returns mean large program effects on demand, prices, profits, etc. Inform and educate on what checkoff programs can reasonably be expected to achieve in terms of magnitudes of effect

30. IMPROVING COMMUNICATION Be careful not to oversell what advertising programs can accomplish $ Sales $ Advertising Expenditures If you take all the credit for periods when sales and prices are growing … … then you’ll have to take the blame if sales and price ever drop. And contributors won’t believe the estimates of positive returns to the advertising paid for with their checkoff assessments. Other Ways to Improve Communication