PUBLIC RELATIONS PROCESS EVALUATION OF PR EFFECTIVENESS

Today’s Learning Objectives Understanding the need for evaluation Learning the ways in which evaluation is conducted Knowing closed system and open system evaluation techniques We will learn the different ways to evaluate the results of a campaign and how to measure those results in order to prove a campaign’s value. Why learn about evaluation?

Evaluation Is Used To Answer… What have we accomplished? What remains to be done? What benchmarks can help us next time?

Evaluation Proves PR Value Competent evaluation demonstrates the value of public relations to management. That means your value, at least to the company. Merging research and evaluation…

Research Results Assist in Evaluation Communication To Target Publics Measurement Techniques Objectives and Strategies Response Results

Evaluating Results Evaluating research can tell: How many audience members received messages. How many changed an attitude or opinion. How many changed behavior. These could equally be analyzed using various web bases media tools like Neilsen, MRI, HookMedia

Using Research Results The public relations practice is cyclical in nature. Research data from the first cycle form the basis for planning subsequent efforts. Changes may be made in: Program strategies and/or tactics Targeted audiences Media and messages The organization’s products, services, policies, or procedures

The National Youth Anti-Drug Media Campaign Click on the image to see how the National Youth Anti-Drug Campaign conducted its evaluation or this link http://www.mediacampaign.org/publications/message99/1_2.html

Evaluating Implementation Is implementation being conducted as planned? After variations are analyzed and explained … Should the plan be modified or … Can the discrepancies otherwise be corrected? In-progress monitoring is similarly handled…

Evaluating Program Progress Determine program effectiveness. Identify unanticipated results. Assess their impact on overall outcome. Make adjustments as necessary. Outcome evaluation is the final need…

Evaluating Outcomes Compare outcomes with objectives. Interpret results. Report results to management. Make recommendations concerning subsequent programs or program cycles. Measuring HIV campaign in Zaire…

Four Measurement Strategies … The extent audiences are covered Levels of audience response The impact of the campaign as a whole The extent of environmental influences We’ll examine them individually …

Audience Coverage Is Measured By … Maintaining complete records of messages sent and media usage to determine the extent that… Targeted audiences are exposed to messages. Non-targeted audiences receive messages. Now we’ll look at audience response …

Messages Can Be Measured By … Pre-testing messages using… The Flesch Formula The Gunning Fog Index The Dale-Chall Formula The Cloze Procedure Post-testing messages by using survey techniques to determine change in attitudes or behaviors Now we’ll look at environmental factors …

External Influences on Campaigns External factors can influence campaign outcomes by… Magnifying success or failure Measurement error can compound problems …

Guidelines for Measurement Trust me, men. I had 500 leaflets passed out today. There will be no strike tomorrow. Volume of output does not determine results. The impact of news releases does not increase with their number. Estimate is not measurement. Experience and intuition are not sufficient to gauge effectiveness.

Sampling Selection Samples should be carefully selected. Wrong decisions often result from volunteered or unsystematically collected comments. More common errors …

Effort = Knowledge The Effort Error The amount of effort expended does not equal understanding of the audience’s response.

Knowledge = Favorable Attitude The Knowledge Error Knowledge = Favorable Attitude What audiences know about a topic does not necessarily produce favorable reactions.

Attitude = Behavior The Attitude Error Favorable perceptions may halt opposition but not produce support.

PR Evaluation Traditional Evaluation Output measurement Survey of distribution system Number of potential audience impressions

Weaknesses of Traditional Approach No Way to assess quality Cost Effectiveness isn’t measured No opportunity for improvement

Measurement should include: Behavior Attitudes Knowledge Awareness Interest/Expectation

Levels & Criteria for Evaluation Preparation Adequacy of background information Appropriateness of message content & organization Quality of message presentations

Implementation Number of messages sent—distribution Number of messages appearing in media Number of people receiving message Number of people attending to the message

Impact Number who learn message content Number who change opinions Number who change attitudes Number who behave in desired fashion Goal achieved/problem solved Social and Cultural change

Evaluation Techniques May Include … Impact analysis Audience coverage Audience response Campaign impact Environmental assessment

Measurement Techniques Closed and open systems of evaluation help determine the effectiveness of programs

Closed System Evaluation … The closed system is limited to planned messages and events and their effects on publics.

The Closed System Has Two Tests Pre-test of Message and Media Post-test of Message and Media Campaign Time

Pre-test/Post-test Design Examines … Amounts and costs of public relations tools used Channels used to convey messages The extent audiences heeded messages Whether audiences received messages Readability of messages transmitted The process has limitations…

Weaknesses In Closed System Method Successfully transmitting messages does not equal campaign success. Potential for success is influenced by external as well as internal factors. Pre-test/post-test techniques measure only the extent attitudes have changed. Now we’ll look at the open system method…

Open System Evaluation … Is most needed in connection with continuing or long-range programs, which require feedback throughout the process. Attempts to account for factors outside practitioner’s ability to control. Emphasizes organizational and external influences on the process. Turning to techniques…

Open Systems Evaluation Has Two Tools Environmental monitoring and social audits are used as tracking tools. Social Audits Environmental Monitoring Slide 34 of 50

Environmental Monitoring Environmental monitoring can be quite extensive and expensive. There are companies that provide environmental monitoring--another item that can be outsourced.

Open Systems Evaluation Can … Assess the program’s impact on the environment, and vice versa.

Open Systems Evaluation Can … Track both internal and external influences. An open system approach in practice …

AT&T’s Open System Includes: Administrative processes Employee publications Media relations

Another Way to Classify Evaluation Summative: Evaluation completed after the campaign or program is over “how did the campaign do?” Formative: Ongoing evaluation of in-progress programs or campaigns “how are we doing?”

Five Publication Measures Readability Gunning readability Understanding Average question response Recall and comprehension percentage Reliability Awareness Average recall Percentages of issues delivered within 38 hours Distribution

Publication Measures Explained … Distribution – Percentages of issues delivered within 38 hours Awareness – Average recall of data Reliability – Average response to questions about comprehension, balance and information content Understanding – Percentage recalling and comprehending a story Readability as measured with the Gunning scale

In Summary … Evaluation is a new beginning rather than a final step. The process provides new direction for on-going programs. Evaluation establishes the value of public relations programs.