Show Me the Money: Moving From Impact to ROI Patti Phillips, Ph.D. patti@roiinstitute.net

Objectives Isolate the effects of a program Convert benefits to monetary value Identify the intangible benefits Tabulate the fully-loaded costs Calculate ROI .

The ROI calculation is simple. Program Benefits Program Costs BCR = Net Program Benefits Program Costs X 100 ROI =

Try it! $750,000 $425,000 BCR = $750,000 - $425,000 $425,000 ROI = X 100

Try it! $750,000 $425,000 = 1.76:1 BCR = $750,000 - $425,000 $425,000 ROI = X 100 = 76%

While we don’t strive for a negative ROI, negative is not always bad! What Makes a Good ROI? Set the value at the same level as other investments – 15% Set slightly above other investments – 25% Set at break even – 0% Set at client expectations While we don’t strive for a negative ROI, negative is not always bad!

xxx Training Presenter Guide Levels of Evaluation Measurement Focus 7 Typical Measures 0. Inputs and Indicators The input into the project in terms of scope, volume, efficiencies, costs Participants, Hours, Costs, Timing 1. Reaction & Perceived Value Reaction to the project or program, including the perceived value Relevance, Importance, Usefulness, Appropriateness, Intent to use, Motivation to take action 2. Learning Learning to use the content and materials, including the confidence to use what was learned Skills, Knowledge, Capacity, Competencies, Confidence, Contacts 3. Application & Implementation Use of content and materials in the work environment, including progress with actual items and implementation Extent of use, Task completion, Frequency of use, Actions completed, Success with use, Barriers to use, Enablers to use 4. Business Impact The consequences of the use of the content and materials expressed as business impact measures Productivity, Revenue, Quality, Time, Efficiency, Customer Satisfaction, Employee Engagement 5. ROI Comparison of monetary benefits from program to program costs Benefit-Cost Ratio (BCR), ROI%, Payback Period

ROI Methodology Stage 1 Stage 2 Data Collection Evaluation Planning Collect Data After Solution Implementation Level 3 Level 4 Develop Objectives Of Solution Develop Evaluation Plans and Baseline Data Collect Data During Solution Implementation Level 1 Level 2

Stage 4 Communicate Results Capture Costs Of Solution Generate Impact Study Report Stage 4 Communicate Results Stage 3 Data Analysis Isolate the Effects of Solution Convert Data to Monetary Value Calculate the Return On Investment Level 5 Identify Intangibles Measures Intangible Benefits

Operating Standards Report the complete story Conserve resources Use the most credible sources Choose the most conservative alternatives Isolate the effects of the program No data no improvement Adjust estimates for error Throw out the extreme and unsupported claims Use first year benefits for short-term programs Include fully-loaded costs Report intangible benefits Communicate results to all stakeholders

Isolate the Effects of the Program Reaction Learning Application Impact ROI Isolate the Effects of the Program Intangible Benefits Inputs

Methods to Isolate Program Effects Use of a control group arrangement Trend line analysis of performance data Use of forecasting methods of performance data Participant’s estimate of program impact (percent) Supervisor’s estimate of program impact (percent) Manager’s estimate of program impact Use of expert/previous studies Calculate/estimate the impact of other factors Customer input NOTES: Which techniques are appropriate in your organization?

Isolating the Effects of the Program Frequency of Methods Used * Survey of Users, N = 235

Classic Control Group Design What is the difference in improvement? Experimental Group Pre-Measure Program Post-Measure Control Group Post-Measure Pre-Measure

Post-Program Only Design What is the difference? Experimental Group Program Post-Measure Control Group Post-Measure

Example of Trend Line Analysis 1.85% Pre Program Average CPI Program Conducted 2% 1% 1.45% Projected Average ERROR RATE .7% Post Program Average J F M A M J J A S O N D J MONTHS

Average Confidence (%) Example of Estimation Monthly increase in credit card accounts: 175 (fact) Contributing Factors Consensus Impact (%) Average Confidence (%) Sales Training 32% 83% Incentives 41% 87% Management Reinforcement 14% 62% Market Fluctuations 11% 91% Other _________ 2% 100%

Example of Estimations Influence Fact % Contrib. Est. Impact Confidence Adjusted Impact Sales Training 175 32% 56 83% 46.48 Fact: 175 New Credit Card Accounts % Contribution: 32% Est. Impact: 56 Uncertainty 17% Margin of Error: +/- 9.52 65.52 56 46.48

Data are converted by: Converting output to contribution – standard value Converting the cost of quality – standard value Converting employee’s time – standard value Using historical costs Using internal and external experts Using data from external databases Linking with other measures Using participants’ estimates Using supervisors’ and managers’ estimates Using staff estimates

Cost of A Sexual Harassment Complaint 35 Complaints Actual Costs from Records Additional Estimated Costs from Staff Legal Fees, Settlements, Losses, Material, Direct Expenses EEO/AA Staff Time, Management Time 1877 807 4646 $852,000 Annually $852,000 35 Cost per complaint = $24,343

5 Steps to Convert Impact to Money Step 1: Focus on a unit of measure Step 2: Determine the value (V) of each unit Step 3: Calculate the change in performance (∆P) Step 4: Determine the annual amount of the change (A∆P) Step 5: Calculate the total annual value of the improvement (A∆P x V)

Example Using Internal Experts Step 1: One grievance Step 2: V = $6,500 (from Director of nursing and HR Experts) Step 3: ∆P = average of 7 out of 10 grievances prevented per month Step 4: Annual ∆P = Step 5: A∆P x V =

Example Using Internal Experts Step 1: One grievance Step 2: V = $6,500 (from Director of nursing and HR Experts) Step 3: ∆P = average of 7 out of 10 grievances prevented per month Step 4: Annual ∆P = 84 Step 5: A∆P x V = $546,000

Example Using Standard Values Step 1: One sale Step 2: V = 30% profit margin Step 3: ∆P = $20,000 revenue per month Step 4: Annual ∆P = Step 5: A∆P x V =

Example Using Standard Values Step 1: One sale Step 2: V = 30% profit margin Step 3: ∆P = $20,000 revenue per month Step 4: Annual ∆P = $240,000 Step 5: A∆P x V = $240,000 x .30 = $72,000

Fully-Loaded Cost Profile Assessment Costs (Prorated) Development Costs (Prorated) Program Materials Instructor/Facilitator Costs Facilities Costs Travel/Lodging/Meals Participant Salaries and Benefits Administrative/Overhead Costs Evaluation Costs Notes: Which cost categories are included in your calculations? ____________________________________

Financial ROI = 258% Program Skills Based Pay Systems Business Impact Employee Turnover Staffing Level Customer and Job Satisfaction Product Sales Cross Selling ROI = 258%

Telecommunications ROI = 163% Program All-Inclusive Workforce Program (AIW) Business Impact Attrition Rate Employee Satisfaction Communication Cooperation & Teamwork Diversity Mix ROI = 163%

Health Systems ROI = 1,051% Program Sexual Harassment Prevention Workshop Business Impact Turnover Reduction Complaint Reduction Job Satisfaction Absenteeism Stress Reduction Recruiting ROI = 1,051%

Technology ROI = -85% Program Customer Service Training Business Impact (Target) Customer Complaint Escalation ROI = -85%

The people who like ROI do so because the ROI Methodology: Why do people like ROI? The people who like ROI do so because the ROI Methodology: Generates a balanced set of measures Employs a step-by-step process Bridges evaluation disciplines Balances research and reality Offers a flexible evaluation solution Provides a credible approach to measurement and evaluation

Next Steps Assess your readiness for ROI Identify stakeholders and their data needs Determine the purpose of your evaluation practice Identify programs suitable for ROI Develop capability in the ROI Methodology

Success with measurement begins with your worldview There are no absolutes. If you are measuring, you are estimating. Be able to explain what you did, how you did it, and why you did it that way. Sometimes the crowd knows best. Process without standards is no process. A statistic is an estimate of what probably is… maybe.

The rest is merely a balancing act. Benefits Costs