Show Me The Money Measuring ROI for People, Projects, and Programs Jack J. Phillips, Ph.D.

Learning Objectives Make the business case for "showing the money" for a few select programs Identify the steps, guiding principles, and issues involved in "showing the money" Use impact and ROI data to drive improvement and change

The Value Evolution

The Many Applications of the ROI Methodology Communications Policies/Procedures Consulting/OD Procurement/Supply Chain Educational Systems Public Policy Programs Human Resources/Human Capital Public Relations / Public Affairs Leadership and Coaching Quality / Six Sigma Learning and Development Research and Development Marketing/Advertising Social Programs Meetings and Events Technology / Systems

The “New” Definition of Value Value Must: Be balanced, with qualitative and quantitative data Contain financial and non-financial perspectives Reflect strategic and tactical issues Represent different time frames Satisfy all key stakeholders Be consistent in collection and analysis Be grounded in conservative standards Come from credible sources Reflect efficiency in its development Create a call for action

The Changing Face of Value Old New Activity cost center Results profit center Expense control Maximize Value Human Resources as expenses Human capital as investment Rule centered Client centered We value what IBM and GE value Our value systems are unique Tolerate overhead Outsource or automate overhead Add value with small pieces Add value with integrated applications Most of us are in a support role All of us are in a sales role Our measures are based on benchmarking Our measures are based on what we need We view value from one perspective We view value from everyone’s perspective Just another days work in the cubicle We do something of value A job A performance Treat old ideas as new ideas Treat old ideas as old ideas

The “Show Me” Evolution Term Issue Show Me! Collect Impact Data Show Me the Money! And Convert Data to Money Show Me the Real Money! And Isolate the Effects of the Project Show Me the Real Money, And Make Me Believe it! And Compare the Money to the Cost of the Project

How “Show Me the Money” Connects Organizational Measures (Sponsor) Personal Measures (Stakeholders) Financial Measures (CFO) SHOW ME THE MONEY!!

Varieties of Terms and Applications Name Example Program A leadership development process for senior executives Project A re-engineering project for the plastics division System A fully networked system for all branches Initiative A faith-based initiative to reduce recidivism Policy A new pre-school policy for the disadvantaged Procedure A new scheduling procedure for truck drivers Event A golf outing for customers Meeting US Coast Guard innovations conference Process Quality sampling process People Staff additions in the customer care center Tool A new selection tool for the hotel staff

Results reacted very positively to the program and found it to be very relevant to their work; learned new skills and gained new insights about themselves; utilized the skills and insights routinely with their teams, although they had some difficulty in a few areas; improved several important work unit measures, with some measures improving as much as 28%; achieved an impressive 105% return on investment; and reported an increase in job satisfaction in the work unit.

Key Issues with this Level of Analysis Objectives? Credibility of data? Source of data? Consistent methodology? Scope? Standards? Use of data? Cost of process? Fear of data?

Project or Program Value Chain Level Measurement Focus 0. Input Measures input such as volume and efficiencies Reaction & Planned Action Measures participant reaction to the program and captures planned actions Learning Measures changes in knowledge, skills, and attitudes Application Measures changes in on-the-job behavior or actions and progress with actions Business Impact Captures changes in business impact measures 5. ROI Compares program benefits to the costs

Global Trends in Measurement and Evaluation Organizations are moving up the value chain in their evaluation strategies Investment is increasing to 3-5% of the budget Increase focus is driven by clients and sponsors ROI is the fastest growing metric . . . and

Global Trends in Measurement and Evaluation Evaluation data is used to drive improvement and secure funding Evaluation is addressed early and often in the implementation cycle Processes are systematic and methodical, often designed into the delivery and implementation processes Technology is significantly enhancing processing

Paradigms are shifting to results? Activity Based Results Based No business need for the program Program linked to specific business No assessment of performance issues Assessment of performance effectiveness No specific measurable objectives Specific objectives for behavior & business impact No effort to prepare program participants to achieve results Results expectations communicated to participants and . . .

Paradigms are shifting to results? Activity Based Results Based No effort to prepare the work environment to support transfer Environment prepared to support transfer No efforts to build partnerships with key managers Partnerships established with key managers and clients No measurement of results or benefit-cost analysis Measurement of results and benefits-cost analysis Planning and reporting is input focused Planning and reporting is outcome focused

Why does ROI work? Provides a balanced set of measures Offers a step-by-step process Bridges business evaluation and program evaluation Balances research/statistical methods with practical application Flexible for all types of programs Credible with managers and administrators

Reliance Insurance

Is Your Organization a Candidate for ROI Implementation?

Scoring If you scored: 15 – 30 You are not yet a candidate for ROI. 31 – 45 You are not a strong candidate for ROI, however, it is time to start pursuing some type of measurement process. 46 – 60 You are a candidate for building skills to implement the ROI process. At this point there is no real pressure to show the ROI, which is the perfect opportunity to perfect the process within the organization. 61 – 75 You should already be implementing a comprehensive measurement and evaluation process, including ROI.

Private Sector Organizations using ROI Include: Accenture Allstate Insurance Apple Computer AT&T Bank of America Banner Healthcare Baptist Medical Center Bristol-Myers Squibb Children’s Hospital of Los Angeles Caremark RX Covenant Healthcare Systems Dell Computers Deloitte & Touche Delta Airlines DHL Worldwide Express Federal Express Genentech Georgia Pacific Guthrie Healthcare Systems Hewlett-Packard Intel Lockheed Martin Memorial Foundation Hospital Menlo Logistics Microsoft Motorola NCR Nextel Pricewaterhouse Coopers QUALCOMM Shell Oil UPS Wachovia Bank Wal-Mart

Public Sector Organizations using ROI Include: US Department of Defense US Department of Navy US Department of Labor US National Security Agency Central Intelligence Agency US Department of Veteran’s Affairs NASA State of Mississippi State of Texas State of New York Government of New Zealand Government of Singapore Government of Italy Government of Australia Government of Canada

The ROI Methodology Generates six types of measures: Reaction and Planned Actions Learning Application and Implementation Business Impact Return on Investment Intangible Measures . . . and includes a technique to isolate the effects of the program or solution

The ROI Methodology is Grounded in Research and Application 5,000 impact studies each year. 20,000 individuals have attended a two-day ROI workshop. 4,000 individuals are certified to implement the ROI methodology. The ROI methodology has been adopted by hundreds of organizations in manufacturing, service, non-profit, and government settings in 44 countries. and . . .

The ROI Methodology is Grounded in Research and Application The process has been refined over a 20-year period. 16 books have been developed to support the process. ROI Network has 600 members Several ROI Network conferences are conducted annually Ongoing research on the ROI methodology is conducted by the ROI Institute, Inc. to support practitioners.

The ROI Calculation Program Benefits Program Costs BCR = Net Program Benefits Program Costs ROI = X 100

Measurement of Communication Projects Level Measurement Category Current Status Coverage Recommending Comments About Status Inputs/Indicators Measures inputs into communication projects including number of projects, audience, costs, and efficiencies. 100% This is being accomplished now. 1 Reaction and Perceived Value Measure reaction to, and satisfaction with, the medium, content, and value of the communication. 80-100% Need more focus on content and perceived value. 2 Learning Measures what participants understand or learned from the communication – information, knowledge, skills, and contacts (take- aways). 50-60% Must use simple learning measures.

3 Application and Implementation Measures progress after the communication – the use of information, knowledge, skills, and contacts. 15-25% Need more follow-up. 4 Impact and Consequences Captures changes in business impact measures such as output, quality, time, and cost-linked to the communication. 10% This is the connection to business impact. 5 ROI Compares the monetary benefits of the business impact measures to the costs of the project. 5% The ultimate evaluation.

Five Levels of Measurement - Examples Level 0 Input and Indicators Level 1 Reaction and Perceived Value Level 2 Learning Level 3 Application and Implementation Level 4 Impact and Consequences Level 5 Return on Investment

Five Levels of Measurement - Examples Input React Learn Apply Impact ROI

Evaluation is like a Puzzle An Evaluation Framework Case Applications and Practice Implementation A Process Model Operating Standards and Philosophy

ROI Methodology Stage 1 Stage 2 Data Collection Evaluation Planning Develop Objectives Of Solution Develop Evaluation Plans and Baseline Data Collect Data During Solution Implementation Collect Data After Solution Implementation Level 1 Level 2 Level 3 Level 4

Stage 4 Communicate Results Capture Costs Of Solution Stage 4 Communicate Results Stage 3 Data Analysis Generate Impact Study Report Isolate the Effects of Solution Convert Data to Monetary Value Calculate the Return On Investment Level 5 reinforce the four-part test; refer to the flow chart on the next page. Identify Intangibles Measures Intangible Benefits

The Guiding Principles 1. When a higher level evaluation is conducted, data must be collected at lower levels. 2. When an evaluation is planned for a higher level, the previous level of evaluation does not have to be comprehensive. 3. When collecting and analyzing data, use only the most credible sources. 4. When analyzing data, choose the most conservative among alternatives. At least one method must be used to isolate the effects of the project/initiative. If no improvement data are available, it is assumed that little or not improvement has occurred. and . . .

The Guiding Principles Estimates of improvement should be adjusted for the potential error of the estimate. Extreme data items and unsupported claims should not be used in ROI calculations. Only the first year of benefits (annual) should be used in the ROI analysis of short-term projects/initiatives. Project/program costs should be fully loaded for ROI analysis. Intangible measures are defined as measures that are purposely not converted to monetary value. The results from the ROI methodology must be communicated to all key stakeholders.

Report the Chain of Impact Reaction & Planned Action Learning Application & Implementation Isolate the Effects of the Program Impact ROI Intangible Benefits

Multiple Stakeholder Chain of Impact Reaction & Planned Action Learning Application & Implementation Stakeholder 2 Impact Organization Impact Stakeholder 3 Impact ROI Intangibles ROI Intangibles ROI Intangibles

Characteristics of Evaluation Levels Power to Show Results Chain of Value of Difficulty of Impact Information of Use Assessment Lowest Frequent Easy Reaction Learning Application Impact ROI Highest Infrequent Difficult Focus Consumer Lowest Client Highest Consumers = The customers who are actively involved in the process. Client = The customers who fund, support, and approve the project

Cost-benefit Level 5 comparison Evaluation Purpose Changes in key Program Need Program Profile Stakeholder Needs Level 4 Changes in key outcome measures Level 3 Changes in performance Level 2 Need for skills or knowledge Level 1 Preferences © 2001 Patricia Pulliam Phillips

When Selecting Programs for Level 4 and 5 Evaluation, Consider the Following: Life cycle of the program Linkage of program to operational goals and issues Importance of program to strategic objectives Top administrator interest in the evaluation Cost of the program Visibility of the program Size of target audience Investment of time required Refer to the matrix on the next page and explain how to use it. Also, refer to examples of programs that have been evaluated to ROI from the public sector. One we will walk through in just a moment.

Evaluation Targets Percent of Courses Level Current Target Suggested Reaction Learning Application (Behavior) Impact Return on Investment 90 - 100% 40 – 60% 30% 10 - 20% 5 - 10%

ROI Application Executive Education Leadership Diversity Programs Wellness/Fitness Initiatives Total Quality Management Self-Directed Teams Skill-Based/Knowledge-Based Compensation Organizational Development Meeting Planning Competency Systems Career Development Programs Recruiting Strategies Orientation Systems Associate Relations Programs Gainsharing Programs Technology Implementation e-Learning Safety & Health Programs Project Management

Matching Evaluation Levels with Objectives Reaction Learning Application Business Impact Return on Investment

Program Objectives Provide Direction to designers and developers Guidance to instructors and facilitators Goals for participants Satisfaction for program sponsors A framework for evaluators

Assessment Objectives Evaluation Needs Program Assessment Objectives Evaluation Payoff ROI ROI Needs Objectives 5 5 Business Impact Business Needs Objectives Impact 4 4 3 Job Performance Application Application Needs Objectives 3 2 Skills/Knowledge Learning Learning Needs Objectives 2 1 Preference Satisfaction Reaction Needs Objectives 1

5 5 4 4 3 3 2 2 1 1 Absenteeism is costing $100,000 month Unexpected absenteeism problem exists Dialogue between team leader/supervisor is not occurring when there is an unexpected absence Deficiency in counseling/discussion skills Two-day communication skills workshop must provide usable and relevant skills; facilitator-led; participants include supervisors and team members ROI of at least 25% Weekly absenteeism rate will reduce by 10% six months after course Dialogue takes place in 95% of situations when an unexpected absence occurs. Environment supports safe dialogue. Acquisition of dialogue skills are demonstrated Program receives favorable rating of 4 out of 5 on relevance; participants identify three planned actions Calculate the ROI Monitor absenteeism data for six months Follow-up questionnaire to participants to check frequency of skill application Identify barriers to dialogue- three months post program Skill practice sessions during program Reaction questionnaire at the end of program

Key Alignment Questions Is this a problem worth solving? Is there a potential pay off? Needs Program Assessment Objectives Evaluation 5 Potential ROI ROI Payoffs Objectives 5 What is the actual ROI? What is the BCR?

Key Alignment Questions What is the specific measure? What happens if we do nothing? Needs Program Assessment Objectives Evaluation 4 Business Impact Business Needs Objectives Impact 4 Which business measure improved? How much is related to the program?

Key Alignment Questions What is occurring or not occurring on the job that influences the business measure? Needs Program Assessment Objectives Evaluation 3 Job Performance Application Application Needs Objectives 3 What has changed? Which skills/knowledge has been applied?

Key Alignment Questions What skills or knowledge is needed to support the job performance need? Needs Program Assessment Objectives Evaluation 2 Skills/Knowledge Learning Learning Needs Objectives 2 What did they learn? Who did they meet?

Key Alignment Questions How should the solution be structured? Needs Program Assessment Objectives Evaluation 1 Satisfaction Preferences Objectives Reaction 1 What was the reaction to the program? Do we intend to implement the program?

Developing Reaction Objectives Developing Learning Objectives

Developing Application Objectives Developing Impact Objectives

Developing Level 3 and 4 Objectives

Option 1, When You Don’t Have a Clue Option 2, When the Measure is in a Defined Set Option 3, When the Measure is Known

Plan Your Project Evaluation Worksheet

Data Collection Plan

ROI Analysis Plan

Collecting Post Program Data Level 3  Level 4  Follow-Up Surveys Follow-Up Questionnaires Observation On the Job Interviews with Participants Follow-Up Focus Groups Program Assignments Action Planning Performance Contracting Project Follow-Up Session Performance Monitoring

Factors to Consider When Selecting Data Collection Methods Include: Type of data Time – Participant / Supervisor Costs Accuracy – Validity / Reliability Utility Culture / Philosophy

Factors to Consider When Determining Timing of Follow-up Include: Availability of data Ideal time for behavior change (Level 3) Ideal time for business impact (Level 4) Convenience of collection Constraints on collection

Methods to Isolate Program Effects Use of a control group arrangement Trend line analysis of performance data Use of forecasting methods of performance data Participant’s estimate of program impact (percent) Supervisor’s estimate of program impact (percent) Manager’s estimate of program impact Use of expert/previous studies Calculate/estimate the impact of other factors Customer input NOTES: Which techniques are appropriate in your organization?

Applications of Data Collection Instruments A – Survey B – Test C – Questionnaire D – Interview E – Focus Group F – Observation G – Performance Records

International Sales

Control Group Method Design Experimental Group M1 Program M2 Control Group M1 M2

Control Group Method Design Experimental Group Program M1 Control Group M1

Control Group Method Design Experimental M1 Program M2 Control M1 M2 Experimental (No pre-measure) Program M1

Example of Trend Line Analysis 1.85% Pre Program Average CPI Program Conducted 2% 1% 1.45% Projected Average ERROR RATE .7% Post Program Average J F M A M J J A S O N D J MONTHS

Isolating the Effects of a Program A – Control Group B – Trend line analysis C – Forecasting D – Participant’s estimate E – Use of customer input F – Expert’s estimate

Example of Estimation

Credibility of Data is Influenced by: Reputation of the source Source of data Source of the study Biases Motives of the researcher Personal bias of the audience Methodology Assumptions made in the analysis Realism of the outcome data Type of data Scope of analysis

Data are Converted by: Converting output to contribution – standard value Converting the cost of quality – standard value Converting employee’s time – standard value Using historical costs Using internal and external experts Using data from external databases Linking with other measures Using participants’ estimates Using supervisors’ and managers’ estimates Using staff estimates When we offer a time management program. A manager attends. In the past she spent about 5 hours on Saturday working. As a manager she’s exempt for overtime. She goes to the time management program. She no longer comes in on Saturday. Are her 5 hours counted as times savings for the organization? No. She’s exempt. Yes, there are a lot of intangibles – may her attitude at work is better. Maybe her team is working better together because she feels less stressed. But because the 5 hours were not paid for by the company, those 5 hours saved do not count. However, if she saved five hours productive time on the job (during the work week), we would count them. We ask participants what percent of their time on the job did they save due to the program. They say, 5 hours. Okay, how much of that was actually productive time? They think about it and say well, about half was spent making customer calls, so let’s say 50% was productive time saved. So, we put 2.5 hours saved. What method on the list would we use to convert time savings to monetary value? Probably salary & benefits (the third one could be considered a standard value). If we don’t use salary & benefits we’ll work our way through the list… these are listed from most to least credible, staff estimates (that’s us) being least credible.

Example: Cost of One Turnover from External Database Salary of Middle Manager $70,000/annually Value of Turnover* 150% of annual salary Cost of Turnover $105,000 * Value obtained from industry-related study (external data)

Cost of a Sexual Harassment Complaint 35 Complaints Actual Costs from Records Additional Estimated Costs from Staff Legal Fees, Settlements, Losses, Material, Direct Expenses EEO/AA Staff Time, Management Time $852,000 Annually $852,000 35 Cost per complaint = $24,343

To Convert or Not Convert Is there a standard value? Is there a method to get there? Can we get there with minimum resources? Can we convince our executive in two minutes that the value is credible

Converting Data to Money A – Profit/savings form output B – Standard Value C – Employee time as compensation D – Historical Costs/savings form records E – Expert input F – External database G – Linking with other measures H – Participant estimation I – Management estimation J – Estimation for HR staff

5-Step Data Conversion Focus on a unit of measure Determine the value (V) of each unit Calculate the change in performance data (∆P) Determine an annual amount for the change (A∆P) Calculate the total value of the improvement (A∆P x V)

Example using Internal Experts Step 1: One grievance Step 2: V = $6,500 (from Director of Nursing and HR experts) Step 3: ∆P = 7 out of 10 grievances prevented per month due to program Step 4: Annual ∆P = 84 Step 5: A∆P x V = See notes in WTD workbook. 1 grievance V = $6,500 (internal experts) – the cost of one grievance ChangeP = six months after the program, 7 out of 10 grievances are related to the training (keep in mind there are other influences, so we can only claim 7 out of 10 grievances to have been influenced by our program. Using the six month value of training of 7 per month, yielded an annual improvement of 84 84 x 6,500 = $546,000 – cost savings (this goes in the numerator of the ROI equation) $546,000

Which Cost Category is Appropriate for ROI? B Operating Costs Support Costs Administrative Costs Participant Compensation and Facility Costs Classroom Costs C D Program Development Costs Participant Costs Analysis Costs Development Costs Delivery Costs Evaluation Costs

Fully-Loaded Costs Profile Assessment Costs (Prorated) Development Costs (Prorated) Program Materials Instructor/Facilitator Costs Facilities Costs Travel/Lodging/Meals Participant Salaries and Benefits Administrative/Overhead Costs Evaluation costs Notes: Which cost categories are included in your calculations? ____________________________________

Potential Intangible Benefits Increased Job Satisfaction Increased Organizational Commitment Improved Teamwork Improved Customer Service Reduced Complaints Reduced Conflicts Reduced Stress

ROI is Reported One of Two Ways Program Benefits Program Costs BCR = Net Program Benefits Program Costs ROI (%)= X 100

Calculate the ROI Costs per program (25 participants) $80,000 Benefits per program (1st year) $240,000 BCR = = ROI = x 100 = $240,000 3 $80,000 BCR = 230,625/88,500 = 2.61 ROI = 142,125 / 88,500 X 100 = 161% $160,000 200% $80,000

What is an Acceptable ROI? Set the value as with other investments -15% Set slightly above other investments - 25% Set at break even - 0% Set at client expectations

Evaluation Targets

Criteria for Selecting Programs for Levels 4 & 5 Expected life cycle of the program The importance of the program in meeting the organization’s goals Cost of the program Visibility of the program The size of the target audience Extent of management interest

When Properly Implemented, High ROI Values can be Achieved with Programs on: Leadership Team Building Management Development Supervisor Training Sales Training . . . 100% to 700% ROI is not uncommon

ROI Best Practices The ROI methodology is implemented as a process improvement tool and not a performance evaluation tool for the learning/development staff. ROI impact studies are conducted very selectively, usually involving 5-10% of programs. A variety of data collection methods are used in ROI analysis. For specific ROI evaluations, the effects of learning/development are isolated from other influences. Business impact data are converted to monetary values. and . . .

ROI Best Practices 6. ROI evaluation targets are developed, showing the percent of programs evaluated at each level. 7. The ROI methodology generates a micro level scorecard. ROI methodology data are being integrated to create a macro scorecard for the learning/development function. The ROI methodology is being implemented for about 3-5% of the learning/development budget. ROI forecasting is being implemented routinely The ROI methodology is used as a tool to strengthen/improve the learning/education process.

Time/Cost Savings Tips Plan for Evaluation –Early Build in Evaluation Tools Share Responsibilities Communicate Expectations Use Standard Tools and Templates

Time/Cost Saving Tips Use Estimates Use Shortcuts Methods Use Sampling, Routinely Streamline Reporting Develop Internal Capability Use Technology

Healthcare, Inc. Sexual Harassment Prevention Workshop

Why be Concerned with Communication? Measurement and evaluation are meaningless without communication Communication is necessary for making improvements Communication is a sensitive issue Different audiences need different information

Communication Principles Keep communication timely Target communication to specific audiences Carefully select communication media Keep communication consistent with past practices Incorporate testimonials from influential individuals Consider the training function’s reputation when developing the overall strategy

There are Four Types of Reports Complete Report Executive Summary General Audience Summary Streamlined Report

The Complete Report Includes the Details General Information Methodology for Impact Study Data Analysis Costs Results Barriers and Enablers Conclusions and Recommendations Exhibits

Sample Table of Contents of an ROI Impact Study Sample Table of Contents For an Executive Summary

Sprint/Nextel Program Title: Diversity Target Group: Managers and Employees Solution: All-Inclusive Workforce Program (AIW)

Sprint/Nextel Level 1: Reaction Composite Rating: 4.39 out of 5 (for six items) Level 2: Learning Averaged 4.28 out of 5 (for learning on six objectives)

Sprint/Nextel Level 3: Application Managers: Supports AIW (87%) Addresses Problems (81%) Encourages Staff (78%) Employees: Supports AIW (65%) Identifies Differences (63%) Encourages Staff (60%) 91% of Managers successful completed action plans

Sprint/Nextel Level 4: Impact Attrition Rate Improvement = 9.77% Level 5: ROI BCR: 2.6 ROI: 163% Intangible Benefits Employee Satisfaction Communication Cooperation Diversity Mix Teamwork

Sprint/Nextel Technique to Isolate Effects of Program: Manager’s estimate, adjusted for error Technique to Convert Data to Monetary Value: Standard cost item ($89,000 per Turnover) Fully-loaded Program Costs: $1,216,836

Building an ROI Scorecard Provides macro-level perspective of success Serves as a brief report versus detailed study Shows connection of training’s contribution to business objectives Integrates various types of data Demonstrates alignment between programs, strategic objectives, and operating goals

The Scorecard Includes Seven Types of Data Indicators / Scope / Volume Level 1 – Reaction / Satisfaction Level 2 – Learning Level 3 – Application / Barriers / Enablers Level 4 - Business Impact Level 5 - ROI Intangibles

Reporting for a Corporate University

What Barriers will Prevent you From Implementing ROI?

ROI Quiz

Increasing Response Rates Provide advance communication Clearly communicate the reason for the questionnaire Indicate who will see the results Show how the data will be integrated Keep the questionnaire simple and brief Make it easy to respond Use the local manager to help distribute the questionnaires and show support Let the target audience know that they are part of a carefully selected sample Notes: Can you add to this list? ____________________________________ and . . .

Increasing Response Rates Use one or two follow-up reminders Have the introduction letter signed by a top executive Enclose a giveaway item with the questionnaire Provide an incentive for quick response Send a summary of results to target audience Distribute questionnaire to a captive audience Consider an alternative distribution channel Have a third party gather and analyze data. Notes: Can you add to this list? ____________________________________ and . . .

Increasing Response Rates Communicate the time limit Consider paying for the time it takes to complete the questionnaire Review the questionnaire at the end of the formal session Carefully select the survey sample Allow completion of the survey during work hours Add emotional appeal and . . .

Increasing Response Rates Design questionnaire to attract attention, with a professional format Let participants know what actions will be taken with the data Provide options to respond Use a local coordinator to help distribute and collect questionnaires Frame questions so participants can respond appropriately and make the questions relevant

Wisdom of Crowds In this case, the average estimate is near perfect Estimates are used everywhere Set up your own experiment Estimates should be adjusted Estimates are okay – defend them; don’t prefer them

Action Plan

Barriers to ROI include: Resources Funding Time People Support Managers Colleagues Participants Skills Systems Culture

Actions to Make ROI Work Adapt versus adopt Planning and discipline Establish goals and targets Assign responsibilities Improve needs analysis process Implement cost-savings approaches Communicate progress Develop staff skills ROI Network International ROI Networks Workshops CSTD ROI Network ROI Certification Competency groups Read books/case studies/articles Develop your own case study Teach others

Key Points to Remember The Basics: ROI is the ultimate measure of profitability of our projects, programs, and processes. Reporting ROI alone is insufficient. The ROI process develops a balanced set of measures representing a chain of impact. Not all programs should be evaluated to ROI.

Key Points to Remember The Barriers: Resources Support Skills Funding Time People Support Managers Colleagues Participants Skills

Key Points to Remember Solutions to Barriers: Adapt versus adopt Planning and discipline Improve needs analysis process Implement cost savings approaches Communicate progress Develop staff skills

Key Points to Remember The Benefits: Show the contribution of programs Earn respect of senior management Gain the confidence of clients Improve support for training and performance improvement Enhance training and performance improvement processes Identify inefficient programs that need to be redesigned Identify successful programs

What if you do nothing?

Sample of Published ROI Studies Performance Management Process Improvement Skill-Based Pay Sexual Harassment Prevention Safety Incentive Plan Diversity Retention Improvement Absenteeism Control/Reduction Program Stress Management Program Executive Leadership Development E-Learning Internal Graduate Degree Program Executive Coaching Competency Development First Level Leadership Development

Southeast Corridor Bank Program Title: Managing Retention Target Group: Bank Tellers Solution: Skills Based Pay System to Reduce Employee Turnover

Southeast Corridor Bank Level 1: Reaction Composite Rating: 4.2 out of 5 Level 2: Learning Positive self assessment on program understanding Positive self assessment on each course with few exceptions (only two failed to be promoted because of performance in training)

Southeast Corridor Bank Level 3: Application 95% participation rate 86 requests for training compared to 46 the year before 138 review situations 257 promotions compared to 139 the year before Level 4: Impact Turnover reduced from 71% to 35% Staffing level reduced by 4%

Southeast Corridor Bank Level 5: ROI BCR: 3.58 ROI: 258% Intangible Benefits Customer Satisfaction Job Satisfaction Product Sales Cross Selling

Southeast Corridor Bank Technique to Isolate Effects of Program: Estimates from branch managers and branch staff, adjusted for error Technique to Convert Data to Monetary Value: External studies at similar institutions and standard values (for staffing) Fully-loaded Program Costs: $857,196 First year; $433,200 Second year

Case Study Application

Retail Merchandise Company

Program Profile Title: Interactive Selling Skills Target Group: Sales Associates in Electronics Vendor Produced and Delivered 3 Days - (2 Days Plus 1 Day) Significant Use of Skill Practices 3 Groups Trained (48 Participants from 3 Stores)

ROI Analysis Profile Post Program Data Collection (4) Performance Monitoring 3 months (3) Questionnaire 3 months (3) Program Follow-up Session 3 weeks (last session) Isolating the Effects of Training Control Group Arrangement Participant’s Estimate (For Back-up) Converting Data to Monetary Values Profit Contribution of Increased Output

Level 1 - Selected Data Success with Objectives 4.3 Relevance of Material 4.4 Usefulness of Program 4.5 Exercises/Skill Practices 3.9 Overall Instructor Rating 4.1

All Participants Demonstrated Level 2 - Selected Data All Participants Demonstrated That They Could Use The Skills Successfully

Level 3 - Selected Data (2 Questions out of 20) I utilize the skills taught in the program Frequency of use of skills

Level 4 Data: Average Weekly Sales Post Training Data Weeks After Training Trained Groups Control Groups 1 $ 9,723 $ 9,698 2 9,978 9,720 3 10,424 9,812 13 $13,690 $11,572 14 11,491 9,683 15 11,044 10,092 Average for Weeks $12,075 $10,449 13, 14, 15

Annualized Program Benefits 46 participants were still in job after 3 months. Average Weekly Sales per Employee Trained Groups $12,075 Untrained Groups 10,449 Increase 1,626 Profit Contribution (2% of Store Sales) 32.50 Total Weekly Improvement (x 46) 1,495 Total Annual Benefits (x 48 Weeks) $71,760

Cost Summary 48 participants in 3 courses Facilitation Fees: 3 courses @ $3750 $11,250 Program Materials: 48 @ $35/participant 1,680 Meals/Refreshments: 4,032 3 days @ $28/participant Facilities: 9 days @ $120 1,080 Participant Salaries Plus Benefits (35% factor) 12,442 Coordination/Evaluation 2,500 Total Costs $ 32,984

Level 5 Data BCR = = ROI (%) = X 100 =

ROI Example: Retail Merchandise Company Tabulating Program Costs $32,984 Calculating the Return on Investment Converting Data to Monetary Value Collecting Post Program Data Isolating the Effects of the Program Follow-up Session Questionnaire Performance Monitoring Control Groups Participants’ Estimates Standard Values $71,760 118% Identifying Intangible Benefits

The ROI Process Takes A Balanced View by Measuring And Reporting: Reaction to program Learning and attitudes Application on the job Impact in work unit Impact on the customer The financial results Intangible benefits Nature and source of problems and opportunities