Evaluating Return on Investment European Event ROI Institute Dr. Elling Hamso Managing Partner European Event ROI Institute

ROI Surgery St James Room (next to Fairmont Empress breakfast room) Monday 17:15 – 18:15 Tuesday 08:00 – 09:00 10:15 – 10:45 (coffee break) 15:00 – 18:00 Wednesday 0800 – 1300

The Phillips ROI Methodology Refined over a 20-year period Thousands of ROI studies conducted every year 100 case studies published 3 000 individuals certified 50 books written to support the model Adopted by hundreds of organizations in more than 40 countries Key Concept: What are the elements of the benefit/cost ratio, and how does it differ from the ROI calculation. Instructor notes: The ROI calculation, which is the way senior level executives calculate return on investment, takes the NET MEETING BENEFITS (the benefits minus the costs), divides by the meeting costs, and multiplies by 100 to achieve a percentage value.  Let’s look at an example.

Some Users of the Methodology AT&T Coca Cola General Motors Hewlett Packard Hilton Hotels Motorola PricewaterhouseCoopers Shell Oil Singapore Airlines US Department of Defense NASA Government of Italy Government Canada Key Concept: What are the elements of the benefit/cost ratio, and how does it differ from the ROI calculation. Instructor notes: The ROI calculation, which is the way senior level executives calculate return on investment, takes the NET MEETING BENEFITS (the benefits minus the costs), divides by the meeting costs, and multiplies by 100 to achieve a percentage value.  Let’s look at an example.

Some Areas of Application Management Training Sales Training Team Building E-Learning Coaching Executive Training Customer Binding Meetings and Events Organization Development Recruiting Strategies Safety & Health Programs Self-Directed Teams Compensation Strategies Technology Implementation Quality Six Sigma Wellness/Fitness Initiatives

A stakeholder is anyone who would say: Stakeholders A stakeholder is anyone who would say: ”It is important to me that this meeting is successful and meets my objectives”

The Purpose of Corporate Events To INFLUENCE* participants to DO something which adds VALUE to stakeholders at the lowest possible COST * or educate, empower, inspire, persuade

The ROI of Conference Attendance To LEARN something which enables me to DO something which provides VALUE to me or my company

The Event ROI Pyramid ROI Business Impact Application Learning Key Concept: Clear, detailed definitions of each level of data collection. This is an animated slide; bring in each level separately as you discuss. Ask participants to identify each level as a review activity before you show it. We will collect these six types of data as part of the data collection phase. Let’s not forget the intangibles. The intangibles are those measures that we purposely choose not to convert to monetary value. If we can credibly convert data to a monetary value, we’ll do it. But if we cannot, we leave it as an intangible because it’s still very important. Many of your executives buy into those intangibles. They appreciate those intangibles. And for the most part, executives have funded our meetings in the past on those intangibles. Notice the last line. If you’re taking your evaluation to the ROI level, you MUST include some technique to isolate the effects of your meeting from other influences. Let’s look at the Evaluation Levels. What’s our first level of measurement? Reaction and satisfaction with the meetings. We ask attendees for their reaction. We ask them if and what they plan to do differently as a result of the meeting. This is planned action. “Smile sheets” or on-site evaluation surveys are an example of this level of measurement. What’s our next level of measurement? 2. Learning. What did you learn? What are your takeaways? Did you change your perception or attitude about something? Tests or exams are an example of this level of measurement. So is an electronic audience-response system. Application and implementation…what did we do differently as a result of the meeting? What new skills, new tasks, new procedures, new policies have I implemented? What have I done differently? Performance records, interviews with attendees several weeks after an event are ways to measure this. What are the consequences of doing something different? That consequence is your business measure. That could be productivity, quality improvement, sample time improvement, employee retention, accidents, attitudes, customer service, sales, customer satisfaction. ROI, where we compare the monetary value of the consequence. The monetary benefits of the meeting measured against the total costs of the meeting. Here’s where we need to think of issues such as isolating the effects of our meeting and converting the data to monetary value. BA: Explain Sample based on current presentation Level 1 (Reaction, Satisfaction and Planned Action) is the usual smilesheet -> What did you LIKE / What do you PLAN to do? Level 2 (Learning) is e.g. if you will remember this 5 levels at the end of the presentation Level 3 (Application) is if in your next survey, you apply these 5 levels and calculate the ROI according to the methodology Level 4 (Impact) is if you get new clients, increased budget or a new carreer opportunity based on this knowledge Level 5 (ROI) will be the benefits of above impact calculated against the cost you had to come here… Satisfaction & Planned Actions

How will you use what you learned? No Probably Not Maybe Probably Yes Definitely Average Score Discuss with my colleagues 0% 37.5% 62.5% 4.6 Review risk management procedures in my organisation 50% 4.5 Conduct formal risk assessments before major events 12.5% 3.6 Review how we check on suppliers’ safety procedures 62.5 12.5 3.8 Meet with our insurance brokers to review current policies 2.9 Other (specify below) 0.0

Do you know the basic principles of how to assess risk? 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100 % Avg BEFORE the seminar 2 4 AFTER the seminar 6 87% Do you feel comfortable with leading a risk assessment exercise for a major event? 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100 % Avg BEFORE the seminar 2 4 31% AFTER the seminar Could you draw a matrix for classification fo different types of risk? 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100 % Avg BEFORE the seminar 6 2 14% AFTER the seminar 4 Could you list the four different risk management alternatives? 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100 % Avg BEFORE the seminar 8 2 13% AFTER the seminar 6 4 71%

Isolating the effects of the meeting Data Analysis Isolating the effects of the meeting The Problem: Other Influences Meeting Impact Total measured Impact The Solution: Control groups Forecasting methods Trend analysis Participants’ estimates Supervisors’ estimates Outside experts Key Concept: It is possible to isolate the effects of a meeting. Instructor Notes: If you recall your learner outcomes, performing an isolation isn’t one of them – this is a fairly involved process. However, it is important to recognize that it can be done. Control Groups: You could have one group that attends a meeting and one that doesn’t, and measure the changed behavior of each against the other. Trend Analysis: You could measure your meeting attendees’ performance against industry trends. Forecasting Methods: You could use past data to forecast the change you would expect to see from a meeting. End User/Performance Estimates: You can ask attendees to tell you the level of change in their behavior they attribute to the meeting, and their confidence in this estimate. Supervisor Estimate: You can ask supervisors to tell you the level of change in attendee behavior they attribute to the meeting, and their confidence in this estimate. Outside Experts: Can provide you with an estimate of the change caused soley by the meeting. Let’s use an example of sales training. Participants have gone through a sales training program that increases their prospecting skills. How would each of the following be impacted and measured? They are listed in the order of their credibility. Converting output to contribution: What percentage of post-training output over a set amount of time (eg: 3 months) is directly attributable to the prospecting skills learned in the training program? Converting employee’s time: How much sales time has been “saved” because of these new skills which can then be converted to a monetary amount based on the employees wages and benefits? Using historical costs: What is the average cost to produce a sale? This information is developed from cost statements within the organization. The next 4 are based on values assigned by “experts”. This, of course, is based on the credibility of the “experts”. Using internal and external experts:. That said, the internal Directors of Sales and/or Marketing can estimate the value of the newly developed prospecting skills. The trainer who delivered the training can be used as an external expert. Using participants’ estimates: What monetary does the participant place on their newly developed skills in prospecting? Using supervisors’ and managers’ estimates: Based on the supervisors’ and managers’ expertise, what value do they place on these skills? Using staff estimates: Survey the staff to determine a value on the skills learned in the training. © Copyright 2004 ROI Institute

Using control groups to isolate the effects of the meeting Data Analysis Using control groups to isolate the effects of the meeting Retail Merchandise Company – Interactive Selling Skills Programme Two groups of 3 stores matched on: Store size Store location (median houshold income in the area) Customer traffic levels Previous store perfomance Weekly sales 3 months after training ($) Trained group 12,075 Control group 10,449 Increase due to training 1,626 Key Concept: It is possible to isolate the effects of a meeting. Instructor Notes: If you recall your learner outcomes, performing an isolation isn’t one of them – this is a fairly involved process. However, it is important to recognize that it can be done. Control Groups: You could have one group that attends a meeting and one that doesn’t, and measure the changed behavior of each against the other. Trend Analysis: You could measure your meeting attendees’ performance against industry trends. Forecasting Methods: You could use past data to forecast the change you would expect to see from a meeting. End User/Performance Estimates: You can ask attendees to tell you the level of change in their behavior they attribute to the meeting, and their confidence in this estimate. Supervisor Estimate: You can ask supervisors to tell you the level of change in attendee behavior they attribute to the meeting, and their confidence in this estimate. Outside Experts: Can provide you with an estimate of the change caused soley by the meeting. Let’s use an example of sales training. Participants have gone through a sales training program that increases their prospecting skills. How would each of the following be impacted and measured? They are listed in the order of their credibility. Converting output to contribution: What percentage of post-training output over a set amount of time (eg: 3 months) is directly attributable to the prospecting skills learned in the training program? Converting employee’s time: How much sales time has been “saved” because of these new skills which can then be converted to a monetary amount based on the employees wages and benefits? Using historical costs: What is the average cost to produce a sale? This information is developed from cost statements within the organization. The next 4 are based on values assigned by “experts”. This, of course, is based on the credibility of the “experts”. Using internal and external experts:. That said, the internal Directors of Sales and/or Marketing can estimate the value of the newly developed prospecting skills. The trainer who delivered the training can be used as an external expert. Using participants’ estimates: What monetary does the participant place on their newly developed skills in prospecting? Using supervisors’ and managers’ estimates: Based on the supervisors’ and managers’ expertise, what value do they place on these skills? Using staff estimates: Survey the staff to determine a value on the skills learned in the training. (ROI was in this case calculated to be 118%) © Copyright 2004 ROI Institute

Confidence of estimate Adjusted influence factor Data Analysis Participants’ estimates of factors influencing sales increase following training seminar Elements influencing sales increase: Sales training seminar Advertizing campaign Incentive programme Product improvements Other factors Total Influence factor 9% 16% 7% 49% 19% 100% Confidence of estimate Adjusted influence factor 65% (i.e. 6 – 13%) 6% Key Concept: It is possible to isolate the effects of a meeting. Instructor Notes: If you recall your learner outcomes, performing an isolation isn’t one of them – this is a fairly involved process. However, it is important to recognize that it can be done. Control Groups: You could have one group that attends a meeting and one that doesn’t, and measure the changed behavior of each against the other. Trend Analysis: You could measure your meeting attendees’ performance against industry trends. Forecasting Methods: You could use past data to forecast the change you would expect to see from a meeting. End User/Performance Estimates: You can ask attendees to tell you the level of change in their behavior they attribute to the meeting, and their confidence in this estimate. Supervisor Estimate: You can ask supervisors to tell you the level of change in attendee behavior they attribute to the meeting, and their confidence in this estimate. Outside Experts: Can provide you with an estimate of the change caused soley by the meeting. Let’s use an example of sales training. Participants have gone through a sales training program that increases their prospecting skills. How would each of the following be impacted and measured? They are listed in the order of their credibility. Converting output to contribution: What percentage of post-training output over a set amount of time (eg: 3 months) is directly attributable to the prospecting skills learned in the training program? Converting employee’s time: How much sales time has been “saved” because of these new skills which can then be converted to a monetary amount based on the employees wages and benefits? Using historical costs: What is the average cost to produce a sale? This information is developed from cost statements within the organization. The next 4 are based on values assigned by “experts”. This, of course, is based on the credibility of the “experts”. Using internal and external experts:. That said, the internal Directors of Sales and/or Marketing can estimate the value of the newly developed prospecting skills. The trainer who delivered the training can be used as an external expert. Using participants’ estimates: What monetary does the participant place on their newly developed skills in prospecting? Using supervisors’ and managers’ estimates: Based on the supervisors’ and managers’ expertise, what value do they place on these skills? Using staff estimates: Survey the staff to determine a value on the skills learned in the training. ROI estimated to be120% (manufacturer) 320% (UK import agent)

ROI Handbook Objectives; issues; challenges Session/speakers; contacts;ideas; sales leads; notes Activities/ follow-up Results; successes; values/savings

Books and Resources Some useful websites: www.eventroi.org European Event ROI Institute www.roiinstitute.net ROI Institute, Inc. Books: “Proving the Value of Meetings & Events” Jack J. Phillips, Monica Myhill and James B. McDonough, available from www.eventroi.org ”Measuring the Return on Investment in Meetings and Events” Jack J. Phillips and Terri Brining, available from www.amazon.com “Return on Investment in Training and Performance Improvement Programs” Jack J. Phillips, available from major booksellers and www.amazon.com ”Show me the Money” Jack J. Phillips, available from major booksellers and www.amazon.com

ROI Courses in Europe 5-Day Event ROI Certification Course with Jack Phillips 31 August – 4 September 2009, Brussels, Belgium 1-Day Event ROI Courses London, Amsterdam, Brussels, Dublin, Frankfurt, Dusseldorf, Hamburg, Barcelona, Madrid, Milan, Turin, Copehagen, Stockholm, Helsinki, Warzawa, Tallin. Information and registration on www.eventroi.org

ROI Training & Consulting Services In-house Training Courses 1 – 3 days staff training Customer Training Courses 1-day course with event agency and customers ROI Impact Studies Evaluate the ROI of your event ROI Quickscan Review and report on the potential improvements in ROI for your event www.eventroi.org

Event Evaluation Awards Best Event Evaluation Awards Spain, Italy, Belgium, Sweden International Awards EIBTM – Best Event Awards EuBEA - European Best Event Award

Contact Details European Event ROI Institute Dr. Elling Hamso Managing Partner E-mail: elling.hamso@eventroi.org Telephone: +47 90 12 24 18