1. Evaluating Return on Investment European Event ROI Institute
Dr. Elling Hamso
Managing Partner
European Event ROI Institute

2. ROI Surgery St James Room (next to Fairmont Empress breakfast room)
Monday 17:15 – 18:15
Tuesday 08:00 – 09:00 10:15 – 10:45 (coffee break) 15:00 – 18:00
Wednesday – 1300

3. The Phillips ROI Methodology
Refined over a 20-year period
Thousands of ROI studies conducted every year
100 case studies published
3 000 individuals certified
50 books written to support the model
Adopted by hundreds of organizations in more than 40 countries
Key Concept: What are the elements of the benefit/cost ratio, and how does it differ from the ROI calculation.
Instructor notes:
The ROI calculation, which is the way senior level executives calculate return on investment, takes the NET MEETING BENEFITS (the benefits minus the costs), divides by the meeting costs, and multiplies by 100 to achieve a percentage value. 
Let’s look at an example.

4. Some Users of the Methodology
AT&T
Coca Cola
General Motors
Hewlett Packard
Hilton Hotels
Motorola
PricewaterhouseCoopers
Shell Oil
Singapore Airlines
US Department of Defense
NASA
Government of Italy
Government Canada
Key Concept: What are the elements of the benefit/cost ratio, and how does it differ from the ROI calculation.
Instructor notes:
The ROI calculation, which is the way senior level executives calculate return on investment, takes the NET MEETING BENEFITS (the benefits minus the costs), divides by the meeting costs, and multiplies by 100 to achieve a percentage value. 
Let’s look at an example.

5. Some Areas of Application
Management Training
Sales Training
Team Building
E-Learning
Coaching
Executive Training
Customer Binding
Meetings and Events
Organization Development
Recruiting Strategies
Safety & Health Programs
Self-Directed Teams
Compensation Strategies
Technology Implementation
Quality Six Sigma
Wellness/Fitness Initiatives

6. A stakeholder is anyone who would say:
Stakeholders
A stakeholder is anyone who would say:
”It is important to me that this meeting is successful and meets my objectives”

7. The Purpose of Corporate Events
To INFLUENCE* participants
to DO something
which adds VALUE to stakeholders
at the lowest possible COST
* or educate, empower, inspire, persuade

8. The ROI of Conference Attendance
To LEARN something
which enables me to DO something
which provides VALUE to me or my company

9. The Event ROI Pyramid ROI Business Impact Application Learning
Key Concept: Clear, detailed definitions of each level of data collection.
This is an animated slide; bring in each level separately as you discuss. Ask participants to identify each level as a review activity before you show it.
We will collect these six types of data as part of the data collection phase.
Let’s not forget the intangibles. The intangibles are those measures that we purposely choose not to convert to monetary value. If we can credibly convert data to a monetary value, we’ll do it. But if we cannot, we leave it as an intangible because it’s still very important.
Many of your executives buy into those intangibles. They appreciate those intangibles. And for the most part, executives have funded our meetings in the past on those intangibles.
Notice the last line. If you’re taking your evaluation to the ROI level, you MUST include some technique to isolate the effects of your meeting from other influences.
Let’s look at the Evaluation Levels.
What’s our first level of measurement?
Reaction and satisfaction with the meetings. We ask attendees for their reaction. We ask them if and what they plan to do differently as a result of the meeting. This is planned action. “Smile sheets” or on-site evaluation surveys are an example of this level of measurement.
What’s our next level of measurement?
2. Learning. What did you learn? What are your takeaways? Did you change your perception or attitude about something? Tests or exams are an example of this level of measurement. So is an electronic audience-response system.
Application and implementation…what did we do differently as a result of the meeting? What new skills, new tasks, new procedures, new policies have I implemented? What have I done differently? Performance records, interviews with attendees several weeks after an event are ways to measure this.
What are the consequences of doing something different? That consequence is your business measure. That could be productivity, quality improvement, sample time improvement, employee retention, accidents, attitudes, customer service, sales, customer satisfaction.
ROI, where we compare the monetary value of the consequence. The monetary benefits of the meeting measured against the total costs of the meeting. Here’s where we need to think of issues such as isolating the effects of our meeting and converting the data to monetary value BA:
Explain Sample based on current presentation
Level 1 (Reaction, Satisfaction and Planned Action) is the usual smilesheet -> What did you LIKE / What do you PLAN to do?
Level 2 (Learning) is e.g. if you will remember this 5 levels at the end of the presentation
Level 3 (Application) is if in your next survey, you apply these 5 levels and calculate the ROI according to the methodology
Level 4 (Impact) is if you get new clients, increased budget or a new carreer opportunity based on this knowledge
Level 5 (ROI) will be the benefits of above impact calculated against the cost you had to come here…
Satisfaction & Planned Actions

10. How will you use what you learned?No
Probably Not
Maybe
Probably Yes
Definitely
Average Score
Discuss with my colleagues 0%
37.5%
62.5%
4.6
Review risk management procedures in my organisation
50%
4.5
Conduct formal risk assessments before major events
12.5%
3.6
Review how we check on suppliers’ safety procedures
62.5
12.5
3.8
Meet with our insurance brokers to review current policies
2.9
Other (specify below)
0.0

11. Do you know the basic principles of how to assess risk?0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100 %
Avg
BEFORE the seminar 2 4
AFTER the seminar 6
87%
Do you feel comfortable with leading a risk assessment exercise for a major event? 0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100 %
Avg
BEFORE the seminar 2 4
31%
AFTER the seminar
Could you draw a matrix for classification fo different types of risk? 0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100 %
Avg
BEFORE the seminar 6 2
14%
AFTER the seminar 4
Could you list the four different risk management alternatives? 0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100 %
Avg
BEFORE the seminar 8 2
13%
AFTER the seminar 6 4
71%

12. Isolating the effects of the meeting
Data Analysis
Isolating the effects of the meeting
The Problem:
Other Influences
Meeting
Impact
Total measured Impact
The Solution:
Control groups
Forecasting methods
Trend analysis
Participants’ estimates
Supervisors’ estimates
Outside experts
Key Concept: It is possible to isolate the effects of a meeting.
Instructor Notes:
If you recall your learner outcomes, performing an isolation isn’t one of them – this is a fairly involved process. However, it is important to recognize that it can be done.
Control Groups: You could have one group that attends a meeting and one that doesn’t, and measure the changed behavior of each against the other.
Trend Analysis: You could measure your meeting attendees’ performance against industry trends.
Forecasting Methods: You could use past data to forecast the change you would expect to see from a meeting.
End User/Performance Estimates: You can ask attendees to tell you the level of change in their behavior they attribute to the meeting, and their confidence in this estimate.
Supervisor Estimate: You can ask supervisors to tell you the level of change in attendee behavior they attribute to the meeting, and their confidence in this estimate.
Outside Experts: Can provide you with an estimate of the change caused soley by the meeting.
Let’s use an example of sales training. Participants have gone through a sales training program that increases their prospecting skills. How would each of the following be impacted and measured? They are listed in the order of their credibility.
Converting output to contribution: What percentage of post-training output over a set amount of time (eg: 3 months) is directly attributable to the prospecting skills learned in the training program?
Converting employee’s time: How much sales time has been “saved” because of these new skills which can then be converted to a monetary amount based on the employees wages and benefits?
Using historical costs: What is the average cost to produce a sale? This information is developed from cost statements within the organization.
The next 4 are based on values assigned by “experts”. This, of course, is based on the credibility of the “experts”.
Using internal and external experts:. That said, the internal Directors of Sales and/or Marketing can estimate the value of the newly developed prospecting skills. The trainer who delivered the training can be used as an external expert.
Using participants’ estimates: What monetary does the participant place on their newly developed skills in prospecting?
Using supervisors’ and managers’ estimates: Based on the supervisors’ and managers’ expertise, what value do they place on these skills?
Using staff estimates: Survey the staff to determine a value on the skills learned in the training.
© Copyright 2004 ROI Institute

13. Using control groups to isolate the effects of the meeting
Data Analysis
Using control groups to isolate the effects of the meeting
Retail Merchandise Company – Interactive Selling Skills Programme
Two groups of 3 stores matched on:
Store size
Store location (median houshold income in the area)
Customer traffic levels
Previous store perfomance
Weekly sales 3 months after training ($)
Trained group 12,075 Control group 10,449
Increase due to training 1,626
Key Concept: It is possible to isolate the effects of a meeting.
Instructor Notes:
If you recall your learner outcomes, performing an isolation isn’t one of them – this is a fairly involved process. However, it is important to recognize that it can be done.
Control Groups: You could have one group that attends a meeting and one that doesn’t, and measure the changed behavior of each against the other.
Trend Analysis: You could measure your meeting attendees’ performance against industry trends.
Forecasting Methods: You could use past data to forecast the change you would expect to see from a meeting.
End User/Performance Estimates: You can ask attendees to tell you the level of change in their behavior they attribute to the meeting, and their confidence in this estimate.
Supervisor Estimate: You can ask supervisors to tell you the level of change in attendee behavior they attribute to the meeting, and their confidence in this estimate.
Outside Experts: Can provide you with an estimate of the change caused soley by the meeting.
Let’s use an example of sales training. Participants have gone through a sales training program that increases their prospecting skills. How would each of the following be impacted and measured? They are listed in the order of their credibility.
Converting output to contribution: What percentage of post-training output over a set amount of time (eg: 3 months) is directly attributable to the prospecting skills learned in the training program?
Converting employee’s time: How much sales time has been “saved” because of these new skills which can then be converted to a monetary amount based on the employees wages and benefits?
Using historical costs: What is the average cost to produce a sale? This information is developed from cost statements within the organization.
The next 4 are based on values assigned by “experts”. This, of course, is based on the credibility of the “experts”.
Using internal and external experts:. That said, the internal Directors of Sales and/or Marketing can estimate the value of the newly developed prospecting skills. The trainer who delivered the training can be used as an external expert.
Using participants’ estimates: What monetary does the participant place on their newly developed skills in prospecting?
Using supervisors’ and managers’ estimates: Based on the supervisors’ and managers’ expertise, what value do they place on these skills?
Using staff estimates: Survey the staff to determine a value on the skills learned in the training.
(ROI was in this case calculated to be 118%)
© Copyright 2004 ROI Institute

14. Confidence of estimate Adjusted influence factor
Data Analysis
Participants’ estimates of factors influencing sales increase following training seminar
Elements influencing sales increase:
Sales training seminar
Advertizing campaign
Incentive programme
Product improvements
Other factors
Total
Influence factor 9%
16% 7%
49%
19%
100%
Confidence of estimate
Adjusted influence factor
65% (i.e. 6 – 13%) 6%
Key Concept: It is possible to isolate the effects of a meeting.
Instructor Notes:
If you recall your learner outcomes, performing an isolation isn’t one of them – this is a fairly involved process. However, it is important to recognize that it can be done.
Control Groups: You could have one group that attends a meeting and one that doesn’t, and measure the changed behavior of each against the other.
Trend Analysis: You could measure your meeting attendees’ performance against industry trends.
Forecasting Methods: You could use past data to forecast the change you would expect to see from a meeting.
End User/Performance Estimates: You can ask attendees to tell you the level of change in their behavior they attribute to the meeting, and their confidence in this estimate.
Supervisor Estimate: You can ask supervisors to tell you the level of change in attendee behavior they attribute to the meeting, and their confidence in this estimate.
Outside Experts: Can provide you with an estimate of the change caused soley by the meeting.
Let’s use an example of sales training. Participants have gone through a sales training program that increases their prospecting skills. How would each of the following be impacted and measured? They are listed in the order of their credibility.
Converting output to contribution: What percentage of post-training output over a set amount of time (eg: 3 months) is directly attributable to the prospecting skills learned in the training program?
Converting employee’s time: How much sales time has been “saved” because of these new skills which can then be converted to a monetary amount based on the employees wages and benefits?
Using historical costs: What is the average cost to produce a sale? This information is developed from cost statements within the organization.
The next 4 are based on values assigned by “experts”. This, of course, is based on the credibility of the “experts”.
Using internal and external experts:. That said, the internal Directors of Sales and/or Marketing can estimate the value of the newly developed prospecting skills. The trainer who delivered the training can be used as an external expert.
Using participants’ estimates: What monetary does the participant place on their newly developed skills in prospecting?
Using supervisors’ and managers’ estimates: Based on the supervisors’ and managers’ expertise, what value do they place on these skills?
Using staff estimates: Survey the staff to determine a value on the skills learned in the training.
ROI estimated to be120% (manufacturer) 320% (UK import agent)

15. ROI Handbook Objectives; issues; challenges
Session/speakers; contacts;ideas; sales leads; notes
Activities/ follow-up
Results; successes; values/savings

17. Books and Resources Some useful websites:
European Event ROI Institute
ROI Institute, Inc.
Books:
“Proving the Value of Meetings & Events” Jack J. Phillips, Monica Myhill and James B. McDonough, available from
”Measuring the Return on Investment in Meetings and Events” Jack J. Phillips and Terri Brining, available from
“Return on Investment in Training and Performance Improvement Programs” Jack J. Phillips, available from major booksellers and
”Show me the Money” Jack J. Phillips, available from major booksellers and

18. ROI Courses in Europe
5-Day Event ROI Certification Course with Jack Phillips 31 August – 4 September 2009, Brussels, Belgium
1-Day Event ROI Courses London, Amsterdam, Brussels, Dublin, Frankfurt, Dusseldorf, Hamburg, Barcelona, Madrid, Milan, Turin, Copehagen, Stockholm, Helsinki, Warzawa, Tallin.
Information and registration on

19. ROI Training & Consulting Services
In-house Training Courses
1 – 3 days staff training
Customer Training Courses
1-day course with event agency and customers
ROI Impact Studies Evaluate the ROI of your event
ROI Quickscan Review and report on the potential improvements in ROI for your event

20. Event Evaluation Awards
Best Event Evaluation Awards
Spain, Italy, Belgium, Sweden
International Awards EIBTM – Best Event Awards
EuBEA - European Best Event Award

21. Contact Details European Event ROI Institute Dr. Elling Hamso
Managing Partner
Telephone: