Sharing Lessons on Learning Metrics LINGOs Annual Meeting Sept , 2009 Steven Honeyman and Marie-Laure Curie Global Capacity Building Department

Let’s start with small group discussions Briefly describe your learning metrics (overview only) Discuss advantages and disadvantages of different components Summarize what the group thinks are some new or interesting ideas the whole group should know about LINGOs Annual Meeting - Sept , 2009

Kirkpatrick’s model Includes four levels/steps of outcome evaluation: Level 1—Reaction – satisfaction survey Level 2—Learning – tests Level 3—Behavior – on-the job behavior change survey Level 4—Results – business outcomes LINGOs Annual Meeting - Sept , 2009

Measuring Training Impact at PSI SMILE - Social Marketing Initiative for Learning and Excellence

Introduction to the PSI Behavior Change Framework Behavior change framework drives PSI marketing activities – Fundamental to social marketing Applicable to all behaviors promoted by PSI Based on behavior change theory Determinants measured through TRaC (Tracking Results Continuously) survey LINGOs Annual Meeting - Sept , 2009

PRODUCTPRICEPLACEPROMOTION SOCIAL MARKETING INTERVENTION OPPORTUNITY ABILITYMOTIVATION POPULATION CHARACTERISTICS AT RISK POPULATIONS RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE IMPROVED HEALTH STATUS/QUALITY OF LIFE Availability Brand Attributes Locus of control Willingness to pay Outcome expectation Belief Threat Attitudes Intentions Subjective norm Self Efficacy Social Support Knowledge Social Norm Quality of Care Brand Appeal The overall framework LINGOs Annual Meeting - Sept , 2009

PRODUCTPRICEPLACEPROMOTION SOCIAL MARKETING INTERVENTION OPPORTUNITY ABILITYMOTIVATION POPULATION CHARACTERISTICS AT RISK POPULATIONS RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE IMPROVED HEALTH STATUS/QUALITY OF LIFE Availability Brand Attributes Locus of control Willingness to pay Outcome expectation Belief Threat Attitudes Intentions Subjective norm Self Efficacy Social Support Knowledge Social Norm Quality of Care Brand Appeal What PSI does… Our sphere of influence LINGOs Annual Meeting - Sept , 2009

PRODUCTPRICEPLACEPROMOTION SOCIAL MARKETING INTERVENTION OPPORTUNITY ABILITYMOTIVATION POPULATION CHARACTERISTICS AT RISK POPULATIONS RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE IMPROVED HEALTH STATUS/QUALITY OF LIFE Availability Brand Attributes Locus of control Willingness to pay Outcome expectation Belief Threat Attitudes Intentions Subjective norm Self Efficacy Social Support Knowledge Social Norm Quality of Care Brand Appeal What we try to influence… LINGOs Annual Meeting - Sept , 2009

PRODUCTPRICEPLACEPROMOTION SOCIAL MARKETING INTERVENTION OPPORTUNITY ABILITYMOTIVATION POPULATION CHARACTERISTICS AT RISK POPULATIONS RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE IMPROVED HEALTH STATUS/QUALITY OF LIFE Availability Brand Attributes Locus of control Willingness to pay Outcome expectation Belief Threat Attitudes Intentions Subjective norm Self Efficacy Social Support Knowledge Social Norm Quality of Care Brand Appeal To change a behavior... LINGOs Annual Meeting - Sept , 2009

PRODUCTPRICEPLACEPROMOTION SOCIAL MARKETING INTERVENTION OPPORTUNITY ABILITYMOTIVATION POPULATION CHARACTERISTICS AT RISK POPULATIONS RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE IMPROVED HEALTH STATUS/QUALITY OF LIFE Availability Brand Attributes Locus of control Willingness to pay Outcome expectation Belief Threat Attitudes Intentions Subjective norm Self Efficacy Social Support Knowledge Social Norm Quality of Care Brand Appeal That improves health LINGOs Annual Meeting - Sept , 2009

OAM Opportunity: institutional or structural factors that influence an individual’s chance to perform a promoted behavior. Ability: an individual’s skills or proficiencies needed to perform a promoted behavior. Motivation: an individual’s desire to perform a promoted behavior. LINGOs Annual Meeting - Sept , 2009

What is it? –Repetitive quantitative survey that measures behavior change of a specific behavior of a specific target group –Measure the effectiveness of project interventions and activities Why do we use it? –To better design programs, to monitor program progress (ensure that the target population is adopting the promoted behavior and if needed, make adjustments) and evaluate how we did (target group and behavior) –Informs decision-maker where to invest more/fewer resources Who does it? –All PSI programs When does it occur? - Prior, during (sometimes often) and at the end of an intervention TRaC (Tracking Results Continuously)) LINGOs Annual Meeting - Sept , 2009

CAPACITY BUILDING INTERVENTION OPPORTUNITY ABILITYMOTIVATION POPULATION CHARACTERISTICS BY PSI STAFF USE OF SKILLS ACQUIRED AT THE TRAINING IMPROVED JOB PERFORMANCE AND OVERALL IMPACT OF PSI PROGRAMS Availability Brand Attributes Locus of control Willingness to pay Outcome expectation Belief Threat Attitudes Intentions Subjective norm Self Efficacy Social Support Knowledge Social Norm Quality of Care Brand Appeal The overall Capacity Building framework LINGOs Annual Meeting - Sept , 2009

What is it? –Short, quantitative survey –Measures application of training on-the-job Why do we use it? –Programmatic check to ensure that the training had the intended impact of transferring needed skills/behaviors to staff and to identify gaps in training content/delivery –Informs CB decision-maker where to invest more/fewer resources Who does it? –Global Capacity Building department (for virtual training as well) When does it occur? –Approx. 6 to 8 weeks after the training ended TRaC-T (Tracking Results Continuously for Training)) LINGOs Annual Meeting - Sept , 2009

Kirkpatrick’s model Includes four levels/steps of outcome evaluation: Level 1—Reaction – satisfaction survey Level 2—Learning – tests Level 3—Behavior – TRaC-T survey Level 4—Results – business outcomes LINGOs Annual Meeting - Sept , 2009

TRaC-T survey (example) LINGOs Annual Meeting - Sept , 2009

Some TRaC-T results TRaC-T Budget Review training - August 5 & LINGOs Annual Meeting - Sept , 2009

Some TRaC-T results

TRaC-T might have some limitations: Self reported behavior, Needs to be built into the training program, Needs resources to develop and manage, May not apply learning right away. But it has many potential benefits: Demonstrates training effectiveness to senior managers, Convinces supervisor of the value of a particular training, Allows targeting of training resources to highly effective training, Provides feedback to those leading training on whether they are being effective, and Can be automatically set-up for online training ! LINGOs Annual Meeting - Sept , 2009

For more information: Marie-Laure Curie, Manager, GCB Dept., Steven Honeyman, Director, GCB Dept. LINGOs Annual Meeting - Sept , 2009