1. AETC National Evaluation Center
Kevin Khamarko, MA
Assistant Director, AETC NEC
AIDS Policy Research Center

2. Outline The AETC network and their training levels
Services provided by the NEC
The NEC’s new focus on how people use what they learn and cross-region evaluation
Overview of current NEC evaluation projects, including:
Application Focus:
Barriers and Facilitators Evaluation Project
Minority AIDS Initiative Chart Review Evaluation Project
Cross-site Focus:
Cross-region Training Evaluation Project
Behavior Change Evaluation Project
Cross-region HIV Testing Evaluation Project

3. Overview of the AETCs Federally funded via HRSA/HAB
Provide clinical HIV training and education to medical professionals
National network of 11 Regional Centers and 4 National Centers
Primary audience:
Physicians
Nurses, Nurse Practitioners
Physician assistants
Dentists
Clinical pharmacists
Other ancillary health care staff

4. AETC Program

5. AETC Levels of Training
Didactic presentations, panel discussions, self-instructional materials, and teleconferences
Level 2:
Interactive learning through discussion of cases supplied by trainer, role play, simulated patients, and other skill building activities
Level 3:
Preceptorships, mini-residencies, and observations of clinical care
Level 4:
Clinical consultation (distance-based and on-site)
Level 5:
Technical assistance and capacity building

6. Why Evaluate Trainings?
To determine whether the training program is achieving the identified objectives
To provide information to trainers for improving course design, content and delivery
To determine whether participants acquire the intended knowledge and skills and are able to apply them to the job

7. AETC National Evaluation Center
Serves as resource to regional AETC sites
Evaluation training: web-based modular curriculum
Evaluation community building
Evaluation planning & design
IRB (human subjects) training and TA
Implementation
Linking process data with outcome-oriented data
Analysis
Dissemination

8. NEC Goals…
To provide leadership in the development, design, testing and dissemination of effective evaluation models with emphasis on outcomes
Determine the outcomes of AETC clinical education and training programs with respect to changes in provider behavior and clinical practice

9. Kirkpatrick’s Model (Kirkpatrick, D. L
Kirkpatrick’s Model (Kirkpatrick, D.L., Evaluating Training Programs, 2nd edition, 1998)
Training Level:
Key Evaluation Question:
Level 1: Reaction
How do participants react to the training?
Level 2: Learning
To what extent do participants change attitudes, improve knowledge and/or increase skill as a result of the training?
Level 3: Behavior
To what extent do changes in behavior occur because of participation in the training?
Level 4: Results
What are the final results (eg. patient perception of care or outcomes of care) that occur because of participation in the training?
Kirkpatrick’s model was developed to provide framework for evaluating training programs
Consists of four levels
Each successive evaluation level is built on information provided by the lower level
Each successive level represents a more precise measure of the effectiveness of the training program, but at the same time requires more rigorous and time-consuming analysis

10. Training Evaluation Framework:
Before Training
During Training
After Training
Pre-Training
Conditions:
Individual
Characteristics,
Training Motivation,
Pre-training Environment
Training Process Evaluation:
Number and Type of
Activities,
Trainee Reaction
Training Outcome Evaluation:
Learning,
Provider Behavior Change
Training Impact Evaluation:
Results (Patient Outcomes)
UCSF NEC 2
Columbia NEC
UCSF NEC 1
UCSF NEC 2

11. Current AETC NEC Evaluation Projects
Application Focus:
Barriers and Facilitators Evaluation Project
Minority AIDS Initiative Chart Review Evaluation Project
Cross-site Focus:
Cross-region HIV Testing Evaluation Project
Cross-region Training Evaluation Project
Behavior Change Evaluation Project

12. Barriers and Facilitators Evaluation Project
Purpose:
To identify whether and how trainers/faculty address issues in overcoming barriers to applying the information learned, so trainees can actually make behavior changes in their clinical practice settings
To utilize the information gained to develop case studies and pre-training quantitative questions to evaluate the outcomes of training on provider practice and behavior change
To link trainees’ responses to the pre-training quantitative questions with follow-up questions, which will aid the AETCs in better understanding issues in application

13. Barriers and Facilitators Evaluation Project
Qualitative interviews with AETC regional directors and evaluators
Participants are asked about the organizational structure of their AETC and what kinds of things affect whether their trainees can apply what they’ve learned to their setting
Qualitative interviews with AETC faculty and training coordinators
Participants are asked broadly about whether their trainees had made changes following training and what barriers and facilitators they experienced during that process

14. Barriers and Facilitators Evaluation Project
Project Status:
The NEC began interviewing regional directors and evaluators in April 2008
To date, we’ve interviewed 4 regional directors and 5 regional evaluators
Preliminary findings include:
Facilitator: long-term relationships with trainees and trainee environments facilitate uptake of training
Barriers thus far relate to conducting trainings:
Geographical challenges (e.g., rural areas)
Recruiting trainees can be difficult
In general, regional AETCs are interested in conducting evaluations of training application

15. Minority AIDS Initiative Chart Review Evaluation Project
Purpose:
Cross-region evaluation facilitated by the NEC to measure changes in clinical practice and patient-related clinical indicators associated with improved quality and reduced disparities in care resulting from MAI programs
Project assesses the impact of MAI-funded clinical training on the quality of HIV clinical care using information abstracted from patients’ medical charts

16. Minority AIDS Initiative Chart Review Evaluation Project
AETC Region
Number of Clinics
Florida/Caribbean
12 Clinics
Mountain-Plains
1 Clinic
Northwest
2 Clinics
Pacific
5 Clinics
Southeast
National Minority

17. Minority AIDS Initiative Chart Review Evaluation Project
Methods:
Data from the baseline chart abstraction on 30 patients in each clinic will be used to assess adherence to clinical practice guidelines.
Chart abstraction team provides oral and/or written summary of findings to local clinical and/or AETC trainers.
Local AETC trainers may choose to target training based on feedback.

18. Minority AIDS Initiative Chart Review Evaluation Project
Methods:
1-year later, after implementation of MAI-funded clinical training, return to assess change in targeted aspects of clinical care at each clinic.
We anticipate information from 30 patients in each clinic (N=690) to be abstracted at both baseline and follow-up.
We anticipate a 10% change in the degree to which aspects of clinical care targeted by MAI-funded clinical training in each clinic are consistent with clinical practice guidelines.

19. Minority AIDS Initiative Chart Review Evaluation Project
Project Status:
The NEC’s nurse team leader has been hired and trained by staff at the Florida/Caribbean AETC in the administration of their chart review protocol
Baseline data collection has been completed in 14 clinics; we anticipate completion of baseline data collection by the end of June 2008
Follow-up data collection has been completed in 9 clinics
Retrospective data collection for both baseline and follow-up will be employed in 3 clinics; this abstraction will be complete by August 2008

20. Minority AIDS Initiative Chart Review Evaluation Project
Preliminary findings include:
Baseline chart abstraction has served as a valuable needs assessment tool
Local AETC trainers have been receptive to feedback from chart abstraction and have used information to target future trainings
We have identified many individual, organizational, structural and social barriers which impede the provision of primary HIV care

21. Linking the Two Evaluation Projects
Because of the time spent within a clinic, NEC evaluators are uniquely situated to understand potential barriers (individual, organizational, structural and/or social) providers may face in attempting to apply what they learned during a training
Based on our 3-day experience in the clinics, the NEC can provide important insight into previously unforeseen obstacles that may impede the uptake of training information and help to inform and tailor local trainings to best meet the needs of clinics

22. Cross-region Training Evaluation Project
National project involving all regional AETCs and the National Minority AETC
3 quantitative questions selected for implementation in Level 1-3 training programs
Questions will be asked immediately after the training program
Data from the evaluation questions will be sent to the AETC NEC for cross-site analysis
More questions will be added in the near future

23. Cross-region Training Evaluation Project
Questions:
How would you rate your level of knowledge about this content?
Likert-type scale with responses ranging from Novice (1) to Expert (5)
How would you rate the overall quality of the program?
Likert-type scale with responses ranging from Poor (1) to Excellent (5)
I can apply the information learned in my practice/service setting.
Likert-type scale with responses ranging from Disagree Strongly (1) to Agree Strongly (5)

24. Behavior Change Evaluation Project
Purpose:
To measure whether trainees demonstrate and apply in their practice the information learned (knowledge and skills) following the level 2 training attended
A set of open-ended questions, based on response logic, were created to better understand the effects of training across AETC regions
To utilize the information gained to translate the open-ended questions into quantitative questions for use across the AETC network
Enhancement of previous project, which focused on immediate post-training evaluation

25. Behavior Change Evaluation Project
Collaborating regions include:
Midwest AETC, Northwest AETC, Pacific AETC, Texas/Oklahoma AETC and the National Minority AETC
Methods:
Online evaluation project, requiring the collection of addresses
Participants will be contacted 6-8 weeks following the training

26. Behavior Change Evaluation Project
Trainees able to apply the information learned are asked the following questions:
Provide an example or examples of how you have been able to apply the information learned during the blank training.
What helped facilitate your ability to apply the information learned?
What obstacles did you face in applying the information and how did you address these obstacles?

27. Behavior Change Evaluation Project
Trainees not able to apply the information learned are asked:
Why were you not able to apply what you had learned at blank training?
Trainees not yet able to apply the information learned are asked:
If you plan to apply what you learned, describe how you will do so?

28. Cross-region HIV Testing Evaluation Project
National project involving all regional AETCs and the National Minority AETC
Purpose: to assess changes in the ability to provide HIV screening in clinics receiving enhanced capacity building assistance as part of the AETC HIV Training Initiative
The evaluation for this project is still in the planning phase

29. Benefits of Cross-region Evaluation
Allows the AETC network to present their successes and/or challenges in a collective manner
Provides larger sample size for data analysis
Also benefits our funders, as they are able to report the accomplishments across the AETC network and advocate for us based on our findings

30. Lessons Learned
Easier to design a cross-region evaluation for the HIV testing initiative since most regions do not have an evaluation program in place
Developing quantitative questions for 12 sites can be difficult, as the context of each site is different, so requires different response categories, etc.
Regions are doing an excellent job of evaluating their programs and many of their questions can be tailored by other regions for evaluation purposes

31. Questions? Email: kevin.khamarko@ucsf.edu