Repeat Trainees: What Are We Learning? Paul F. Cook, PhD MPAETC Evaluator University of Colorado at Denver & Health Sciences Center

HRSA’s Mission for AETCs Improve health outcomes of people living with HIV/AIDS through … Training on clinical management of HIV disease in such areas as: –use of ART and –prevention of HIV transmission Targeting providers who treat vulnerable populations

Types of Evaluation Process Number of Trainings (ERs) Number of Trainees (PIFs) Type of Trainees Type of Patients Served by Trainees Impact/Outcome Satisfaction with Training Increased Knowledge Change in Practice Behavior Better Patient Outcomes

The AETC Data Report (2007) Across all AETCs, 94,638 trainees filled out a PIF (80%). There were 60,648 unique individuals (i.e., 36% repeaters) 88% attended at Levels 1-3 “More intensive training likely contributes to … impact on clinical practice, however, … documentation of such outcomes is … beyond the scope of this [report].” (p.9)

Studying Repeat Trainees Aims of the Current Study #1 – Determine whether AETC trainees change their practice behavior over time #2 – Test the relationships between satisfaction, knowledge gain, intent to change, and actual practice behavior change #3 – Evaluate level of training as a predictor of outcome

Participants FY , first 3 quarters MPAETC trainees (one LPS not included – data problems) Attended 2+ events each Completed a PIF (with a valid PIF ID) and “Quality of Presentation” (QOP) evaluation form each time N = 402 PIFs;1,395 attendees; or about 10% of unique IDs

Sample Representativeness Population 70% Female Discipline: –22% MD/PA –9% Pharm –10% Dental –28% MH/Other 87% White 4% AI/AN 38% clinicians 19% see HIV+ pts (M = 29) Sample 74% Female Discipline: –33% MD/PA –16% Pharm –2% Dental –19% MH/Other 83% White ½% AI/AN 41% clinicians 38% see HIV+ pts (M = 55)

Procedure All MPAETC sites use QOP Levels 1-2: QOP form at end of each training event, linked via PIF ID Level 3: QOP items pretest/ posttest, same PIF ID All data stored in central Access database Query to identify repeaters Data analysis using HLM 6.0

Instrument – QOP adaptation of existing tool by Vojir et al. Subsections measure: –Trainee satisfaction (5 items) –Intent to change (1 item) –Pretraining knowledge (1 item, retrospective) –Posttraining knowledge (1 item) –Practices w/ HIV- pts (3 items) –Practices w/ HIV+ pts (4 items)

Instrument Psychometrics Satisfaction: alpha =.77; Knowledge: retest reliability (eta 2 ) =.97; subjective change correlates r =.32 with improvement on a test Practice Behavior items: retest reliability (r) = ; subscale alphas = Weak or moderate correlations across domains

Analysis Plan Hierarchical Linear Modeling to determine change within participants Key predictor = time –Initial coding: days since first event for that participant –Final coding: event number (starting from 1) for that participant, since 7/1/06

1. Training → Behavior Change T = 2.41, p =.016 Small r =.09 Significant for number of trainings, not for days Effect on total & HIV+; not HIV-

Bonus Finding: CDC Testing Recommendations released 9/22/06 on universal testing HLM for repeat trainees  total Bx, T = 2.77, p =.006 No effect on testing, but  risk assessment, prevention! No effect on any other behavior variable Caveat: only for repeaters

2. Other Outcomes Time → Δ Satisfaction, Knowledge or Intent Satisfaction → Δ Knowledge & Δ Intent Higher Pretest Knowledge → –  Posttest Knowledge, but –  Δ Knowledge and  Intent Δ Knowledge → Δ Intent Satisfaction, Intent, Knowledge → Δ Behavior

3. Effect of Training Level Whether a participant attended a Level 3 event had no effect on behavior Percent of all trainings attended that were Level 2 did have a significant effect: T = 2.79, p =.006 Some support for the idea that higher training levels produce better outcomes

What Changes HCP Behavior? Number of trainings led to behavior change –Mainly for HIV+ patient care More level 2 events led to behavior change –Mainly for HIV- patient care CDC recommendations led to behavior change –Mainly for risk assessment and prevention

Limitations/Future Directions Longer follow-up needed –Higher % repeaters possible –Greater change possible –Results need replication Repeaters may be unusual –Diverse groups, esp. AI/AN –More non-“HIV providers” Measure may show ceiling effects for expert trainees –Pilot separate “expert” items

Thank You If we knew what we were doing, it wouldn’t be called ‘research.’ —Albert Einstein