Kirkpatrick model of evaluation to assess public health workforce preparedness training American Public Health Association November 5, 2013 Boston, MA Geraldine Aglipay, BS and Karen Peters, DrPH Illinois Preparedness and Emergency Response Learning Center MidAmerica Center for Public Health Practice University of Illinois at Chicago, School of Public Health

Presenter Disclosures (1)The following personal financial relationships with commercial interests relevant to this presentation existed during the past 12 months: Karen E. Peters “No relationships to disclose”

Overview: IPERLC 1 of 14 Preparedness and Emergency Response Learning Centers (PERLC) Funded by CDC’s Office of Public Health Preparedness and Response Provides education, training and technical assistance services to enhance capacity/performance of IL public health workforce regarding preparedness and emergency response

Overview: Mini-Grants for Emergency Preparedness To increase emergency response capability and capacity in Illinois – To support short-term, targeted preparedness related activities for which there was a need but no discretionary resources available – To gauge the efficacy of individual training on organizations, their staff and partners (e.g., fire depts, LTC facilities)

Overview: Mini-Grants for Emergency Preparedness Grant awards: Up to $4,500 for 6 month projects (March 1-August 30, 2012) Eligibility/Awardees (n=13): – Local health departments [n = 10] – Primary care clinics (FQHCs or Stand alone) [n=2] – CBOs (NFPs, Schools, FBOs) [n=1]

Overview: Mini-Grants for Emergency Preparedness Project descriptions: – Conduct training activities; Run exercises; Develop response plans; Conduct needs assessments; Create resources; Build support networks Audiences and Partners (at risk): – Disabled persons, rural, seniors, economically disadvantaged, isolated, infants/children, limited/non-english speaking persons

Overview: Evaluation using Kirkpatrick Model Adopted by CDC for PERLC evaluation, organized as 4 levels: – Level (1) Trainees' reaction – Level (2) Knowledge acquired, skills improved, or attitudes changed – Level (3) Practice behavior changed (field) – Level (4) Results, or impact (at agency/organizational level)

The Issue Little literature on the intermediate and long term impacts of preparedness and emergency response training on agency capacity to respond to emergencies Focus is generally on individual KSA capacity (knowledge, skills, attitudes)

The Level 3 & 4 Grantees IPERLC selected 3 grantee LHDs based on training content and stakeholders (LTCF) for participation in Level 3 & 4: – 1 rural, 1 suburban, 1 urban – All 3 sites completed Level 1 & 2 – Level 3 evaluations (6,12 months) at 2 sites – Level 4 evaluations (at 12 months) at all 3 sites

Results: Level 3 Self report increased strength and confidence in ability to apply training lessons to field experience Work policy changes help with work preparedness Many semi- or fully retired, so future training delivery/content needs to be considered for their needs [attrition]

Results: Level 3 “additional training on policies and procedures needed”; “more exercises”; “trained people more willing to assist with other activities as they realize that preparedness is about team work”

Results Level 4 Partnerships have been strengthened [LTCF, special needs populations (4.6/5.0) Training outcomes meet organization’s needs/responsibilities for preparedness (4.0/5.0)

Results Level 4 New resources brought into the partnership 2 of 3 report new policies developed In absence of minigrant training, not able to accomplish same outcomes

Results Level 4 ‘Continued relationship building with special needs and LTC facilities…not historically involved with preparedness’ ‘We grow and expand relationships with partners and staff are more aware of their responsibilities’ ‘A foundation to prepare for full scale evacuation all LTCF participated and better able/quite confident to respond to emergencies’

Discussion Level 3 and 4 labor intensive Can do Level 4 (retrospectively) without other levels to see organizational changes but don’t have resources to conduct all 4 Kirkpatrick levels but… Requires key informant interviews, document policy reviews not only with primary public health agencies, but their top 1-3 key partners

Conclusions Evaluation of (preparedness) training using Kirkpatrick methodology provides guidance but is labor intensive to implement Instability of funding for preparedness training and personnel changes at local level make comprehensive evaluation difficult

Contact Information Victoria Wiebel, Assistant Director Illinois Preparedness and Emergency Response Learning Center

Thank you!