1. Quality Assurance in Graduate Health and Medical Training Initiatives in Programs supervised by SCFHS
Amal AlAmoudi, PhD Candidate Health and Social policy
DG Quality Management, SCFHS

2. Contents 1 3 2 4 Success factors Overview of SCFHS & Time line
Context & Approach
GHM Quality Matrix 3
Success factors 2 4
9 October 2019

3. Overview of SCFHS & Time line1
9 October 2019

4. Overview
KSA is undergoing a significant transformation guided by the ambitious 2030 Vision.
The Saudi Commission for Health Specialties (SCFHS) is the country’s regulatory body for certification of all GHM since its establishment in 1992.
The GHM Training Quality Initiative provides a national framework for continuous quality improvement and its associated quality indicators for GHM training supervised by the SCFHS.
“Improve healthcare professional performance in the Kingdom to meet international standards.”
Realizing this and that no reform in healthcare can be complete without improving healthcare workforce training; the SCFHS has recently developed a far-reaching strategy with plans to
(1) Increase the training capacity of its graduate and medical programs to meet the increasing demands posed by the healthcare workforce gap,
(2)Ensure that with this capacity increase training quality will not be negatively impacted, and,
(3) Advance the standards of training to meet international standards and benchmarks.
SCFHS developed over a hundred Residency, Fellowship, and Diploma training programs.
Training for these programs is currently conducted in over a thousand sponsoring institutions across Saudi Arabia and the Gulf Region.
This growth of training programs and activities over the years, coupled with the anticipated significant expansion necessitates an ambitious plan for ensuring the quality of these training programs.
To achieve this, the SCFHS has launched its Training Quality Assurance Initiative, which aims to develop, implement, and advance a system for setting quality standards, benchmarks, measures, and improvement guidelines for its postgraduate training programs.
9 October 2019

5. GHM Training Quality development Time Line
June-2017 Formation of GHM committee
Feb-Dec 2018 GHM Quality Matrices induction
Workshops
Focus groups census meetings
Finalization
Rollout plan
May 2019 Recommended PI program initiatives identified
July- Dec 2019 Creation of Training Quality Standards and international bench mark
January 2018 GHM Quality Matrices Approval
March First cycle report generated
June 2019 GHM Training excellence Follow-up PI Report
Jan-2020 Introduction of GHM Quality Standards Nation wide

6. Context & Approach
9 October 2019

7. Context
Few countries have actually taken the initiative to pursue Quality measurement in competency-based postgraduate training.
In June 2017, the first quality indicators committee was formed in SCFHS to create the postgraduate quality indicators.
Objectives:
• To improve the quality of GHM training;
• To provide a means for subjective assessment of residency programs and training centers;
• To guide Training programs and training centers through a systematic performance improvement system; and
• To assist Program Directors in reviewing the conduct and educational quality of their programs.
Canada, France, Switzerland, the Netherlands and the UK. Saudi Arabia, through the SCFHS, will be among the few who have developed a system that measures the quality outcome of GHM training.
Collaborative efforts between two main divisions in SCFHS (Academic Affairs; Planning, Quality & Accreditation), with a committed team consisted of: Chief Academic Affairs, Chief , Executive Director of Training and Assistant, Executive Director of Assessment and Assistant, Director General of Quality Department, Director of Knowledge Management.
9 October 2019

8. Approach
The model was inspired by the Kirkpatrick Model to evaluate the effectiveness of the current training model.
The decision to evaluate training is mainly conducted to:
Decide whether to continue or discontinue training programs;
To gain information how to improve future training programs;
To justify the existence of training by showing how it contributes to the overall objectives and goals of the national healthcare system.
Fifth stage was developed (Training Governance)
Stage 1: Reaction: Measures how the trainees reacted to the training through measuring the following:
Were the participants pleased with the program.
Perception if they learned anything.
Likelihood of applying the content.
Effectiveness of particular strategy.
Effectiveness of the packaging of the training program.
Stage 2: Learning: Measures what the trainees have learned by looking to answer the following:
What exactly did the trainees learn or not learn?
The extent to which training increase knowledge or increase skill.
Stage 3: Behavior: Measures how the trainees have changed their behavior based on the training they have received, by looking at the following:
How does training affect performance?
The extent to which change in behavior occurred.
Was the learning transferred from the educational setting to the real world?
Stage 4: Results: Measures the results that occurred because of training.
Training Governance This fifth stage will allow for a comprehensive 360-effectiveness evaluation of the GHM training process and outcomes. Measures the training policies that are in place, training strategy, specifications and goals and objectives of the training program settings. This is effectively measured through carefully examining training curricula, rotation schedules, induction processes for trainers, matching process for trainees.
9 October 2019

9. Data Collection Methods
Qualitative subjective surveys that were distributed
Resident surveys
Program director surveys
Departmental operational KPIs
9 October 2019

10. GHM Quality Matrix 3 3

11. GHM Quality Matrix REACTION
Kirkpatrick Domain
KPI ID
KPI
Purpose
Data 2018
Trend
REACTION
GMHQI
% Residents satisfaction
The aim is to estimate the satisfaction of residents
83.1%
GMHQI
% Trainer satisfaction
The aim is to estimate the satisfaction of trainers.
71% -
GMHQI
% PDs satisfaction with center
The aim is to estimate the satisfaction of trainers
72%
GMHQI
% Residents burnout
The aim is to estimate the resident burnout rate
66%
GMHQI
% PDs satisfaction with SCFHS
The aim is to estimate the satisfaction of PD with SCFHS services.
82%
Five KPI were identified in the reaction domain of the Kirkpatrick model.
9 October 2019

12. Cont. GHM Quality Matrix
Kirkpatrick Domain
KPI ID
KPI
Purpose
Data in 2018
Trend
LEARNING
GMHQI
% Program Directors (PD) who attended PD Training Course offered by the SCFHS
The aim is to estimate the number of program directors who attended the SCFHS PD training course in a given year
38%
GMHQI
Number of Trainers in PGMT programs who successfully completed SCFHS Training certification
The aim is to estimate the number of PGMT trainers who completed the SCFHS training Certification course in a given year
238
GMHQI
% Surveyors who’ve successfully completed SCFHS’s Accreditation Training
The aim is to estimate the number pf surveyors who successfully the SCFHS Accreditation certification training course in a given year
47.75%
GMHQI
% Resident compliance to minimal procedure, case exposure policies, required competency index
The aim is to estimate the number of programs that complied with minimal case exposure policies required competency index in a given year
(Can not be measured for now)
GMHQI
% Residents who have received Residents Evaluation by program in a specific period
The aim is to estimate the number of residents who received their evaluation forms on time in a specific period.
67%
GMHQI
% Research included in curricula
The aim is to estimate the number of research projects that residents participated based on their curricula in a given year
50%
13 KPIs are in the learning domain of the Kirkpatrick model.
9 October 2019

13. GHM Quality Matrix Kirkpatrick Domain KPI ID KPI Purpose Data in 2018
Trend
GMHQI
% Burnout policy implementation
The aim is to estimate the policy implementation of burnout detection and prevention programs to all those who are involved in training in a given year
58.4%
GMHQI
% Compliance with implementing incorporated e-log system in each program
The aim is to estimate the compliance rate with implementing incorporated e-log system in each program in a given year
4.7%
LEARNING
GMHQI
% Residents who fulfilled their promotion criteria
The aim is to estimate the number of residents who fulfilled their promotion requirement criteria in a given year.
95.7%
GMHQI
% Residents who passed the board exam
The aim is to estimate the number of residents who passed their board exam in a given year
82.3%
GMHQI
% Disciplines who incorporated simulation in their curricula
The aim is to estimate the number of disciplines who incorporated simulation training in their curricula in a given year.
32%
GMHQI
% Residents who have received an annual master rotation plan
The aim is to estimate the number of residents who have received their annual rotation plan on time in a given year.
80%
GMHQI
% Adherence to the annual Master plan
The aim is to estimate the number of residents who complied with their annual master plan in a given year.
75%
9 October 2019

14. Cont. GHM Quality Matrix
Kirkpatrick
Domain
KPI ID
KPI
Purpose
Data in
2018
Trend
Training Governance
GMHQI
% Centers with complete goals and objectives for residency program
The aim is to estimate the number of training centers with complete goals and objectives for their residency programs
88%
GMHQI
% Completed trainer evaluation per program
The aim is to estimate the compliance of residents in filling out the trainer evaluation forms per rotation annually.
67%
GMHQI
% Adherence to accreditation requirements
The aim is to estimate the adherence rate of training institutions to accreditation requirements in a given year
96%
GMHQI
% PDs Turnover rate
The aim is to estimate the number of PD who did not complete their full term in a given year.
44%
GMHQI
% Accreditation complicane score
The aim is to estimate the compliance of training centers with the accreditation standards in a given year
To be reported:
Q4 2019
GMHQI
# violations with the matching regulations
The aim is to estimate the number of violations in the matching process in a given year
1.31%
6 KPIs are in the training governance domain.
9 October 2019

15. Success factors 4
9 October 2019

16. Training Quality Engagement1
Assess 2
Construct 2
The GHM training programs current status
Initial quality matrix results
comprehensive electronice data management system
Design GHM Training Performance management system
Design Performance improvement system
Design GHM training Quality standards that is internationally compatible 1 2 3 4 4
Post Implementation 3
Implement
Best Practices
Training Quality Monitoring and Reporting
Develop the GHM Training excellence award
GHM Training Quality Standards
PI Projects throughout the GHM training cycle annually
Annual GHM Training Quality Reporting Mechanism
Continues communication with all stakeholders
Knowledge Transfer Assurance
9 October 2019

17. Success Factors
Continuous support from the GHM Training Excellence Committee
Continuous communication with relevant stakeholders and monitoring PI project outcomes
Developing GHM Training Quality matrix electronic system that produces live dashboards for all programs
Developing GHM Training Quality Standards that ensures continues monitoring of training programs
9 October 2019

18. Thank you…