1. Oman Academic Accreditation Authority (OAAA)
Briefing on Institutional and Program Standards Assessment: Approach to decision-making Dr Salim Radhawi 11 March 2014

2. Briefing Outline Evolution of Institutional and Program Accreditation
OAAA
Evolution of Institutional and Program Accreditation
HEI and Program QA processes
Project aims and outcomes
Features of standards and assessment processes
Consultation process and feedback
Organisation of standards, ratings and outcomes
Public Reporting and HEI comparability
Fees for Standards Assessments

3. Evolution of Institutional and Program Accreditation (1)
OAAA
HEI and program accreditation are key parts of OAAA’s mandate (Royal Decree 54/2010)
First set of institutional and program standards were published in ROSQA in 2004
ROSQA standards used in 2004/2006 (two HEIs went through institutional and program accreditation).
A review of ROSQA was carried out in 2006; this was followed by the development of the Quality Plan

4. Evolution of Institutional and Program Accreditation (2)
OAAA
A new two-stage approach was proposed for HEI accreditation (Quality Audit and Standards Assessment)
The first Quality Audits were carried out in 2008
A separate process for program accreditation was proposed
The Institutional Standards Assessment project was originally launched in 2011
The current stage of the project, which includes both Institutional and Program Standards Assessments, commenced in 2013

5. HEI QA Processes OAAA HEI Licensure HEI Accreditation Stage 1 :
Quality
Audit
First cycle
commenced
2008
4 years
≤4 years
HEI Accreditation
Certificate
HEI Accreditation
Stage 2 :
Standards
Assessment
Appeal
Met
Met
Not met
1-2 years on
Probation
Process
Document
Start/End
KEY
HEI Standards
Reassessment
HEI
Accreditation
Terminated 5

6. Program QA Processes
OAAA

7. Projects aims and outcomes
OAAA
Projects aims and outcomes
To review and revise
HEI standards; program standards; and accreditation processes as set out in ROSQA to improve their relevance to the Omani context and to reflect current regional and international best practice (through benchmarking)
To produce
Institutional and program Conceptual Design Frameworks (CDFs) with clear approach to decision-making
a revised set of HEI standards and a revised set of generic standards for programs
Institutional and Program Standards Assessment Manuals which include guidelines for HEIs and external reviewers
Training workshops for the sector

8. Features of HEI and Program Accreditation
OAAA
Features of HEI and Program Accreditation
Recognition that responsibility for quality lies with HEIs
Based on ROSQA but incorporates criteria not included in ROSQA, e.g. academic integrity
Align with the nine areas of the scope of the QAM and reflect quality audit findings
Internationally benchmarked and reviewed
Acknowledge the diversity of HE provision in Oman
Encourage excellence – HEIs can receive Accredited with Merit overall or at standard(s) level
Encourages the use of ADRI
Developed through a consultative process

9. Features of the Institutional Standards Assessment process
OAAA
Features of the Institutional Standards Assessment process
The process looks at all areas of activity within an HEI and considers the quality systems applied to programs
National institutional accreditation is compulsory for all HEIs
Takes into consideration the HEI’s response to formal conclusions in Quality Audit Report
National schedule based on audit schedule with some flexibility

10. Features of the Program Standards Assessment process
OAAA
Features of the Program Standards Assessment process
The program is the unit of analysis but considers the impact of institutional-level activities
National program accreditation is compulsory for all HEI programs
Accreditation is for Omani programs as well as programs developed by overseas HEIs and/or with external accreditation
Program accreditation can be applied for once a cohort of students has graduated

11. Consultation Process
OAAA
Formation of a national Consultative Committee to facilitate comprehensive consultation with the HE sector
Formation of an external panel of experts to provide an international perspective
Working in partnership with MoHE
Dialogue with external stakeholders (including Education Council, MoM, MoH, MoD, and MoE)
Publication of drafts on OAAA website with discussion board
National Symposium October 2013 (to which CC, stakeholders, professional body, employer and student representatives were invited)

13. Response to Feedback: Organisation of the Standards
OAAA
Response to Feedback: Organisation of the Standards
A considerable amount of feedback highlighted the large number of elements to be assessed
There are nine standards
There are 75 criteria
Each criterion has a number of indicators
Indicators no longer need to be met for a criterion to be met – indicators are guidelines
Rating performance and commentary against indicators is no longer required
HEIs may provide other evidence to show that the requirements of a criterion has been met

14. Response to Feedback: Approach to Decision-making
OAAA
Response to Feedback: Approach to Decision-making
Significant feedback was provided on the rating of ‘not applicable’ (NA) with a numerical value (‘0’)
No value is now given to NA
Feedback suggested that the rating scheme was too subjective and difficult to apply
Clearer definitions have been provided for each criterion rating
Descriptions/characteristics will assist HEIs/reviewers in determining the most appropriate rating for each applicable criterion and standard

15. Positive Feedback It was real
OAAA
Positive Feedback
[The CDFs] are in many respects excellent, so congratulations to you all.
Prof Malcolm Cook, Former Pro-VC, University of Exeter, UK
In my view the Conceptual Design Framework is in very good shape.
Prof Ulrich Hommel, Director, EBS Business School, Germany
It was real
I think you should submit both the policies/standards and processes for international recognition.
Dr Mike Hillyard, Former President of the University of St Augustine, USA
It was a pleasure for me to join your national symposium.  I was really impressed with the approach OAAA has taken and the way participants got engaged in it.
Prof Badr Abu Ela, Executive Director, CAA, UAE

16. Previous and Current Approach to Organisation of the Standards
OAAA
Previous and Current Approach to Organisation of the Standards
Previous Approach
Current Approach
Indicators are essential components of each criterion
Indicators are guidelines for the type of evidence HEIs/programs might submit to demonstrate good provision or practice
Indicators need to be met in order to meet the criterion, and for the criterion to meet the standard
HEIs/programs can submit evidence based on other indicators to demonstrate how a criterion has been met
Each indicator is given a rating which informs the overall rating of the criterion
No rating is given to an indicator
Commentary is provided by HEIs/programs regarding how each indicator has been met
No commentary is required at indicator level

17. Comparison of Previous and Current Criterion Rating Scheme
OAAA
Comparison of Previous and Current Criterion Rating Scheme #
Previous Rating
Current Rating 4
Worthy of Merit
Excellent* 3
Good 2
Satisfactory 1
Unsatisfactory
Not met*
Not Applicable NA
* These terms were revised following further benchmarking and in order to improve clarity of understanding

18. ADRI as a Review Tool OAAA IMPROVEMENT APPROACH RESULTS DEPLOYMENT
Internal ADRI Review
Followed by External Review

19. Criterion Rating Definitions
OAAA
Criterion Rating Definitions #
Rating
Definition/characteristics 4
Excellent
Definition:
Provision or practice is consistently very high quality, and is underpinned by effective quality improvement arrangements.
Characteristics:
Provision or practice exceeds the requirements of the criterion.
Provision or practice is highly effective, and is undertaken consistently; it may be a model of good practice.
Provision or practice is exceptional rather than typical of other comparable HEIs/programs.
Provision or practice incorporates systematic and effective quality improvement arrangements.

20. Example of ‘Excellent’
OAAA
Example of ‘Excellent’
‘Excellent’ rating can be compared to ‘Commendation’ in Quality Audit Reports
A revised innovative academic advising system was successfully implemented and has been shown to have had a significant impact on retention and identifying ‘at risk’ students; the system has received significant positive feedback, been periodically reviewed for effectiveness, indicating continuous improvement
Rating against criterion 6.5

21. Criterion Rating Definitions
OAAA
Criterion Rating Definitions #
Rating
Definition/characteristics 3
Good
Definition:
Provision or practice is consistently high quality in most areas and is underpinned by effective quality improvement arrangements.
Characteristics:
Overall, provision or practice exceeds the requirements of the criterion.
Overall, provision or practice is high quality; is undertaken consistently; and meets the norms for good practice.
Provision or practice incorporates effective quality improvement arrangements.

22. OAAA
Example of ‘Good’
‘Good’ rating can be compared to positive text in Quality Audit Reports
As part of its continuous improvement system, the HEI has introduced a revised academic advising system in response to feedback; there is a comprehensive handbook and training for staff and students which has been consistently implemented; the system has had a positive impact; and the HEI has clear plans for how the system will be evaluated
Rating against criterion 6.5

23. Criterion Rating Definitions
OAAA
Criterion Rating Definitions #
Rating
Definition/characteristics 2
Satisfactory
Definition:
Provision or practice is effective most of the time, and is underpinned by adequate quality improvement arrangements.
Characteristics:
Provision or practice meets the requirements of the criterion, and is effective most of the time
Provision or practice is underpinned by adequate quality improvement arrangements, which are effective most of the time.

24. Example of ‘Satisfactory’
OAAA
Example of ‘Satisfactory’
‘Satisfactory’ rating means that the HEI/program has shown that it meets the requirements of the criterion
The HEI has implemented an effective formal academic advisory system which, overall, supports students in meeting their educational goals; the system has been evaluated and improvement plans have been implemented in most departments
Rating against criterion 6.5

25. Criterion Rating Definitions
OAAA
Criterion Rating Definitions #
Rating
Definition/characteristics 1
Not met
Definition:
Provision or practice does not meet the requirements of the criterion.
Characteristics:
Provision or practice lacks effective quality improvement arrangements NA
Not Applicable
An HEI will be expected to justify why the standard or criterion is not applicable to the institution/program.

26. OAAA
Example of ‘Not met’
‘Not Met’ rating could be compared to an Affirmation or Recommendation in Quality Audit Reports.
While the HEI has a policy for academic advising, the system has not been implemented effectively throughout the HEI and has not been monitored or evaluated for its effectiveness
Rating against criterion 6.5

27. Examples of ‘Not applicable’
OAAA
Examples of ‘Not applicable’
‘Not Applicable’ rating needs to be justified by the HEI/program
From XX HEI:
Criteria 3.1 – 3.6 do not apply to XX HEI as it is classified as a College and does not run Student Learning by Research Programs
(see SM001 HEI license approval)

28. Criteria ratings towards meeting the standard
OAAA
Criteria ratings towards meeting the standard
Description 
Standard Rating
Most* of the criteria are rated as ‘Excellent’; no criteria are rated as ‘Not Met’.
Excellent
Most of the criteria are rated as ‘Good’; no criteria are rated as ‘Not Met’.
Good
Most criteria are rated satisfactory; no criteria are rated ‘Not Met’.
Satisfactory
One or more criteria are rated ‘Not Met’
Not met
* ‘Most’ in this context means more than 50%

29. How Standards Ratings inform the Assessment Outcome
OAAA
How Standards Ratings inform the Assessment Outcome
Standard 
Assessment Outcome
Most of the standards are rated ‘Excellent’
Accredited with Merit
All standards are rated, as a minimum, ‘Satisfactory’, and one or more standards are rated as ‘Excellent’
Accredited with Merit in one or more standard(s)
All standards are rated either ‘Satisfactory’ or ‘Good
Accredited
One or more of the standards are rated as ‘Not Met’
On Probation
One or more of the standards are rated as ‘Not Met’ by the Standards Reassessment Panel
Not accredited

30. Standards Assessment Outcome: Outcome Deferred
OAAA
Standards Assessment Outcome: Outcome Deferred
Accreditation Outcomes
Description
Outcome Deferred
Where action(s) by the HEI is required to meet a standard (s), and where the Standards Assessment Panel determines that the HEI can implement the action(s) is a short time period (up to three months), the OAAA may grant the HEI a period of time to demonstrate that it has addressed these outstanding issues, and that it meets the standard. An accreditation outcome decision will not be made public during this time, and this stage will be considered part of the accreditation process. Once the OAAA is satisfied that the standard(s) has been fully met, the HEI/program will be accredited, and the outcome made public.
If the standard is not met within the designated timeframe, the HEI/program will be placed on probation, and the outcome made public.

31. Status of HEIs and Programs on OAAA website
Licensed
Audited (for HEIs only)
On probation
Accredited
Accredited with Merit
Accredited with Merit in one or more standards
Not accredited (after Standards Reassessment)

32. Public Reporting and Comparability
OAAA
Public Reporting and Comparability
Accreditation outcomes will be published on OAAA website, with ratings against each standard and criterion
A 4-point rating scale provides a transparent means for stakeholders to identify/compare how an HEI/program has performed
Stakeholders will be able to apply their own priorities in order to identify the HEI/program which meets their needs
This approach to public reporting avoids institutional/program ‘league tables’

33. Stakeholder Comparison of HEIs
OAAA
Stakeholder Comparison of HEIs
Overall score
HEIs

34. Stakeholder Comparison of Programs
OAAA
Stakeholder Comparison of Programs
Stakeholders input their own weightage for every criterion
Overall score
Programs

35. Fees for Standards Assessments
OAAA
Fees for Standards Assessments
A proposal for fees for Institutional Standards Assessment has been submitted to the Ministry of Finance
The fee will be calculated on the HEI’s Classification and number of FTE students
The fee structure is based on international benchmarking
Fees for Program Standards Assessment are yet to be determined