1. PROGRAM REVIEWS 2019 PRE-REVIEW WORKSHOP
4 July 2019

2. Time
Activity
8.30 – 9.30 am
Registration
9.30  am
Introduction: objectives of workshop, programme review process and role of reviewers
10.00 – am
Documentation: distribution of SERs, Declaration of Interests form, Letter of Appointment and Agreement with UGC
10.30 – am
Tea
10.45 – am
Desk evaluation
11.15 am – pm
Site visit schedule and effective use of meetings
12.00 – pm
Scoring standards and calculation of final grade
12.30 – 1.30 pm
Lunch
1.30 – 2.00 pm
Tentative dates for site visits
2.00 – 2.30 pm
Preliminary report, draft report and final report formats, deadlines for submission
2.30 – 3.00 pm
General discussion
3.00 pm
Tea and close

3. SESSION 1. INTRODUCTION

4. Objectives of workshop
Formal aspects
Hand over SERs to assigned review panels + letters of appointment
Signatures on Declaration of Interest, and Agreement with UGC
Agree on dates for site visits
Training aspects
Ensure that all reviewers know what is required of them in the EQA process conducted by the QA

5. Background Third year of Program Reviews in current cycle (2nd)
2017 – UG programs in Arts, Humanities, Social Sciences
2018 – UG programs in Fine Arts, Education, Law, Management and Commerce
2019 – Programs related to Health, Agriculture (+ 1 from 2018)
2020 – Programs offered by Faculties of Science, Applied Sciences, Engineering, Technology
Total of 39 degree programs offered by 24 Faculties to be reviewed between Sept & Dec 2019

6. Role of reviewers in external quality assurance
Reviewers are vitally important in the effectiveness and success of EQA
Need to work as a team, with Review Chair as the team leader
Tasks need to be divided up between team members before, during and after site visit
Only the initial desk evaluation is to be done individually

7. Reviewer profile – key characteristics
High degree of professional integrity and objectivity
An enquiring disposition
Ability to readily assimilate a large amounts of disparate information
Ability to make appropriate judgments in the context of complex institutions different from their own
Personal authority and presence
Ability to act as an effective team member
Good time management skills
Ability to give effective oral feedback

8. Reviewer profile ctd
Experience in organization and management, particularly in relation to teaching and learning matters
High standard of oral and written communication, preferably with experience in writing formal reports
Knowledge and understanding of the review topics, principles, and concepts
Knowledge of the special characteristics and conditions of the educational provision to be reviewed
Knowledge of quality assurance and quality enhancement procedures

9. Composition of review panels
Tried to include:
At least 2 members with expertise in broad field of study
Balance of
Review Experience,
University,
& Gender
Took into account:
Declarations by reviewers of any conflict of interest
Concerns expressed by Faculty under review

10. Program Review Process
Desk Evaluation of SER by reviewers
SITE VISIT TO VALIDATE CLAIMS IN SER
Preliminary report
Draft report
Comments from Faculty on draft report
FINAL REPORT
FR edited and published by QAC
Faculty Action Plan for implementation of recommendations

11. Deliverables required of reviewers
Individually:
Desk evaluation report
As a team:
2. Preliminary report (key findings)
3. Draft report
4. Final report

12. SESSION 2. DOCUMENTATION

13. SESSION 3. DESK EVALUATION OF SER

14. Deliverable 1.
Each reviewer is expected to assess the SER using a pre-formatted Excel file provided for this purpose
Assign scores for each standard, by comparing what is given in the SER and the best practice listed in the Manual
Make notes on any items that you would like clarified during the site visit
Not necessary to complete final summary sheet (i.e. award final grade)

15. Pre-formatted Excel file for PR scores
Save file with abbreviated name of university and degree, and your initials added at end
E.g. CMB_MBBS_NRdeS.xls, PDN_BSc MLS_NRdeS
Excel file to QAC with copy to before pre-site visit meeting on 2 August
Bring copy to pre-site visit meeting, for discussion with rest of review panel

16. Questions?

17. SITE VISIT SCHEDULE AND EFFECTIVE USE OF MEETINGS

18. Role of Review Chair
Contact Dean / Program Coordinator ahead of site visit and agree on schedule for site visit
Decide on allocation of tasks in consultation with team members
Chairing meetings
Going through supporting documents for specific criteria / standards
Observing teaching-learning activities
Inspecting infrastructure and facilities
Writing up sections of draft report
Take the lead in initial and wrap-up meetings with VC / Dean, etc
Compile and edit draft report
Submit final report as soft copies + one hard copy

19. Site visit schedule Duration – 4 days
Should be broadly agreed upon prior to site visit
Faculty should provide academic timetables so reviewers can decide on which teaching-learning activities are to be observed
Generic format provided by QAC should be modified to suit requirements of Faculty, and any contingencies that emerge during site visit

20. Generic schedule for Day 1
Time
Activity
Participants
8.30 AM – 900 AM
Meeting with the Vice Chancellor
Vice Chancellor/ Dean, Director – IQAU/ Coordinator – FQAC, Chair – SER Preparation
9.00 AM – 9.30 AM
Meeting with the Director - IQAU
Director – IQAU
9.45 AM – AM
Presentation about the Faculty and respective study programs
Working Tea
Dean FMS/ Director-IQAU/Coordinator FQAC/
All HODs of the Faculty/ Cluster Chair and SER Team/ Study program coordinators
10:45 AM -11:45 AM
Meeting with academic staff in permanent cadre (excluding HOD)
Teaching panel of respective programs (excluding HODs) Senate representatives
11:45 AM -12:15 PM
Meeting with Temporary academic staff
Temporary Demonstrators, Tutors etc
12:15 PM -1:00 PM
Meeting with Administrative Staff
Registrar/Bursar/SARs/AB/SAB/Work Engineer/DR Examination
1:00 PM -1:30 PM
Lunch
1:30 PM -2:15 PM
Meeting with Directors of Centres / Units / Cells
All Directors of Centres/ Units/ Cell Coordinators
2:15PM-2:45PM
Meeting with Student Counselors
Senior Student Counselors and student counselors
2:45 PM -4:00 PM
Observing, Physical Facilities
Tea
Review Team/ Facilitators

21. Generic schedule for Day 2
Time
Activity
Participants
8.30 AM – 900 AM
Observing documentation
Review Team/ Facilitators
9.30 AM – AM
Observing teaching sessions and facilities
Review Team
10.30 AM – AM
Meeting with Librarian/Senior Assistant Librarians [Library Visit]
Librarian/Senior Assistant Librarian/ Library Staff
11.00 AM -11:30 AM
Meeting with Technical Officers
All Technical officers
11:30 AM -12:30 AM
Observing Documentation
12:30 PM -1:30 PM
Lunch
1:30 PM -4:00 PM
Working Tea

22. Generic schedule for Day 3
Time
Activity
Participants
8.30 AM – 9.30 AM
Observing Documentation
Review Team
9.30 AM – AM
Meeting with Students
Working Tea
Group of students (30) representative of gender, ethnicity, level of study programs
10.30 AM – AM
Meeting on support for student welfare
Director/Physical Education, University Medical Officer
11:30 PM -12:00 Noon
Meeting on research activities
Chairman / Research committee, members of research committee
12:00 PM -12:30 PM
Meeting with a cross section of academic support staff and non-academic staff
Representative group of academic support staff and non-academic staff (10)
12:30 PM -1:15 PM
Lunch
1:15 PM -2:15 PM
Meeting with external stakeholders and alumni members
Group of external stakeholders (about 20 employers, industry, private sector, representatives with link to or involvement with the University) and Alumni
2:15 PM -4:30 PM

23. Generic schedule for Day 4
Time
Activity
Participants
8.30 AM – 900 AM
Meeting with mentors and Career Guidance staff
Coordinator/mentoring and mentors, and Director – Career Guidance
9.00 AM – 9.30 AM
English Teaching Unit
Members of English teaching unit
9.30 AM – AM
Observing Documentation
Working Tea
Review Team
12:30 PM -1:30 PM
Lunch
1:30 PM -2:00 PM
Private meeting of reviewers and report writing
2:00 PM -3:00 PM
Closing Meeting for debriefing
Vice Chancellor/Dean/Director – IQAU/ HODs/ Coordinator – FQAC/Chair & the SER – Team

24. Effective use of meetings: ‘DO’s
Use meetings as an opportunity to triangulate evidence presented in documentary form
Go prepared with list of questions that need to be answered by participants in each meeting
Make a note of specific questions in relation to the criteria and standards assigned to him /her
Use open-ended questions to start with, and specific questions when clarity is needed
Be punctual and stay with the agreed program for meetings
Keep attendance records and written notes of all discussions

25. Effective use of meetings: DON’T s
Do not allow one or two persons to dominate a meeting with a group
Do not get distracted into discussions that are irrelevant to the PR
Do not go beyond the time allocated for the meeting, and get late for the next one, and the next…

26. Questions?

27. SCORING EACH STANDARD AND CALCULATION OF FINAL GRADE

28. = = = 156 Standards Programme Review Quality Framework Criterion 1 2 34 5 6 7 8
Number of
Standards 27 12 24 19 17 14
Programme Review Quality Framework = = =
156
Standards

29. Assigning Scores for Standards

30. Assigning scores objectively
When scoring a standard, the Panel should determine
Degree of internalization of best practices and level of achievement of Standards, as stated in SER
Degree to which the claims are supported by documented evidence, as indicated in SER
Accuracy of the data and statements made in the SER, as observed during site visit

31. Guidance in Program Review Manual

32. CLAIM IN SER

33. 1. Degree of internalization of best practices and level of achievement of Standards
2. Degree to which the claims are supported by documented evidence
3. Accuracy of the data and statements made in the SER

34. Guidance for decision-making
Question 1. What is the recommended best practice for this standard as stated in Program Review Manual?
Question 2. What is the claim made by the program regarding their own practice(s) as stated in SER?
Question 3. What evidence does the program provide to support this claim, as stated in the SER?
Question 4. Do the Panel’s observations during the site visit support the claim?

35. Claim of internalization of best practice Meets standard
Evidence sufficient to support claim
3 marks
Evidence not sufficient to support claim
1 or 2 marks
Below standard
1 mark
No claim of achievement
0 marks

36. Example
Examples of Sources of Evidence expected in PR Manual: Guidelines and formats of Performance Appraisal System; sample of Annual Appraisal Reports; CPD programmes planned & conducted and follow up action taken; reward scheme that is in place and names of recipients over the past 3 years.
Review Team Observations: SER claims Faculty doesn’t practice the appraisal system except for the annual increment and promotion. Evidence listed to support the claimed level of achievement deserves only 1 mark

37. Calculation of final grade
Raw scores will be automatically converted to weighted actual scores on Excel file (Worksheet entitled ‘Summary scores’)
For each criterion, check if the weighted actual score is above the weighted minimum score
Check on total actual score
Use table provided in worksheet entitled ‘Summary scores’ to determine final grade

38. Award of final grade Criterion-wise actual score
Total actual score (%)
Grade
Equal to or more than the minimum weighted score for all eight criteria A B C
<60 D
Equal to or more than the minimum weighted score for seven of the eight criteria
Equal to or more than the minimum weighted score for six of the eight criteria
Irrespective of minimum weighted criterion scores

39. Final grade Grade descriptors A Very good B Good C Satisfactory D
Review panel must decide on final scores and grade together during last session on Day 4
Convey main findings to Dean and Department during final wrap-up meeting
Give commendations first; then recommendations for improvement
Grade descriptors A
Very good B
Good C
Satisfactory D
Unsatisfactory

40. Questions?

41. SESSION 4. DATES FOR SITE VISIT

42. SESSION 5. TEAM REPORTS

43. DELIVERABLE 2. PRELIMINARY REPORT
Completed Excel file + Word document with brief details of programme under review
Review Chair to submit to DQAC by , within 2 weeks of completing the site visit

44. Format of preliminary report: title page
Program Reviews 2019
conducted by the
Quality Assurance Council
University Grants Commission, Sri Lanka
 PRELIMINARY REPORT
University :
Faculty :
Program :
Review Panel :
Site Visit Dates :

45. Format of preliminary report: contents
Section 1. Background(Half Page)
When the Program was started:
Number of Students in Faculty at present- breakdown in years:
Maximum Capacity of Students allocated by University Grants Commission in the last 4 years:
Number of batches graduated through the program from its inception
Section 2. Criterion-wise strengths and weaknesses
Criterion 1: Programme Management
Strengths:
Weaknesses:
Criterion 2: Human and Physical Resources
Criterion 3 etc

46. Format of preliminary report: contents, ctd.
Section 3. Final evaluation No
Criterion
Weighted minimum score*
Actual criterion- wise score 1
Programme Management 75 2
Human and Physical Resources 50 3
Programme Design and Development 4
Course / Module Design and Development 5
Teaching and Learning 6
Learning Environment, Student Support and Progression 7
Student Assessment and Awards 8
Innovative and Healthy Practices 25
Total score (out of 1000)
Total score (out of 100)
Final grade:

47. DELIVERABLE 3. DRAFT REPORT
Format set out in Programme Review Manual (p 98–102)
Section 1: Introduction to programme
Section 2: Observations on SER
Section 3: Description of review process
Section 4: Faculty’s approach to quality and standards
Section 5: Judgment on each of the 8 criteria
Section 6: Grading of overall performance
Section 7: Commendations and recommendations
Section 8: Summary
Annexures

48. Format of Draft Report: cover page
University logo on left
UGC logo on right
Details of the PR on top with the year and dates of review
Photograph of University / Faculty / Dept
Names of reviewers below photograph
Name-UGC and QAC bottom of page

49. Format of draft report: signature page
Scanned copy of page with signatures to be inserted after cover page
University:
Faculty:
Program:
Review Panel:
Name Signature
Date:

50. Format of draft report
Content page to include relevant pages of each Section and Criterion (1-10/1-8)
Each main Section (1-8) to be started on a new page, with heading aligned centre and subheadings left aligned
Font and size
Calibri font 14 (bold) for main Section headings
Calibri font 13 (bold) for sub-headings
Calibri font 12 for text in paragraphs
Text colour: black
Line Spacing
Main Section headings and text – 3.0
Subheadings and content – 2.0
Paragraphs – 1.5
Sentences in paragraphs – 1.15

51. Annexures Annexures must include: final schedule for site visit
all attendance sheets from meetings
Optional: additional photographs taken during site visit

52. Format of draft report ctd
Numbering of tables and figures: Have the number of the section and relevant number of the section: e.g.1.1, 2.3 
Capitalization in text: use only for abbreviations and as recommended for names of people, departments, universities, etc.
Word limit: Single Program (3 or 4 years duration): words; Cluster Review / 5 year program: 12,000 words

53. Submission of Draft Report to QAC
Deadline for submission to QAC (by , as Word document): within 6 weeks of completing site visit
Draft report will be ed by DQAC to Dean / Rector, for corrections / comments (together with Excel file in Preliminary Report), to be sent back within 3 weeks

54. DELIVERABLE 4. FINAL REPORT
Comments from Dean will be sent to Review Chair, for consideration in finalizing the report in consultation with the Panel
Final Report may be the same as the draft report, if no amendments are requested
If the Review Panel considers it appropriate, comments from the Dean should be incorporated as changes to the draft
Review Chair is required to submit the Final Report in soft copy (Word and pdf formats) and a single hard copy

55. Publication of Final Report
Final Report will be sent back to Dean prior to publication
Minor factual errors may require correction; but no major changes
Final Report is sent to one of a panel of editors prior to publication
Edited Final Report will be uploaded to QAC website and sent to Dean
Programme is required to prepare an Action Plan for implementation of recommendations in report

56. Access to document formats etc