1. 1 Understanding the work of the HKEAA: Quality Assurance C. S. Tong Secretary General 7 June 2013

2. About HKEAA Established in 1977 (independent statutory body) 400 staff in 7 office locations 8,000 exam personnel (seasonal) Mission: To provide valid, reliable and equitable examinations and assessment services in a professional, innovative, efficient and effective manner.

3. Quality Assurance

4. How are examination papers set? Curriculum and Assessment (C&A) Guides Subject Committees to oversee the continual development of each subject Moderation committees to set papers with rigorous security and quality control mechanism Moderation committee members are appointed on an annual basis subject to various security requirements

5. Who are Moderation Committee members? Subject Expert/ University Academics HKEAA Subject Managers Open nomination for membership from schools as well as from the relevant Subject Committee (current S5 or S6 teachers not eligible) Secondary School Teachers

6. 6 How are scripts handled? All scripts are scanned and then archived for safe-keeping: over 10,000,000 pages scanned for Onscreen Marking

7. How are Written Papers marked? Marking scheme Appointment of Markers Sample scripts selection Standardisation meeting Markers’ meeting Allocation of scripts to markers Marking Checkmarking Statistical analysis of marks

8. 8 How are Written Papers marked? OSM facilitates real time, continuous monitoring of the quality of the marking: Marking by questions to enhance consistency in applying the marking criteria Double marking for open-ended questions Use of control scripts to validate marking consistency Scripts are selected for checkmarking by experienced Chief/Assistant Examiners

9. Where are the Assessment Centres? Sheung Shui:185 Tsuen Wan:163 Lai King:376 Sha Tin Wai:92 San Po Kong:73 Fortress Hill:382 Tai Hing:100 Tseung Kwan O:21 About 1,400 workstations

10. Facilities at the Assessment Centres 10 Lounge/PantryConference Room Workstations Reception Area Control Room

11. OSM Procedures 1st checkmarking 2nd checkmarking Markers’ meeting satisfactory marking satisfactory marking Complete Training scripts Qualifying scripts Reference scripts Normal scripts Control scripts

12. What are the advantages of OSM? Security Data availability Efficiency Accuracy Marking quality

13. Markers’ meeting to clarify and finalize marking scheme Provision of Reference Scripts to illustrate the marking requirements and standard Training Scripts & Qualifying Scripts to ensure quality of marking before markers start marking ‘real’ scripts How does the Authority monitor Quality of Marking?

14. Monitoring Quality of Marking Selected scripts (including control scripts etc) have been discussed by a team including the Chief Examiner and awarded standardized scores Control scripts are randomly distributed to markers to ensure consistency of marking standards

15. Monitoring Quality of Marking Marker’s score can deviate from the standardized score to within a specified tolerance When the deviation exceeds the tolerance level, the marker will be contacted for clarification of marking scheme/ further training and qualifying/ or stopped from further marking and problematic scripts can be voided and redistributed to other markers for marking as necessary

16. Standardized Score Marker’s Score Marker’s score agrees with Standardized Score Tolerance level Illustration

17. Standardized Score Marker’s Score Marker’s score much higher than standardized score Marker’s score much lower than standardized score Marker’s score agrees with Standardized Score Tolerance level Illustration

18. Standardized Score Marker’s Score Marker’s score much higher than standardized score Marker’s score much lower than standardized score Marker’s score agrees with Standardized Score Tolerance level Illustration

19. Monitoring Quality of Marking Chief / Assistant Examiners will check each marker’s marking statistics (including marking speed, mark distribution, score deviation from control scripts etc) scrutinize markers’ marking records and select scripts for checkmarking contact markers and further clarify the marking scheme if necessary 19

20. Why Double Marking? To address the issue of inter-rater variability for open ended questions, double marking may be adopted Feasibility study carried out to assess the cost benefit between such factors as Marking reliability Markers’ standardization Increased marking time Marking cost

21. Marker 3 Chief Examiner or Assistant Examiner Discrepancy within range The discrepancy among marks is still out of range Candidate’s mark Discrepancy within range Big discrepancy between the two marks Candidate’s mark Marker 1 Marker 2 Double Marking: how? The Candidate’s final mark will be the sum of the two highest closest marks

22. Post examination review  External vetting of examination papers  Post examination review  Internal audit & risk monitoring  Examination Paper Quality Audit (EPQA)  Internal post-mortem meetings  SBA  Marking process  Mechanisms for handling irregularities, complaints and queries  Standards setting and maintenance  Grading procedures  Quality check of examination data  Feedback to schools Setting of examination papers  Curriculum and Assessment Guide & Assessment Framework  Examination paper development mechanism  Confidentiality and security measures Conduct of examinations  Examination regulations  Public Examinations Information Centre (PEIC)  Public Examinations Communication and Support System (PECSS)  Oral Recording System (ORS)  On-site checking & audits  Examination logistics & security measures HKDSE Recognition Safeguarding of Standards Complianc e & Quality of Delivery Continuous Improvement Processing & reporting of results How to ensure overall Quality?

23. Vetting of Examination papers Different subjects are selected annually for vetting and peer review Vetting reports provide useful feedback for subject managers and Subject Committees Reports and responses from the vetting exercise as well as reports on examination irregularities are submitted to the Public Examinations Board for discussion

24. 24 Discrepancy Checking Scripts are sampled for cross checking at various stages of marking and grading as part of our quality assurance framework: Randomly selected candidates and all their scripts extracted for manual checking Scripts of candidates with large discrepancies in the levels obtained in different subjects Scripts of candidates with large discrepancies in component levels (eg over 1000 scripts from Chinese Language checked in 2012)

25. 25 Discrepancy Checking Performance at the school level are cross- checked across different core subjects: eg LS vs Chin Outlier schools to be manually checked

26. 26 Discrepancy Checking Math vs English

27. Assessment Quality-Assurance Platform Based on Rasch Model Item-level and paper-level analysis to ensure quality: Item difficulties Discrimination power Item reliability Paper reliability (internal consistency)

28. Item Level AQP 評核素質保證平台

29. Paper Level AQP 評核素質保證平台

30. Supporting Quality Assessment at Schools

31. User-friendly version of AQP is available for schools to support Assessment for Learning

32. Student-Level AQP 評核素質保證平台

34. Quality Assessment Management

35. International Recognition

36. 36 International Recognition Over 150 overseas institutions responded to our survey proforma 70 institutions from the Mainland accept students from HK on the basis of HKDSE 149 institutions from Taiwan accept students from HK on the basis of HKDSE

37. 37 International Recognition Comparison UCAS Tariff HKDSE Advanced Placement (AP) GCE Advanced Level (GCE AL) International Baccalaureate (IB) 5** 140 A* 1305* 7 12055A 110 6 100 B 90 4 804 C5 70 60 3D 50 4 403 E 30 20 3 145

38. 香 港 中 學 文 憑 考 試香 港 中 學 文 憑 考 試 一紙文憑世界通行

39. Delivering HKDSE with Pride 香港中學文憑 眾志昂首同行 Thank You!