2. e-marking in large-scale, high stakes assessments conference themes :  role of technology in assessments and teacher education  use of assessments for continuous professional development  assessments, teacher education, and school and systemic reform

3. large-scale, high stakes assessments three questions : a. what do we mean by technology in this context b. how can we best assure assessment validity c. how can e-marking help get the best from current large-scale high-stakes assessment practices

4. what do we mean by technology in this context? a.

5. computer -based testing

6. what do we mean by technology in this context  innovative items … enabling students to show how they arrived at an answer and allowing them to score along a range of possible point values based on how well they understand a particular process.

7. what do we mean by technology in this context  Computer-adaptive tests …that better align with the performance levels of individual students by giving them more challenging or more manageable tasks according to how they perform in early questions.

8. what do we mean by technology in this context  intellectually ambitious assessments … through a combination of innovative items and computer adaptive testing to measure the learning that is difficult to assess in pencil-and-paper tests.

9. in the context of large-scale, high-stakes assessments current issues are:  hardware availability, compatibility and consistency  high speed connectivity  equality of access…

10. in the context of large-scale, high-stakes assessments cost…

11. how can we best assure assessment validity? b.

12. how can we best assure assessment validity  the greatest threat to validity for performance assessments may be construct under-representation — i.e. where the assessment is too narrow or fails to include important aspects of a construct.

13. how can we best assure assessment validity  often, the outcome of large-scale assessments is narrow – a pass or fail, a grade; a raw or standardized number; a percentage.

14. how can we best assure assessment validity The greatest threat to validity for performance assessments may be construct under-representation — where the assessment is too narrow or fails to include important aspects of a construct.  we need help to create good test designs  we need pre-tested performance information

15. how can we best assure assessment validity  often, the outcome of large-scale assessments is narrow – a pass or fail, a grade; a raw or standardized number; a percentage  we need detailed, more useful, information about performance on different aspects of the tests

16. how can e-marking help? c.

17. e-marking not like this!

18. e-marking

20. e-marking: efficient, secure, reliable…. images of student responses captured by scanning scripts are securely stored images are automatically distributed to appropriate markers markers log on to receive and score images on line, on screen, at item level. supervisors track marking progress and quality in real-time e-marking

22. e-marking: security, accuracy, speed Each year: Pearson marks over 60 million exams in the USA and more than 7.2 million national tests and 9 million exams in the UK Some exams taken in the England are marked in Australia Pearson designs and delivers national exam processing systems, including e-marking, for programmes as diverse as national testing in Abu Dhabi to State and national assessment programmes across Australia Results are delivered quicker and in more detail than ever before

23. Key marking benefits…  increased efficiency – removes majority of manual handling of scripts  reduced marking time – concurrent marking, image distribution, no remarking, mark as scanned  reduced risk, secure scripts, secure images  increased control and flexibility  increased marking quality  fairer outcomes for students  Item-level results data capture e-marking

24. Item-level results analysis and reporting Item-level results analysis and reporting

25. information to help desig good tests  item-level facility values  item performance  bias factors  overall test performance analysis Automated analysis of e- marking pre-test results can inform test design and test validity:

26. Item performance characteristics A well performing item: The probability of a student selecting the correct answer increases as the student’s overall test performance increases Facility 0.76

27. Item performance characteristics A poor performing item Success is not dependent on the students, overall test performance The item does not differentiate between a low and high ability student Facility 0.29

28. Multiple choice question distractor analysis

29. information to support school improvement and raise standards of student achievement  e-marking provides a level of accuracy and detail about student performance never before available  automated results analysis can directly benefit: ministry and departments of education to define policy schools and teachers to identify good practice and areas for development in teaching students and parents to take control of learning …helping to raise standards

30. information contributing to professional development helping teachers to address such questions as: how are we doing as a school how are we doing compared with other schools like ours how are we doing compared with last year how are we doing each subject how are we doing in each class how are we doing on each paper; on each question and is each student doing.

31. ResultsPlus | Analysis helping departments

32. ResultsPlus | Analysis helping centres

33. ResultsPlus | Analysis helping teachers

34. ResultsPlus | Skills

35. Summary  role of technology in educational measurement and teacher education existing e-marking technology offers more from paper-based assessments to educate teachers about where their students are at..  use of educational measurement for continuous professional development..leading to the sharing of professional expertise and shifting classroom expectations.  educational measurement, teacher education, and school and systemic reform An emphasis on teacher education combined with quality assured assessments will profoundly affect the efficacy of classroom interactions and raise standards in education.

36. Thank you