Denise Kirkpatrick Pro Vice-Chancellor The Open University, UK Assessment in DE contexts

Issues or challenges? How to assess student learning effectively in distributed environment (assessment techniques) Security of –examinations –submitted assignments –marking –returned assignments Authentication/validation of student & ownership of work Academic standards

QA & assessment Clarify role & purpose of assessment Designation of responsibilities of aspects of assessment Design of assessment Alignment with desired learning outcomes Academic standards

Course Teams Design and get approval of teaching and assessment strategy (via Deanery and University approval structures) Set continuous assessments Notify E&A of monitors for each tutor Evaluate output from monitoring Prepare report for award meeting on course presentation issues

Examination & Assessment Board Set examination question paper & associated marking guide Nominate External Examiner Recommend script marker appointment Brief the markers Quality assure the work of markers

Examination & Assessment Board Standardise markers & end-of-course assessment Recommend course results for each student (through award meeting), including consideration of special circumstance cases Respond to upheld appeals

Examinations & Assessment Operations Support assessment policy making Implement and deliver approved assessment strategies Process all tutor and computer marked assignments Co-ordinate monitoring of tutor marked assignments Appoint External Examiners & scriptmarkers

E & A Operations Produce examination timetable Liaise with Regions over examination centre provision Produce and monitor all schedules relating to examinations and the determination of course results Co-ordinate production of examination question papers and marking guides

E & A Operations Arrange distribution of question papers and materials throughout world Appoint markers on behalf of EABs Co-ordinate the marking process for examined & non- examined course Arrange & support coordination, standardisation & award meetings Process & release course results

Tutor Marked Assessment All tasks linked to learning outcomes All students complete the same task Marking guide prepared by course team for Associate Lecturers Associate Lecturers briefed & trained Reasonable adjustments possible Special circumstances admitted Assignments monitored Statistical analysis Sample sent to external examiner

11 online assessment wizards 11 leadership in learning

Routing of Assignments Paper –Student posts to AL –AL marks and forwards to Walton Hall –E & A record mark, initiate monitoring and returns TMA to students –Overall turnaround time about 16 days Electronic –Student submits to server –AL collects from server, marks and returns to server –Student collects from server –Mark automatically recorded and monitoring initiated semi-electronically

Points to note Tutor marked assessments are about learning as well as marks & grades Large number of markers Marker independence Number of scripts per marker is small Marking guides can be weak or ambiguous Monitoring levels can vary Variable submission dates

End of Course Assessment Task & marking guide moderated by External Examiner Independent marking (not the student’s tutor) Compulsory marking coordination exercise Supervision during marking High level of sample double marking Standardisation of marking Involvement of External Examiner

Quality Assurance- Marking of end of course assessments Co-ordination exercises –Dummy marking exercise Monitoring/double marking –15 scripts from each marker double marked Standardisation process –Allows for scaling of scores

Examinations Common task for all students Few markers involved Good statistical comparison possible Two independent monitors for each marker Score adjustments can be made Re-marking possible High level of confidence in marking Relatively low cost

Examinations… Distributed examination sites Proctored/invigilated examinations Security of examination papers Managing time zones

End of course assessments: Projects Ensuring comparability of tasks & responses Large number of markers Few items per marker High marking loads Statistical analysis more complex & less informative Sample second marking leads to double marking Clear identification of marking issues High level of remarking External involvement more complex Relatively high cost – time & money

Use of plagiarism software As a ‘traffic light’/warning system To identify common text Need to be explicit and consistent about penalties Need for student education about academic integrity

Co-ordination Exercise Compulsory activity as part of marking contract Takes place before marking starts Usually face to face Discuss question paper and marking guide Dummy marking exercise using sample of scripts Discussion of marks awarded Work to achieve common understanding of marking guide and standards

Monitoring and Double Marking Teamleaders Members of Examination and Assessment Board (EAB) Participate in Co-ordination Deal with marker problems Second mark 15 scripts of each marker’s work Provide qualitative feedback to EAB

Standardisation Purpose –Review the marking of scripts –Review the question score profile –Review overall standard achieved on course –Identify and correct problems –Prepare for award meeting

Award Meeting Each EAB has an Award meeting About 1 week after standardisation Attendance by all members and External Examiner Objective to decide course result for each student Review course assessment and standards

Determination of Course Results Continuous Assessment (TMAs, CMAs) Examination (Project, Portfolio etc) COORDINATION MEETING SCRIPT MARKING STANDARDISATION MEETING CONFLATION (POCAS) CONFLATION (Substitution) (FOCAS, FOES, rank) AWARD MEETING RATIFICATION/APPROVAL RESULT

Determination of Course Results Determine grades of pass i.e. set boundaries between results Take into account any special circumstances submitted by students and/or University Identify ‘odd’ cases Recommend result for each student to Senate.

Course Results Senate Guidelines provide framework Formal Examinations and Assessment Board for every course At least one External Examiner Examinations and Assessment Board sets end of course assessment External Examiner monitors marking of end of course assessment External Examiner scrutinises standards Close control of adherence to policy Results subject to scrutiny and approval by designated committee

University Policy on Assessment Varies according to credit point value of course Specifies components of assessment (different methods of assessment) Specifies number of continuous assessment tasks Specifies length of examination Specifies number of summative assessment tasks Specifies weighting of assessment components Specifies opportunities for re-sits Specifies results/grades classifications

Ensuring quality of tutors/marking Role - to grade students’ work & provide detailed teaching feedback Detailed marking criteria provided Marking exercises as part of induction & training Work is monitored through internal assignment monitoring process Statistical analysis provides tutor grade marked analysis reports and profiles of turnaround Sample of marked continuous assessment scripts is provided to External Examiners Information from monitoring is considered by Examination & Assessment Board in the determination of results Grading of examination papers & project work is subject to analogous statistical sampling & standardising procedures

Note Assessment is context specific Role & function of assessment Different assessment strategies for different courses Cost of innovation? Balance between economics & pedagogy Use of e-assessment Questions of scale Need for quality assurance & control