Assessing Teacher-made Common Final Exams: Instruments and Procedures - Tahnee Bucher - Mariana Menchola-Blanco - Angelina Serratos, PhD - Eddy White, PhD Center for English as a Second Language (CESL) Arizona Assessment Conference April 1, 2016 Arizona Assessment Conference April 1, 2016

Contents CESL / IEP Overview Rationale for Common Finals (CF) Project Feedback Review Procedures & Teacher Survey Written Comm. CF Feedback Oral Comm. CF Feedback Q & A CESL / IEP Overview Rationale for Common Finals (CF) Project Feedback Review Procedures & Teacher Survey Written Comm. CF Feedback Oral Comm. CF Feedback Q & A 2

3

Overview - Intensive English Program Founded in 1968 Seven levels in the IEP Curriculum (beginner, intermediate, advanced) Each IEP session is 8 weeks, five per year. 4 courses: Oral Communication, Written Communication, Reading Skills, Grammar Student numbers Summer 2015 to now = IEP teachers:

5

6

The goal of assessment is to... 7

The goal of assessment has to be, above all, to support the improvement of learning and teaching. (Frederiksen & Collins, 1989)

9

Assessment Literacy Know-how and understanding teachers need to assess students effectively and maximize learning 10

Cambridge Guide to Second Language Assessment (2012) Clearly, there is widespread global recognition that language assessment literacy represents an important aspect of teachers’ professional knowledge. Pre-service and in-service teachers often don’t have the necessary skills to effectively administer sound assessments in the classroom. Research continues to characterize teachers’ assessment and evaluation practices as largely incongruent with recommended best practice. 12

There is no getting away from the fact that most of the things that go wrong with assessment are our fault, the result of poor assessment design- and not the fault of our students. (Race et al., 2005) 13

For example...

Common Final Exams Project 15

standardizing final exams Tests are standardized when the directions, conditions of administration, and scoring are clearly defined, and fixed for all examinees, administrations and forms (Cohen & Wollack, 2006). standardizing final exams Tests are standardized when the directions, conditions of administration, and scoring are clearly defined, and fixed for all examinees, administrations and forms (Cohen & Wollack, 2006). 16

Standardizing (i.e. students having the same final exam experience across all sections of the same course) final exams is advantageous for the following reasons: 17

Reasons for Implementing CF’s It benefits students to have a common final that has been created by teachers and vetted for validity, reliability. etc. This new assessment system should add rigor to assessment in the IEP overall, and make it a stronger program. Teachers working together in collaborative test creation is an excellent opportunity for professional development, and also, importantly, for the development of a more assessment literate faculty. It benefits students to have a common final that has been created by teachers and vetted for validity, reliability. etc. This new assessment system should add rigor to assessment in the IEP overall, and make it a stronger program. Teachers working together in collaborative test creation is an excellent opportunity for professional development, and also, importantly, for the development of a more assessment literate faculty. 18

19

What is feedback? (Wiggins, 2004) Feedback is information about what was and was not accomplished, given a specific goal. 20

Feedback: summative vs formative Summative feedback is a summary of students final output or performance, includes a grade or score. Formative feedback - response to student work, while it is in progress - identifies strong and weak aspects of performance, gives suggestions for improvement - plays a part in ‘forming’ or shaping student response to the task being worked on - aims to draw out students’ best possible performance 21

Quality feedbacK: 4 Criteria 1. It must be timely. 2. It must be specific. 3. It must be understandable to the receiver. 4. It must allow the student to act on the feedback (refine, revise, practice and retry ). 1. It must be timely. 2. It must be specific. 3. It must be understandable to the receiver. 4. It must allow the student to act on the feedback (refine, revise, practice and retry ). (Wiggins, 1998) 22

Creating/Assessing Tests Two Primary Considerations 1.Tasks (or task) 2.Scoring 23

Course Leaders for Common Finals Responsible for supervising/advising Common Finals for all levels of one course. Report to Assessment Coordinator. Written Communications = Anglina Oral Communications = Tahnee Grammar = Mariana Reading = Eddy 24

Current CF Plan CF’s for OC, WC, Grammar (Reading added in Spring 2) Worth 20% of final grade Teachers submit first draft of final at end of Week 4 - get feedback from Course Leaders & AC Final draft due Week 6 OC, WC classes finals are not administered or scored by solely by section teacher – objectivity CF’s for OC, WC, Grammar (Reading added in Spring 2) Worth 20% of final grade Teachers submit first draft of final at end of Week 4 - get feedback from Course Leaders & AC Final draft due Week 6 OC, WC classes finals are not administered or scored by solely by section teacher – objectivity 25

We decided to provide feedback on first drafts of submitted exams by focusing on 1) the task (s), and (2) the scoring. We have revised this feedback sheet over time. We decided to provide feedback on first drafts of submitted exams by focusing on 1) the task (s), and (2) the scoring. We have revised this feedback sheet over time.

27

Version 1 (Fall, 2015)

29

(December 2015) 30

Teacher Views of Feedback (from Dec Survey) Did teachers feel that the feedback provided result in a better final exam? 31

Teacher Comments – Dec Survey The comments were to the point and showed that the CF group paid attention to what we submitted. They noticed the things we had missed. I felt bullied into using an approach I would have otherwise not chosen. According to the feedback, there seems to only be one option (the one which the Assessment coordinator agrees with) and none other. I don't always agree with the feedback, but that doesn't mean it isn't helpful/useful. The comments were petty....Our rubric was out of 100 points and we were told that we had to change it from points to percentages. The essay structure we were testing is called Compare & Contrast and yet we were advised to change it to Compare or Contrast. Neither recommendation improved our final, but did create busy work. 32

Version 2 & 3 (Spring, 2016)

Written Comm. CF 36

This following first draft was submitted at the end of Week 4 (of an 8-week session).

Your Assessment of This Task? Feedback?

Scoring (rubric)

First Draft Feedback

Oral Comm. CF 43

This following first draft was submitted at the end of Week 4 (of an 8-week session).

Your Assessment of This Task? Feedback?

Scoring (rubric)

First Draft Feedback

Here are the procedures and questions for the 5 OC Common Final exam. Before the final, students are put into groups of 3-4 students and assigned set times to report for the final exam. The questions in the attachment are printed out and cut with one question per slip of paper and placed randomly inside an envelope During the final, students will sit in a circle with the teacher outside the circle, but in the room. The first student will draw 1 slip of paper from the envelope and read it aloud to their group. That student will then respond to the question initially before opening up the discussion to the other students in the group. He/she then facilitates that discussion for a few minutes. When the students are satisfied with their discussion, they move on by having the next student draw and read aloud a new question initiating a new round of discussion. Each group is allotted about mins. depending on the size and number of students in the group and class as a whole. 5 OC Final Exam

Common Final Exams Project 52