1. Rubrics and feedback in a diverse classroom Major Research Paper – M.A in Applied Linguistics – York U. Jessica King Copyright Jessica King 2010. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author

2. Presentation outline 1. Motivation for project 2. Context for research 3. Research Questions 4. Theoretical Background 5. Methodology 6. Results 7. Recommendations 8. Bibliography

3. Why did I do this project?

4. Context for research – C101 Foundation level writing course for most diploma programs and prerequisite for more advanced communications courses Foundation level writing course for most diploma programs and prerequisite for more advanced communications courses Duration – 42 hours (usually 3.5 months) Duration – 42 hours (usually 3.5 months) Both NES and NNES Both NES and NNES Revision of C101 started in 2006 – creation of “Standard- Varied College English curriculum” Revision of C101 started in 2006 – creation of “Standard- Varied College English curriculum” 2 major assignments with 2 different rubrics 2 major assignments with 2 different rubrics Process approach to writing instruction Process approach to writing instruction Implemented across the College in Fall 2007 Implemented across the College in Fall 2007

5. Context for research – the rubrics A1: 1000 word opinion response to a reading A1: 1000 word opinion response to a reading – 4 criteria: focus & organisation, style, textual analysis, grammar & mechanics – 5 levels of mastery: superior/ advanced, good, fair, basic, below basic A2: 1250 word research essay using APA ref A2: 1250 word research essay using APA ref – Criteria: textual analysis replaced with development & supporting evidence; grammar & mech expanded to include referencing Analytic method with holistic scoring Analytic method with holistic scoring

6. Research questions Teacher Perceptions of Rubrics and Feedback Do the teachers find these rubrics useful? If so, in what ways are they useful? Do the teachers find these rubrics useful? If so, in what ways are they useful? What are the functions of the rubrics for the teachers? What are the functions of the rubrics for the teachers?

7. Theoretical background and relevant research The mature writing process: The mature writing process:  knowledge telling vs. Knowledge transforming. (Scardamalia & Bereiter 1987) Hayes et al’s (1987) Process model of revision Hayes et al’s (1987) Process model of revision

8. Theoretical Background and Relevant Research The teacher’s role in student revision The teacher’s role in student revision  Students DO read feedback (Ferris, 2003)  Varied feedback works best * (Bitchener et al 2005; Goldstein, 2004; Ferris, 2003)  Debate still exists about usefulness of grammar correction (Truscott, 2007; Ferris, 2004)

9. Theoretical Background and Relevant Research Assessment Assessment  Rubrics can be a source of feedback  Basis for student self-assessment  Provides clear explanation of what makes a good paper (Andrade, 2000)

10. Theoretical Background and Relevant Research Advantages More robust profile of student achievement (Andrade, 2000) Acknowledges unequal development of sub-skills Disadvantages May slow down marking process (Hughes, 2003) Interaction with teacher’s internal criteria (Barkaoui, 2007; Hamp-Lyons, 1995) Not easily transferable to different assignments or students (Hamp-Lyons & Henning, 1991)

11. Methodology Participants: N = 8 N = 8 4 contract, 4 full-time status 4 contract, 4 full-time status All had taught new C101 for at least one semester All had taught new C101 for at least one semester 3 different departments: CSHS, GEA, HOSP 3 different departments: CSHS, GEA, HOSP

12. Methodology Interview structure: Introduction Introduction Teacher feedback Teacher feedback Teacher perception of rubrics Teacher perception of rubrics

13. Methodology Study Limitations: No analysis of actual feedback given on student texts – only had teachers’ description of their practice No analysis of actual feedback given on student texts – only had teachers’ description of their practice  But focus of interview was not on “best practices” Different teacher schedules in summer Different teacher schedules in summer  But all teachers had taught C101 before Smaller classes in summer Smaller classes in summer  Result: sample of students (N=5) not representative of actual student experience

14. Results TinaStevePamLucy KenKathyKarenBrian Yrs teach 755161292012 Yrs teach C101 2.5550.5192 ESL exp. EFL, ESL no ESL in coll. ESL lit., LINC ESL Writ. class ESL writ. In college TESL cert. ES/FL Yrs using rubrics 752 “on and off” 2 Used criteria sheets 205

15. Results - Feedback Teachers used 5 different types of feedback: between-the-lines edit, marginal comments, end comments, rubrics, oral conferences Teachers used 5 different types of feedback: between-the-lines edit, marginal comments, end comments, rubrics, oral conferences Time spent on first and final drafts: mean = 20 mins for each. Time spent on first and final drafts: mean = 20 mins for each. One teacher did not give feedback for first draft One teacher did not give feedback for first draft Half the teachers used rubrics to provide feedback to students between drafts (in conjunction with other kinds of feedback) Half the teachers used rubrics to provide feedback to students between drafts (in conjunction with other kinds of feedback)

16. Results – Rubric use Most (n=7) teachers explain rubrics to students. Most (n=7) teachers explain rubrics to students. All explained rubrics by leading class through an explanation of the rubrics. Some gave a short description, some a more involved overview All explained rubrics by leading class through an explanation of the rubrics. Some gave a short description, some a more involved overview

17. Results – Rubric use All teachers stated rubrics assisted them when giving feedback. Rubrics assisted them in 4 ways:  as guidelines or lists of benchmarks (n=3)  as justification or evidence for grades given (n=2)  as communication tools between teachers and students (n=3)  as increasing objectivity or reducing subjectivity in evaluation (n=2)

18. Results – Rubric Use Did the teachers feel the rubrics restricted their written feedback? No (N=4), Yes (N=4) Did the teachers feel the rubrics restricted their written feedback? No (N=4), Yes (N=4) Did the teachers feel that the rubric standards were appropriate for their students in general? Yes (N=5), No (N=3) Did the teachers feel that the rubric standards were appropriate for their students in general? Yes (N=5), No (N=3) Did the teachers find standards appropriate for NNES students? Yes (N=2), No (N=3), unable to answer (N=3) Did the teachers find standards appropriate for NNES students? Yes (N=2), No (N=3), unable to answer (N=3) Did teachers think there were specific L2 writing problems that were not dealt with in the rubric? Yes (N=5) – grammar & mechanics, style sections Did teachers think there were specific L2 writing problems that were not dealt with in the rubric? Yes (N=5) – grammar & mechanics, style sections

19. Results – Rubric Use Did teachers make modifications to the rubrics? Yes (N=8) Did teachers make modifications to the rubrics? Yes (N=8)  (N=3) made minor changes – cosmetic, not affecting meaning  (N=2) made medium-level changes – vocabulary  (N=4) made major changes – altering or adding to criteria

20. Results - Rubrics What did teachers feel were the advantages to using rubrics? What did teachers feel were the advantages to using rubrics?  Made teacher expectations clear, reinforced college standards (N=5)  Increased objectivity and appearance of fairness (N=2)  Saves time (N=2) What did teachers feel were the disadvantages to using rubrics? What did teachers feel were the disadvantages to using rubrics?  Rubrics differed from their ideas of good essays (N=4)  Rubrics are too open to interpretation (N=3)  Rubrics are too time consuming (N=1)

21. Results Dominant Themes: Student and teacher interpretation of rubric language – clarity of feedback and objectivity in grading Student and teacher interpretation of rubric language – clarity of feedback and objectivity in grading  Accessibility of language for students  Teacher interpretation of language Discrepancies between teachers’ general impressions and rubric scores Discrepancies between teachers’ general impressions and rubric scores

22. Recommendations Concentrate feedback on first drafts - formative feedback (Ferris, 2003) Concentrate feedback on first drafts - formative feedback (Ferris, 2003) Consider varied feedback - mix of oral, written, rubric use (Ferris, 2003; Leki, 2007, Bitchener et al 2005) Consider varied feedback - mix of oral, written, rubric use (Ferris, 2003; Leki, 2007, Bitchener et al 2005) Promote student understanding Promote student understanding  Explain rubrics as early as possible – use Cognitive Modeling (Cumming, 1986; Dobson & Feak, 2001; Scardamalia et al 1984)  Portfolio assessments as in Wilhelm (1996) use in EAP class

23. Recommendations Promote teacher understanding (if rubrics are created by college) Promote teacher understanding (if rubrics are created by college)  Teacher training on rubrics (Barkaoui, 2007)  Longer text explanation of rubrics supplied to teachers  Empower teachers to recommend changes Promote mutual understanding – review wording of criteria with NES and NNES students in mind (grammar/ mechanics and style sections) Promote mutual understanding – review wording of criteria with NES and NNES students in mind (grammar/ mechanics and style sections) Revisit “problem” terms: rhetorical mode, diction, subordination, modifiers, lack of awareness of audience, obscuring meaning, structures are varied Revisit “problem” terms: rhetorical mode, diction, subordination, modifiers, lack of awareness of audience, obscuring meaning, structures are varied

24. Bibliography Andrade, G. H. (2000). Using rubrics to promote thinking and learning. Educational Leadership, 13-18. Andrade, G. H. (2000). Using rubrics to promote thinking and learning. Educational Leadership, 13-18. Barkaoui, K. (2007). Rating scale impact on EFL essay marking: A mixed-method study, Assessing Writing, 12, 86-107. Barkaoui, K. (2007). Rating scale impact on EFL essay marking: A mixed-method study, Assessing Writing, 12, 86-107. Bereiter, C., & Scardamalia M. (1987). The Psychology of Written Composition. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Bereiter, C., & Scardamalia M. (1987). The Psychology of Written Composition. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Bitchener, J., Young, S., & Cameron, D. (2005). The effect of different types of corrective feedback on ESL student writing. Journal of Second Language Writing, 14, 191-205. Bitchener, J., Young, S., & Cameron, D. (2005). The effect of different types of corrective feedback on ESL student writing. Journal of Second Language Writing, 14, 191-205. Cumming, A. (1986). Intentional learning as a principle for ESL writing instruction: A case study. In P.M. Lightbown & S.P. Firth (Eds.) TESL Canada Journal: Both Sides of the Desk: Roles and Responsibilities in ESL/EFL Teaching and Learning. 69-83. Cumming, A. (1986). Intentional learning as a principle for ESL writing instruction: A case study. In P.M. Lightbown & S.P. Firth (Eds.) TESL Canada Journal: Both Sides of the Desk: Roles and Responsibilities in ESL/EFL Teaching and Learning. 69-83. Dobson, B. & Feak, C. (2001). A cognitive modeling approach to teaching critique writing to non-native speakers. In D. Belcher & A. Hirvela (Eds.). Linking Literacies: Perspectives on L2 Reading-Writing Connections. (pp. 186-199). Ann Arbor: The University of Michigan Press. Dobson, B. & Feak, C. (2001). A cognitive modeling approach to teaching critique writing to non-native speakers. In D. Belcher & A. Hirvela (Eds.). Linking Literacies: Perspectives on L2 Reading-Writing Connections. (pp. 186-199). Ann Arbor: The University of Michigan Press. Ferris, D. R. (2003). Responding to writing. In B. Kroll (Ed.). Exploring the Dynamics of Second Language Writing (pp. 119-140). Cambridge: Cambridge University Press. Ferris, D. R. (2003). Responding to writing. In B. Kroll (Ed.). Exploring the Dynamics of Second Language Writing (pp. 119-140). Cambridge: Cambridge University Press. Ferris, D. R. (2009). Teaching college writing to diverse student populations. Ann Arbor: University of Michigan Press. Hamp-Lyons, L. (1995). Rating nonnative writing: The trouble with holistic scoring. TESOL Quarterly, 29, 759-762. Hamp-Lyons, L. & Henning, G. (1991). Communicative writing profiles: An investigation of the transferability of a multiple- trait scoring instrument across ESL writing assessment contexts. Language Learning, 41, 337-373.

25. Bibliography Hayes, J., Flower, L., Schriver, K.A., Stratman, J.F., Carey, L. (1987). Cognitive processes in revision. In S. Rosenberg (Ed.), Advances in Applied Psycholinguistics (Vol. 2) (pp. 176-240). New York: Cambridge University Press. Hughes, A. (2003). Testing for language teachers (2nd ed.). Cambridge: Cambridge University Press. Leki, I. (2007, June). Research on feedback to L2 writing, TESL Contact, 33(2), 7-27. Silva, T. (1993). Toward an understanding of the distinct nature of L2 writing: The ESL research and its implications. TESOL Quarterly, 27, 657-677. Wilhelm, K. H. (1996). Combined assessment model for EAP writing workshop: Portfolio decision-making, criterion- referenced grading, and contract negotiation, TESL Canada Journal, 14(1), 21-33.