1. An Institutional Writing Assessment Project Dr. Loraine Phillips Texas A&M University Dr. Yan Zhang University of Maryland University College October 2010

2.  Why a Writing Assessment Project?  Data Source of the project  College Participation  Assessment Rubric  Scoring of the papers  Inter-rater agreement  Results  Discussion and future of the project.

3.  Undergraduate core competencies were established for Texas A&M graduates.  Effective communication skills are crucial to student success.  Using this project to provide evidence of the quality of student writing allows participating faculty to understand their students performance more comprehensively.  The project includes the Office of Institutional Assessment in conjunction with the University Writing Center.

4.  The data in this writing assessment project were student papers from: Upper-Division Capstone or Upper-Division “W” (writing intensive) courses.  Assignments were approximately 1-20 pages in length.  Assignments that appealed to a general academic audience were preferred.  Examples given included: persuasive or argument papers, summary papers, analysis papers, letters or correspondence, lab or other reports, and case studies.

5. CollegeDepartment# of Papers Submitted % of College Participation Agriculture and Life Sciences Ecosystem Science and Management 6512% ArchitectureConstruction Science5210% Mays Business School Accounting111 34% Finance38 Information and Operations Management 32 Education and Human Development Education Administration and Human Resource Development 29 10% Health and Kinesiology24 GalvestonMaritime Administration204% GeosciencesGeology and Geophysics173% Liberal Arts Communication17 11% English43 QatarEngineering/Philosophy61% ScienceBiology7514% Total529100%

6.  The writing assessment rubric was developed in conjunction with the University Writing Center, the assessment liaisons, and the Core Curriculum Council.  The rubric was designed to promote validity, uniformity, and consistency in the grading process.  The assessment rubric was categorized into four specific criteria to help manage grading.  After feedback on the rubric from the project pilot, the rubric was adapted to the following figure.

8.  All identifiable information for students and faculty was redacted from the papers.  All day grading sessions were conducted with Dr. Valerie Balester, Executive Director of the University Writing Center, and Dr. Candace Schaefer, Associate Director of the University Writing Center, serving as facilitators.  Faculty members were calibrated by paper genre for the scoring session.  The scoring sheet is provided in the following slide.

9. Grader # Development123123123123 Style123123123123 Organization123123123123 Conventions123123123123  The grading was done by faculty members of the institution from across disciplines.  Grader participation is included in the following slide.

10. College# of Graders Participated Architecture2 Agriculture and Life Science5 Education and Human Development5 Galveston 1 Geosciences2 Liberal Arts9 Library3 Mays Business School 2 Science2 Veterinary Medicine and Biomedical Sciences1 University Writing Center2 Total34

11.  Each writing assessment assignment was scored by two independent graders, with a third if large disagreement.  Interrater agreement was judged to be statistically substantial (.624).  As the intraclass correlation coefficient (6.24) approaches 1.0, there is less variance within item ratings.

12.  The rate at which two graders agreed on an assessed item by giving that item the same score was reviewed.  Simple agreement between raters on the scores of the items assessed showed a descriptive mean of.676.  Thus, approximately 67% of the time, two independent graders assessed an item and then scored that item the same value.

13.  As previously noted, the scoring of each category was on a scale of 1 to 3 (3 being highest quality).  The following table displays the university averages based on the departments that participated.  Each category scored has a mean and standard deviation.

14. Writing Skills AssessedTexas A&M (n=459)Standard Deviation Style The choices the writer makes for specific audiences. Features may include word choice, tone, and sentence length and structure. 1.91.678 Idea or Content Development The depth or sophistication of thoughts and ideas. Features may include research, reasoning, evidence, detail, and development. 1.91.623 Organization The coherence of the writing. Features may include balance and ordering of ideas, flow, transition, and appropriate format. 1.92.647 Conventions Includes grammar, punctuation, capitalization, spelling, documentation, etc. 1.76.626 Overall Writing Assessment Score1.87.647

15. Writing Skills AssessedTexas A&M (n=441)Standard Deviation Aim The level at which the paper addresses the aim of the assignment (to argue, to explain, to evaluate, etc.) 2.4270.560 Development The depth or sophistication of thoughts and ideas. Features may include research, reasoning, evidence, detail, and development. 2.1750.572 Style The choices the writer makes for specific audiences. Features may include word choice, tone, and sentence length and structure. 2.0340.547 Organization The coherence of the writing. Features may include balance and ordering of ideas, flow, transition, and appropriate format. 2.1830.595 Conventions Includes grammar, punctuation, capitalization, spelling, documentation, etc. 1.9720.555 Overall Writing Assessment Score2.1580.584

16.  Faculty engagement and participation  Cross-disciplinary approach  Helps faculty define student writing quality  Helps faculty calibrate expectations for the quality of student writing

17.  Getting the papers!  Representative sample  Calibrating faculty  Long day of scoring—stay nourished!

18.  As a component of Vision 2020, the Academic Master Plan highlights effective communication as a necessary student ability.  Participating departments can take the information given from this project to better understand the performance of their students.  Steps have also been taken to assess areas of potential improvement and enhancement of this project.  Consider VALUE Rubrics from AAC&U

19. February 20-22, 2011 College Station, Texas http://assessment.tamu.edu/conference Call for Proposals now open! Plenary Speakers: Dr. Carol Geary Schneider Dr. Peter Ewell

20.  What was the most valuable thing you learned?  What is one question that you still have?  What do you think is the next step that your program needs to take?