1. A National Online Essay Evaluation Service: Valid, Reliable, & Cost Effective Writing Assessment Leslie C. Perelman Leslie C. Perelman Program in Writing and Humanistic Studies Massachusetts Institute of Technology

2. TestWrite™ (a. k. a. iMOAT) Enhancing the best way of assessing writing Enhancing the best way of assessing writing Enthusiastic reception from both students & faculty Enthusiastic reception from both students & faculty Cost effective Cost effective Reliable scoring and delivery Reliable scoring and delivery

3. Motivations Existing modes of placement inadequate Existing modes of placement inadequate  Standardized tests  Timed impromptu  Portfolio Students write on computers not paper Students write on computers not paper

4. The Common Practice Student walks into large room and writes with pen and paper on small desk Student walks into large room and writes with pen and paper on small desk Student reads, at most, a two or three paragraph text for each essay Student reads, at most, a two or three paragraph text for each essay Student writes one or two essays on specific questions Student writes one or two essays on specific questions Handwritten essays are scored (usually by two readers) Handwritten essays are scored (usually by two readers) Score is sent on paper to student and / or advisor Score is sent on paper to student and / or advisor Student works at home Student works at home Student views readings of any length online Student views readings of any length online Student accesses questions online Student accesses questions online Student writes one or two essays at home within a specified period using a computer. Student writes one or two essays at home within a specified period using a computer. Essays are scored on either printed copy or on screen by readers in real or virtual grading sessions Essays are scored on either printed copy or on screen by readers in real or virtual grading sessions Student and advisor access both score and text of essays. Student and advisor access both score and text of essays. TestWrite

5. A Better Way to Assess Writing

6. Developed by Experts in Writing Assessment at Five Diverse Universities

7. With TestWrite Students Write This Way Not This Way

8. Intelligent Readings

9. Give Students Time to Think & Write

10. Assess the Writing Process

11. More Time (and savings) May Allow for Two Essays “if you want to give real weight to [student] essays, you must secure two test essays from each student with some separation in time as well as in topic, and the least possible separation in time as well as in topic, and the least possible separation in time is to have one test essay written in the morning and the other in afternoon of the same day.” “if you want to give real weight to [student] essays, you must secure two test essays from each student with some separation in time as well as in topic, and the least possible separation in time as well as in topic, and the least possible separation in time is to have one test essay written in the morning and the other in afternoon of the same day.” -- Paul Diederich of ETS The founder of modern writing assessment

12. TestWrite High-Level Workflow Create Class AdministratorStudentReader Create Test Upload StudentsView Readings & QuestionsWrite & Submit Essay [Write Comments] Assign ReadersFinalize ScoresPublish Results Reporting Register for Exam View Results [& Comments] Read & Score Essays

13. TestWrite High-Level Workflow Create Class AdministratorStudentReader

14. Admin - Create Class

16. Admin - Create Test Group

18. TestWrite High-Level Workflow Create Class AdministratorStudentReader Create Test

19. Admin - Create Test - Define

20. Admin - Create Test – Set Dates

21. Admin - Create Test – Enter Readings

24. Admin - Create Test – Enter Questions

25. TestWrite High-Level Workflow Create Class AdministratorStudentReader Create Test Upload Students

26. Admin - Upload Students

30. TestWrite High-Level Workflow Create Class AdministratorStudentReader Create Test Upload Students Register for Exam

31. Student - Login

32. Student - Registration

36. TestWrite High-Level Workflow Create Class AdministratorStudentReader Create Test Upload StudentsView Readings & Questions Register for Exam

37. Student – View Readings

39. Student – View Assignment

41. TestWrite High-Level Workflow Create Class AdministratorStudentReader Create Test Upload StudentsView Readings & Questions Write & Submit Essay Register for Exam

42. Student – Submit Essay

44. Student - Confirmation Email

45. TestWrite High-Level Workflow Create Class AdministratorStudentReader Create Test Upload StudentsView Readings & QuestionsWrite & Submit Essay Assign Readers Register for Exam

46. Admin – Assign Readers

48. TestWrite High-Level Workflow Create Class AdministratorStudentReader Create Test Upload StudentsView Readings & QuestionsWrite & Submit EssayAssign Readers Register for Exam Read & Score Essays

49. Reader – Get Essay to Read

51. Reader – Read Essay

52. Reader – Grade Essay

53. Reader – Second Reader, Second Read

54. TestWrite High-Level Workflow Create Class AdministratorStudentReader Create Test Upload StudentsView Readings & QuestionsWrite & Submit EssayAssign Readers Finalize Scores Register for Exam Read & Score Essays

55. Admin – Resolve Split Score

58. Admin – Real-time Reader Reporting

62. TestWrite High-Level Workflow Create Class AdministratorStudentReader Create Test Upload StudentsView Readings & QuestionsWrite & Submit EssayAssign ReadersFinalize Scores Publish Results Register for Exam Read & Score Essays

63. Admin – Define Result Categories

66. Admin – Set Test Cuts

67. Admin – Send to Results

68. Admin – Finalize and Publish

69. TestWrite High-Level Workflow Create Class AdministratorStudentReader Create Test Upload StudentsView Readings & QuestionsWrite & Submit EssayAssign ReadersFinalize ScoresPublish Results Register for Exam View Results Read & Score Essays

70. Student – View Results (not yet available)

71. Student – View Results

74. TestWrite High-Level Workflow Create Class AdministratorStudentReader Create Test Upload StudentsView Readings & QuestionsWrite & Submit EssayAssign ReadersFinalize ScoresPublish Results Reporting Register for Exam View Results Read & Score Essays

75. Reporting – Download Results

79. Reporting – Download Results (Excel)

80. TestWrite High-Level Workflow Create Class AdministratorStudentReader Create Test Upload StudentsView Readings & QuestionsWrite & Submit EssayAssign ReadersFinalize ScoresPublish Results Reporting Register for Exam View Results Read & Score Essays

81. Architectural Overview Web Server MS Windows 2000 / IIS MS ASP.NET C# Database MS SQL/2000 Internet Administrator Student Reader

82. TestWrite Features Automates all components of best practices in writing assessment Automates all components of best practices in writing assessment  Focusing on writing process  Allowing students modern tools  Online holistic scoring sessions with real time quality control measures

83. Digital Archives of Essays For program assessment For program assessment For portfolios For portfolios For institutional research For institutional research

84. Two Perpetual Questions What about cheating? What about cheating? Will all students have access? Will all students have access?

85. Reponses & Results

86. Feedback from Students “I am from a small town in Missouri. My writing essay was easy for me because I enjoy thinking about these things, such as philosophy and the future. I went out to my cabin and I just sat around the campfire the whole time, just enjoying the silence and writing whatever came into my head. The next day I typed it up.” (DePaul) “I am from a small town in Missouri. My writing essay was easy for me because I enjoy thinking about these things, such as philosophy and the future. I went out to my cabin and I just sat around the campfire the whole time, just enjoying the silence and writing whatever came into my head. The next day I typed it up.” (DePaul)

87. Feedback from Students “Taking the online writing tests, I found, gave me the opportunity to leave the information when I needed time to think and come back to it when I was ready. The freedom also allowed me to revise my work and not feel though my work was rushed or inhibited in any way.” (DePaul) “Taking the online writing tests, I found, gave me the opportunity to leave the information when I needed time to think and come back to it when I was ready. The freedom also allowed me to revise my work and not feel though my work was rushed or inhibited in any way.” (DePaul)

88. Benefits for Writing Faculty Allows essays and questions to reflect the university’s and program’s specific educational needs, priorities, and goals Allows essays and questions to reflect the university’s and program’s specific educational needs, priorities, and goals Maintains control of all aspects of evaluation process within the university and writing program Maintains control of all aspects of evaluation process within the university and writing program Online grading sessions maintain “staff consensus building ” benefit while potentially reducing costs Online grading sessions maintain “staff consensus building ” benefit while potentially reducing costs Reflects and measures the writing process used and valued in college Reflects and measures the writing process used and valued in college Encourages sharing of “best practices” among schools using the system Encourages sharing of “best practices” among schools using the system

89. Reliability

90. Reliability of Assessment “Faculty involved in English placement strongly feel that the Online represents much better the kind of writing students will do in our courses, and, so far, it is working equally well for a wide range of students from honors to very low developmental.” (U. of Cincinnati) “Faculty involved in English placement strongly feel that the Online represents much better the kind of writing students will do in our courses, and, so far, it is working equally well for a wide range of students from honors to very low developmental.” (U. of Cincinnati)

91. Reliability of Assessment at MIT: Fall 2002 Survey of Instructors Teaching Introductory Writing Classes 1=Should not be in class 4=Definitely should Be in class

92. Reliability of Assessment at MIT: Fall 2002 Survey of Faculty Teaching Writing-Intensive Classes

93. Reliability of Assessment: Reduction in Complaints and Appeals at MIT 1992-2003

94. Savings

95. Greater Efficiency in Scheduling and Staffing Classes

96. Saving Money: Reliability & Timeliness of Assessment Reduce Number of Sections Needed

97. Accuracy of Online Evaluation = Fewer Sections LSU (4,000 1 st Year Students) is scheduling 4 fewer sections per year because of fewer students being misplaced LSU (4,000 1 st Year Students) is scheduling 4 fewer sections per year because of fewer students being misplaced Extrapolate to UC’s nine undergraduate campuses Extrapolate to UC’s nine undergraduate campuses

98. Save Money: Typed texts (even if they are longer) cost less to grade At MIT (@$25 / hr.) At MIT (@$25 / hr.)  the average handwritten text costs $7.15 to be scored by two readers  the average typed text costs $5.40 to be scored by two readers

99. Save Money: Automatically Accommodate Students with Disabilities

100. Save Money: Eliminate Data Entry

101. Save Money: Eliminate Paperwork

102. Save Money: Avoid Hotel Expenses for Readers

103. Save Money: Eliminate Travel Expenses

104. Save Money: Reduce Expense of Feeding Readers

105. Options

106. Three Choices UC joins current consortium UC joins current consortium UC collaborates in further development and implementation through a non-profit corporation UC collaborates in further development and implementation through a non-profit corporation We all expand by partnering with for-profit company We all expand by partnering with for-profit company

107. Staying Small Benefits Benefits  Keep what we are doing  Maintain control Costs Costs  Problem with reliability  Expansion difficult  Future uncertain

108. Non-Profit Benefits Benefits  Maintain control  Potential for growth  Low economic cost Costs Costs  Too small for economies of scale  Build organization that actively functions for only a few months a year  Ensuring reliability still a problem

109. Partnering Benefits Benefits  Professional reliability  Economies of scale  Limitless expansion Costs Costs  More expensive  Some loss of control  Possible culture conflicts

110. A Possible Partner

111. Sapient | Higher Education Experience Harvard University Harvard University  Active relationship since 1997  Multiple administrative initiatives including:  ERP  Technology Strategy  Grants Management  Telecommunications Billing  Asset/Facility Management MIT MIT  Primary partner for OpenCourseWare digital publishing initiative  Program management  Technology implementation  Business process outsourcing  Application management services Open University, UK Open University, UK  Enterprise eLearning Fusion  Enterprise Architecture Planning and Implementation  Customer Relationship Management

112. Sapient | Some Support Clients Janus Reduced costs and eliminated bottlenecks through a 24x7 application that processes millions of dollars of online transactions a year Nissan Reduced turnaround time for issue resolution to 30 minutes and drove record volumes of unique visits Royal Mail Achieved cost and process efficiencies in the development and deployment of new applications and work packages Opodo 24x7 support and uptime for Europe's leading travel portal with access to 400 airlines, 30,000 hotel properties, and a fleet of over 750,000 hire cars worldwide, as well as travel insurance MIT Provide 24x7 support for the workflow application that feeds the MIT OCW web site. Lloyd’s of London 24x7 uptime, significantly reduced costs, and transitioned to new releases without impacting the business

113. TestWrite Management Team Leslie C. Perelman, Ph.D. Leslie C. Perelman, Ph.D.  Associate Dean, Undergraduate Education  Director of Writing Across the Curriculum, MIT Chris Felknor Chris Felknor  Former System Architect, Staples.com Glenn P. Strehle Glenn P. Strehle  Principal, Campus Business Advisors  Former CFO, MIT Frank Andryauskas Frank Andryauskas  Former CIO, Staples Inc.

114. Finally A more valid and reliable assessment A more valid and reliable assessment Greater faculty and student satisfaction Greater faculty and student satisfaction Significant reduction in costs Significant reduction in costs Proven and reliable platform Proven and reliable platform

115. UC System is Taking Some Hard Hits Most of these cuts are threatening one of the world’s greatest educational experiences Most of these cuts are threatening one of the world’s greatest educational experiences TestWrite offers the rare opportunity for Win – Win TestWrite offers the rare opportunity for Win – Win  Significant reduction in costs  Better educational practice