1. ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting professor of english, drexel university founder and president, subjective metrics, inc. www.gowaypoint.com ajmccann@subjectivemetrics.com 215.713.9393

2. ©2003-2005 Subjective Metrics, Inc. 05.11.20052 Agenda Background Introduce waypoint –Brief overview Drexel applications Live Demonstration –Evaluate –Manage –Libraries

3. ©2003-2005 Subjective Metrics, Inc. 05.11.20053 Problem: Feedback Handwritten and/or manually typed “Lost” Little accountability Consistency issues No data Needed: not AI – but a technological tool

4. ©2003-2005 Subjective Metrics, Inc. 05.11.20054 Waypoint: Evaluation Tool Encourages pre-written feedback: clearer, more detailed explanations of key concepts Facilitates sharing of assessments amongst instructors Quantifies evaluations by skill Archives all feedback Web-based: cross-platform backed up zero maintenance

5. ©2003-2005 Subjective Metrics, Inc. 05.11.20055 Waypoint Process

6. ©2003-2005 Subjective Metrics, Inc. 05.11.20056 Develop Assessment

7. ©2003-2005 Subjective Metrics, Inc. 05.11.20057 Develop Assessment

8. ©2003-2005 Subjective Metrics, Inc. 05.11.20058 Receive Paper or Exam (Hard copy shown here)

9. ©2003-2005 Subjective Metrics, Inc. 05.11.20059 Receive Paper or Exam (Hard copy shown here)

10. ©2003-2005 Subjective Metrics, Inc. 05.11.200510 Evaluate Against Skills

11. ©2003-2005 Subjective Metrics, Inc. 05.11.200511 Evaluate Against Skills

12. ©2003-2005 Subjective Metrics, Inc. 05.11.200512 Respond to Student

13. ©2003-2005 Subjective Metrics, Inc. 05.11.200513 Respond to Student

14. ©2003-2005 Subjective Metrics, Inc. 05.11.200514 Snapshot Analysis

15. ©2003-2005 Subjective Metrics, Inc. 05.11.200515 Snapshot Analysis

16. ©2003-2005 Subjective Metrics, Inc. 05.11.200516 Longitudinal Cohort Analysis

17. ©2003-2005 Subjective Metrics, Inc. 05.11.200517 Process Summary

18. ©2003-2005 Subjective Metrics, Inc. 05.11.200518 Drexel Applications tDec Humanities (Dr. Valarie Arms) –Evaluation of all major writing assignments (600 students) Writing Center (Harriet Millan) –Customized assessments for WITs College of Engineering (Kevin Scoles & Adam Fontecchio) –Evaluation of lab reports with WITs and TAs College of Business (Frank Linnehan) –Structured response to student writing & accreditation data generation Engineering Management (Mike Scheuerman) –Peer review

19. ©2003-2005 Subjective Metrics, Inc. 05.11.200519 Demonstration www.gowaypoint.com Evaluate –“Final Report” Peer Review –“Final Report” Manage –Quantification –Sorting –Data analysis

20. ©2003-2005 Subjective Metrics, Inc. 05.11.200520 Additional Features Self-assessment Collaborative assessment –Multiple instructors can contribute to evaluation

21. ©2003-2005 Subjective Metrics, Inc. 05.11.200521 Questions and Discussion

22. ©2003-2005 Subjective Metrics, Inc. 05.11.200522 Testimonials “I've been doing this for twenty years and have my own system of grading and commentary…and found little need to improve. And then Waypoint came along. Somehow--and I really can't even explain it--but my grading time has been cut down by more than half and my students are actually thanking me for the in- depth commentary.” –Professor Ken Bingham “Usually I have trouble criticizing a peer’s paper if I’m not given certain criteria to judge. During this peer review, I was actually focused and excited about judging a peer’s paper.” –Erin Williams, COE 2008 “If this peer review program was available in high school, I would have probably done a lot better in English.” –Ed Itaas, COE 2008

23. ©2003-2005 Subjective Metrics, Inc. 05.11.200523 Testimonials “An innovative teacher-friendly, student-friendly, efficient approach to grading writing—the most creative and time-saving method for evaluating writing I have ever found or used.” –Gayle, a high school English teacher with 35 years’ experience “I don’t know how I ran a writing program for three years without it.” –Harriet Millan, Director of the University Writing Program

24. ©2003-2005 Subjective Metrics, Inc. 05.11.200524 Quantitative Data Anonymous survey of Peer Review process –114 freshman engineers: 51 use waypoint, 63 the ‘old’ method 1-5 Scale (1 – strongly disagree, 5 – strongly agree) wp (N=51) Mean ‘old’ (N=63) Mean P value (t-test, 95% CL) 1. I received clear and helpful criticism of my draft. 3.903.52p<0.025 2. Evaluating other students’ papers helped me better understand the assignment and the play. 3.863.48p<0.02 3. I made significant changes to my first draft based on peer review feedback. 3.813.25p<0.005 4. I made significant changes to my first draft independent of peer review feedback. 3.493.43p=0.38