2. Department of Humanities College of Sciences and Liberal Arts Writing Program Assessment at New Jersey Institute of Technology Carol Siri Johnson Associate Professor and Chair

3. Outline of Presentation Personal Introduction New Jersey Institute of Technology ABET Writing / Technical Writing Assessment Using Online Portfolios Using Analytic Scoring Examples of Data Analysis Results

4. Personal Introduction Mount Holyoke College English Ph.D., but few jobs New – Technical Writing Field Tenure-track Position at NJIT Chair, Research EA and TC History

5. My Assessment History Knew nothing about it Disliked standardized tests Was unconvinced of its value Then I saw a best-paper reading The Power of Assessment

6. Assessment at NJIT Department of Humanities active in assessment for ~20 years Previous Chair, NJCBSPT Norbert Elliot and ETS Course / Program Assessment Analytic Online Portfolio Assessment

7. On a Scale – Norbert Elliot

8. What is Assessment? ●Giving a student a grade? ●End-of-semester course evaluations ? ●Faculty review of portfolios? ●Student surveys? ●The chair’s opinion?

9. What is Assessment? ●For Teachers, GRADES ●For Departments, COMMON EXAMS ●For Administration, COURSE EVALUATIONS ●For Learning, OUTCOMES – stating and reaching curricular goals in Courses and Programs

10. Assessment Can Be: Program assessment Curricular assessment Method of automatic updating Method of teaching collaboration Gathering statistics for multiple purposes.

11. ABET 2000- Accreditation Board for Engineering and Technology

12. ABET’s 8 General Criteria:

13. #4 Continuous Improvement “The results of these evaluations must be systematically utilized as input for the continuous improvement of the program.”

14. Outcomes Assessment Cycle

15. NJIT Core Curriculum – GURs (General University Requirements) ● English Composition – 6 credits ● Cultural History – 6 credits ● Humanities Electives – 6 credits ● Senior Seminar – 3 credits

16. GURs Outcomes Assessment ●English Composition – every semester ●Cultural History – every other spring ●Humanities Electives – every other fall ●Senior Seminar – every semester

17. Assessment Basics – Validity The issue of “validity” means we are measuring what we want to measure; the test is appropriate to the subject The problem with writing assessment has been using multiple choice tests only – they alone are not valid as a measure of writing Multiple measures are often the best

18. Assessment Basics – “Associative” Validity Correlating Multiple Inputs: Machine-scored essay scores Portfolio scores Grade in course Student GPA

19. Assessment Basics – Reliability Can we be sure that the measurement, if taken again, would yield the same results? Inter-reader Reliability (scored by two separate readers, discrepancies resolved by a third) Content Reliability (portfolio contains work from entire semester)

20. Two Important Steps: Decide on and define goals or criteriaDecide on and define goals or criteria Develop method to assess outcomes (goals/criteria)Develop method to assess outcomes (goals/criteria)

21. Large Scale Writing Assessment Initiatives ●1950sETS – Holistic Scoring ●1970sCalifornia (Edward White) ●1980sNew York (CUNY) ●1990sNew Jersey (NJCBSPT) ●2000sNJIT – Program Assessment

22. Example: Technical Communication Analytic Online Portfolio Assessment Established Criteria by “Online Delphi” – email exchanges by facultyEstablished Criteria by “Online Delphi” – email exchanges by faculty 2004 – Failed effort (network down)2004 – Failed effort (network down) 2005-2009 – Successful Analytic Online Portfolio Assessment, every semester2005-2009 – Successful Analytic Online Portfolio Assessment, every semester 2009-Present – once per year2009-Present – once per year

23. Technical Communication Cycle

24. Using Online Portfolios

25. Procedure 1.Have Students Make Online Portfolios with Examples of the Main Assignments 2.Collect URLs 3.Select Random Sampling (~60-100) 4.All Instructors Meet 5.~1 Hour Calibration with Sample Portfolios where instructors discuss scoring processes 6.~3 Hours Scoring 7.Assessment Director Conducts Necessary Adjudications 8.Enter Scores into Excel or SPSS Dataset

26. Old Humanities Rubric

27. Old Score Sheets

28. Revised Score Sheet

29. Sample Data Collection

30. A Community of Assessment Shared Instructional Content Self-Evaluation and Norming Curricular Goals Evaluation Continual Improvement (ABET) Interesting Statistics