1. Assessing for Transformation Gary Brown The Center for Teaching, Learning & Technology

2. We should not expect the guidance for change of this magnitude—in institutional culture and values—to come from the faculty ranks. After all, faculty are deeply rooted in the traditional values of higher education. Fundamentally, this is a leadership issue. --Carol Barone, Vice President Educause

3. Agenda Assumptions—the need for transformation To arms, to arms… About those jobs What Transformation is NOT What Transformative Assessment is NOT Stories Goals, Activities, & Processes (GAPs) Inquisitions and Assessments Critical Thinking—the rubric paradigm The Sleeping Bear & Principles of Transformative Assessment TAPS

4. To Arms, To Arms… PHOENIX Phoenix Online is growing faster than any of the university's 40 physical campuses 1,700 online support specialists 7,000 mostly part-time faculty 49,400 students 70% increase $64.3 million in net income

5. Phoenix —from the Chronicle “Some critics note that Phoenix strips faculty members of their central role in higher education.” “One of the greatest risks we face as a nation in the growth of new educational providers is the unbundling of teaching, research, and service functions” —Art Levine However, counters Phoenix, “Professors at traditional universities who attempt online education are learning as they go, and often give students a bad experience as a result.”

6. Cost The USA currently has 3,500 colleges and universities with an enrollment of 14 million students. We spend $175 billion on education. $12,500 per student But now, eleven International Mega- universities serve 3 million students.

7. Corporate Tax Support

8. Job Growth: 1992-2005* Predicted Percentage Increase… *US Department of Labor Statistics (1993)

9. Job Growth: 1992-2005* Or in Total Numbers *US Department of Labor Statistics (1993)

10. Job Market Projections* *US Department of Labor Statistics (1993)

11. The Real Story We need: 9 cashiers for every technical worker. 1.5 janitors for all the lawyers, accountants investment bankers, stock brokers, & computer programmers combined. “The projected shift … in educational requirements… can be accomplished if those entering the labor force have…one fourth grade level more education than those retiring from the labor force.” Economic Policy Institute

12. College Graduates @ Work *US Department of Labor Statistics (1993)

13. Already, In Fact… % College Graduates working in Blue Collar Jobs *US Department of Labor Statistics (1993)

14. What Transformation Is NOT

15. Attrition Rates in WSU Distance Courses %

16. Student Enrollments in WSU Online Learning Spaces

17. Transformation?

18. What Transformative Assessment is NOT

19. Evaluations as Barriers to Improvement Many evaluation instruments are subtly biased in favor of traditional instruction. They often penalize innovators, measuring what faculty aren’t doing but failing to measure what they are doing.

20. Purpose of Evaluation is Not Clear Most evaluation focuses on personnel decisions. Inconsistent interpretation and use of evaluation results. Little focus on areas where that might best benefit from feedback Evaluation usually occurs at the very end of a semester —Too late for change

21. Students Are in the Dark, Too The traditional evaluation process is not designed to help students become more involved with what will really help them learn. Students don’t take evaluations seriously.

22. Teaching-Learning Practices are NOT Linked to Learning Course evaluations rarely focus on practices that have been shown to produce better learning outcomes. Assessment rarely improves learning.

23. No Link Between Evaluation & Faculty Development Inadequate support for addressing results. Literature from past 10-15 years emphasizes staff development as key to effective evaluation practices that promote teacher growth and improvement” (1997, Annunziata in Stronge, pp 289) Administrators frequently calculate an average score for all evaluation items and then rank faculty. The method does not foster dialogue among faculty about good practices.

24. The Goals, Activities, and Practices Invite faculty to join in the process of formulating the assessment. A series of three short, online surveys designed to provide formative assessment: one instructor survey and two student surveys.

25. The 3 Survey Goals 1.To help faculty gather useful data about their teaching goals, values and strategies… 2.To help faculty learn how those strategies address and influence their students’ goals, values, and learning behaviors… 3.And to examine the interaction between goals and practices that faculty might share with each other —and so establish a culture of evidence.

26. GAPs: Encouraging Participation Brown Bags - to initiate the conversation on assessing teaching and learning. We emailed each faculty approximately 10 times Student surveys were linked to course sites of instructors who answered the instructor survey. We analyzed and returned results promptly... Regular updates were sent to instructors with their class results and overall results. We offered and have co-authored papers with faculty... We have talked with chairs, deans, and the assessment coordinator’s to include GAPS in teaching portfolios. We wore out our shoes…

27. Sample Findings Significant mismatch between faculty and student goals Online courses are significantly more effective than video based courses Designed courses are significantly more likely than courses not formally designed to evidence principles of good practice Faculty motivation predicts perceptions of efficacy Faculty who want to keep abreast of the scholarship of T & L report that online learning is positive Faculty who teach online primarily for the money … Student age and gender predict perceptions of testing efficacy

28. Time & Cost Estimated Hours

29. The Assessment Gold Standard Participants Who Used GAPs Data to Transform….

30. However…. One of the two programs endured a budget related, faculty initiated inquisition….. Demonstrated, by virtue of GAPS, systematic, formative data gathering and responsiveness to that data… Demonstrated evidence of good practice. Demonstrated the “creative use of technology” and gains in critical information literacy Demonstrated the program was cost effective The program improved freshman retention Demonstrated the program improved student grades, including “special admits” expected to struggle.

31. The Learning Context: Student Performance @ WSU Cum GPA by Admissions Quartiles

32. Which has resulted in…. The faculty committee that initiated the inquiry was commended by administration for its attentiveness to issues of quality…. They have subsequently been invited to examine other programs… Which has transformed the CTLT We are seeing the inklings of a culture of evidence sprouting in a rough, research dominated terrain…

33. small steps….

34. Guide faculty grading Guide student learning Provide measures of growth The Rubric Paradigm

35. Dimensions of Critical Thinking 1.Identifies and summarizes the problem/question at issue (and/or the source's position). 2.Identifies and presents the STUDENT’S OWN perspective and position as it is important to the analysis of the issue. 3.Identifies and considers OTHER salient perspectives and positions that are important to the analysis of the issue. 4.Identifies and assesses the key assumptions. 5.Identifies and assesses the quality of supporting data/evidence and provides additional data/evidence related to the issue. 6.Identifies and considers the influence of the context * on the issue. 7.Identifies and assesses conclusions, implications and consequences.

36. Critical Thinking and Measures of Growth Emerging____________________________ Mastering Identifies and summarizes the problem/question at issue (and/or the source's position). Does not identify and summarize the problem, is confused or identifies a different and inappropriate problem. Does not identify or is confused by the issue, or represents the issue inaccurately. Identifies the main problem and subsidiary, embedded, or implicit aspects of the problem, and identifies them clearly, addressing their relationships to each other. Identifies not only the basics of the issue, but recognizes nuances of the issue.

38. Critical Thinking: 8 Courses: 4 with CT, 4 w/o CT

40. Critical Thinking: One course—two semesters

41. Faculty Development The critical thinking rubric was valuable for helping faculty articulate their goals and communicate expectations to students. Faculty who used the rubric were enthusiastic and expressed plans to integrate the rubric more intensively in future courses.

42. Critical Thinking Study — Results Gains in courses when rubric is used—when the faculty in this project integrated the WSU Critical Thinking Rubric into their instruction and assessment, evidence of student gains in critical thinking increased dramatically. Gains from freshmen to junior years—Critical thinking was significantly higher among juniors than among freshmen. But even the writing of juniors had only a mean of 3.1 on a 6 point scale.

43. Additional Findings & Implications The dimension of least gain was in students’ abilities to articulate their own viewpoints. The greatest gains by juniors reflect improved abilities to analyze issues from multiple perspectives. Comparisons to WSU’s writing assessment— As critical thinking scores rise, writing placement scores and portfolio exam scores sink... The faculty questionnaire revealed a focus on grading over fostering critical thinking for broader life-long learning.

44. Transformation?

45. The Sleeping Bear Those most oblivious of the teeth are often the first to fodder… The growling bear will elicit benchmarks & comparisons The bear will feast on standardization But bears are omnivorous…

46. Emerging Principles of Transformative Assessment Institutional leadership is imperative Assessment focuses on institutional efforts to provide students with rich learning experiences Assessment includes student reports of their own increasingly unique experiences Qualitative measures are valued and may be supported by quantitative measures Emphasizes student learning defined by development and change over time Dissemination engages the responsibility for shaping as well as reflecting society needs “High Standards” is not the same as Standardization

47. Dimensions of Transformation 1.Purpose 2.Data Acquisition 3.Application 4.Dissemination

48. Criteria for Rating Transformation Rater:Project: *Faculty*Designer *Student*Assessment Specialist *Community Colleague*Administrator *Other _________ Rating Assessment Purpose Data & Data Acquisition Application Dissemination Bonus—Faculty Rewarded for Assessment Total Score Comments: The Scoring Form

49. U#1U#2U#3U#4U#5Dimension Average Purpose 33.13.643 3.34 Data 2.43.53.83.53 3.24 Application 2.53.43.73.54.5 3.52 Dissemination 3.333.524.5 3.26 University Average 2.83.33.73.33.8 3.34 3.38 Pilot Findings

51. Key Comments U#1 Doesn’t stretch student learning or acknowledge what students may bring. Focus satisfies industry’s demand for skills. While faculty are included in this process, audience is clearly the upper echelon of the institution and future employers.

52. Key Comments U#2 Not many rewards for assessment. No mention of disseminating results beyond unit; no feedback to faculty and students. Limited to classes.

53. Key Comments U#3 Assesses key concern: learning to learn. No plan to apply or fund results of findings. Faculty support neglected. Feedback to institution and external agencies is minimal—how are constituents engaged in process?

54. Key Comments U#4 Design for infrastructure for hybrid classes—assumptions that tech alone equates with transformation

55. Key Comments U#5 Good start but…