1. Transformative Assessment: A Progress Report from NLII Vicki Suter, EDUCAUSE Stephen C. Ehrmann, TLT Group Gary Brown, WSU Joan Lippincott, CNI

2. Copyright Statement Copyright Gary Brown, Stephen Charles Ehrmann, and Joan K. Lippincott, 2002. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

3. Outline I.Introduction and some guidelines for assessment– Steve Ehrmann, The TLT Group’s Flashlight Program II.Results of NLII Online workshop and a rubric for assessing transformative assessment – Gary Brown, Washington State University III.Making changes – institutional and team issues – Joan Lippincott, Coalition for Networked Information

4. Steps to Date Focus Workshop in Denver, March, 2002 26 institutions; 11 teams and 15 ‘singles’ Online workshop in May-May, 2002 11 institutions started; about 5 finished Largest teams from U Washington, U Maryland, Colorado system Gary and I will be talking about what we’ve learned about evaluation strategies Joan will discuss the team and institutional strategies and concerns

5. Seven Dangerous Questions Using IT and Assessment to Transform Academic Programs Stephen C. Ehrmann, Ph.D

6. Thanks! 330+ institutional subscribers to TLT Group services EDUCAUSE/NLII

7. Outline I.“Transformation?” II.Seven Dangerous Assessment Questions III.“Transformative Assessment?”Transformative Assessment IV.How can assessment promote improvement? V.The READY web site

8. The Vision (1960 – present) Technology, properly used, can improve education and its outcomes New participants (learners, experts) New materials and resources More options, responsibilities for learners, instructors, staff New ways of organizing & supporting learning Therefore with every passing decade we ought to be able to make further improvements in education and its outcomes. Can we see them? Can we even make improvements if we can’t see them?

9. Def’n of Transformation? Important, qualitative change in a “triad”: I.T. Goal - Outcome Activity – structure in use {

10. Not Enough Outcome/GoalActivity- Structure Technology(s) and other needed elements Whatever Infrastructure used a lot, with great satisfaction Infrastructure Tech support

11. Not Enough, Either Reference, “What Outcomes Assessment Misses”What Outcomes Assessment Misses Outcome/GoalActivity- Structure Technology(s) and other needed elements Improve learning, retention, etc. ? Infrastructure Tech support

12. Example of a Transformation Outcome/GoalActivity- Structure Technology(s) and other needed elements Better learning outcomes (e.g., higher ‘scores’, better retention) 7 Principles7 Principles on the upswing across the curriculum; Wider variety of learners, teachers, other resources Communications; Productivity tools Skilled faculty, spread of low threshold activities (e.g.,, managing collaboration)

13. Another Transformation Outcome/GoalActivity- Structure Technology(s) and other needed elements Community, during & after college Plentiful interaction, formal and informal, including outsiders Virtual spaces; communications; directories, etc. Staff who are good at collaboration; outside partnerships

14. And Another Outcome/GoalActivity- Structure Technology(s) and other needed elements Speed to degree; access Hybrid courses; Altered recruiting and student info CMS, e-mail, Web, etc. Help in redesigning courses so they take more advantage of online; less need for f2f

15. Assessment isn’t for Wimps “Dangerous questions” are: Risky to ask! Even riskier not to ask! (especially if other institutions or your constituents are beginning to ask) [hypothesis]

16. Seven Dangerous Questions 1.WHICH educational outcomes are we trying improve (baseline)? 2.We can’t change outcomes without changing what we do. So far, is there enough change in how people teach and learn? 3.Where there isn’t, what’s standing in our way? 4.Is IT useful in changing those T&L activities? 5.Where IT hasn’t been used that way, why not? 6.What are the costs and stresses of key activities? Can they be reduced? 7.Now that years have passed, what have we achieved (compare with #1)

17. TA & Alignment Programmatic Outcome (who learns, what they learned, etc.) & Strategy Assessment – Use of Data Mission Statement Budget Process Program Dev Staff Dev Admissions Legislative/Alumni Capital plans (etc.)

18. Assessment’s Powers 1.The discussion of how to collect data can help to make discussion of goals and strategies less abstract, agreements more meaningful. Medium for people to talk with one another. 2.Discussion of alignment can help in those ways, too. 3.Periodic findings can refocus attention 4.Spotlight problems as well as achievements in order to attract resources 5.Steering, e.g., grab hidden opportunities, reduce or remove barriers

19. Getting READY Check out the READY website at EDUCAUSE, section on transformative assessmentREADY website at EDUCAUSE Self-assessment checklist of elements needed if your institution is to use data to help accomplish any transformative change

20. Summary: What Does TA/TLT Look Like?  Growing institution sense of direction(s) of progress for using IT (Triads)  Good infrastructure to support assessment at many levels  Already using data to help move in that direction (LOTS of relevant studies feeding on one another)  Alignment (two-way) of assessment with other functions (budgeting, staff development, fund-raising, recruitment, etc.)