1. Program Outcomes Assessment Techniques: Overview and Examples Steve Steele, Consultant Faculty Online Technology Training Consortium (FOTTC) Applied Sociologist Anne Arundel Community College sfsteele@mail.aacc.cc.md.us 410-541-2369fsteele@mail.aacc.cc.md.us “I have never let my schooling interfere with my education.” - Mark Twain

2. Common Conceptualizations of Evaluation/assessment Is it viewed a “a reign of terror?” Is it “stuck on like a barnacle”; an afterthought Is it an ongoing process in itself? An organized entity in itself Part of a management information system

3. Where are We Going Today? Evaluation? Assessment? Definitions Reasons For Doing This Placement of Evaluation/assessment The Dynamic Fit Between Planning, Evaluation/assessment and Doing In the Improvement Process Examples Summary: Strengths and Weaknesses

4. Evaluation? Assessment? Definitions The systematic assessment of the operation and/or outcomes of a program or policy, compared to explicit or implicit standards, in order to contribute to the improvement of the program or policy. (Weiss, 1998) “The key sense of the term... refers to the process of determining the merit, worth, or value of something...” (Scriven, 1991, pp. 139-141) “evaluation/assessment research... refers to research purpose rather than research method. Babbie,1989, 326)

5. Critical Components of Definitions A systematic process A judgment will likely be made Standards for judgment, for success Intervention Orientation toward Improvement?

6. Reasons For Doing This Internal Clients/Stakeholders The President The Dean Your Chairman Your students You! External Clients/Stakeholders Societal Middle States Professional Granting agencies

7. Reasons For Doing This Have you considered the client’s “corporate culture” when constructing this project/design? Are you sure that you have focused on the client’s needs for ALL relevant clients and stakeholders?

8. Placement of Evaluation/assessment Think of evaluation/assessment before the program, not after. Integrate evaluation/assessment into the grant proposal, plan or program. Use evaluative data “for all it’s worth!” Consider evaluation/assessment as constructive criticism, rather than “doing time!”

9. The Dynamic Fit Between Planning, Evaluation/assessment and Doing In the Improvement Process HERE!! Based on E. Deming’s work

10. “Plan” and Define it.

11. What is it? Intervention Definitions

12. Example: A Distance Learning Business Plan http://www.aacc.cc.md.us/ola/ http://www.aacc.cc.md.us/ola/Assessment/BusPlanNov98.html

13. Goals and Objectives... http://www.aacc.cc.md.us/ola/Assessment/BusPlanNov98.html#To implement

15. “Check” Some Approaches to “Checking” Keeping track... is it operating (not necessarily successfully!) Forming something, improving something Sum up, impact… did it work? Based on Michael Scriven’s Formative & Summative Evaluation

16. Some Tools for Tracking, Forming and Summing Frequency of something Rates or percentages of something Level of Need Opinions, behaviors, feelings Indicators, Estimates and Forecasts Survey Use patterns Focus groups

17. A Plan Level Example #1 Wells, Kepner, Barnes, Steele AACC

18. Toward an Evaluation/assessment Plan http://www.aacc.cc.md.us/ola/Assessment/Baldrige.html

20. http://www.aacc.cc.md.us/ola/Assessment/assessmentStudentStakeholder.htm Scroll

22. Measures…. http://www.aacc.cc.md.us/ola/Assessment/studentsurveyresults.htm

23. A Plan Level Example #2 Barnes, Warner, Steele AACC

24. Some Tools for Tracking, Forming and Summing Frequency of something Rates or percentages of something Level of Need Opinions, behaviors, feelings Indicators, Estimates and Forecasts Survey Use patterns Focus groups

25. On Line Enrollment

26. A Course Level Example #3 Steele AACC

27. Some Formative Checks on Soc111 Online

30. How Many Links on the Module Did You Visit (all modules, N=188 response)?

31. Rate this Module's Learning Value (all modules, N=188 response)?

32. Rate this Module's Interest Value (all modules, N=188 responses)?

33. Summary: Strengths and Weaknesses Strengths Opportunity for continuos growth Possibility for continuous improvement Makes planning useful Improves program and course design Weaknesses Resource and infrastructure commitment Requires a variety of skills Difficult to stay “tangent” Demands attention