Program Outcomes Assessment Techniques: Overview and Examples Steve Steele, Consultant Faculty Online Technology Training Consortium (FOTTC) Applied Sociologist Anne Arundel Community College “I have never let my schooling interfere with my education.” - Mark Twain

Common Conceptualizations of Evaluation/assessment Is it viewed a “a reign of terror?” Is it “stuck on like a barnacle”; an afterthought Is it an ongoing process in itself? An organized entity in itself Part of a management information system

Where are We Going Today? Evaluation? Assessment? Definitions Reasons For Doing This Placement of Evaluation/assessment The Dynamic Fit Between Planning, Evaluation/assessment and Doing In the Improvement Process Examples Summary: Strengths and Weaknesses

Evaluation? Assessment? Definitions The systematic assessment of the operation and/or outcomes of a program or policy, compared to explicit or implicit standards, in order to contribute to the improvement of the program or policy. (Weiss, 1998) “The key sense of the term... refers to the process of determining the merit, worth, or value of something...” (Scriven, 1991, pp ) “evaluation/assessment research... refers to research purpose rather than research method. Babbie,1989, 326)

Critical Components of Definitions A systematic process A judgment will likely be made Standards for judgment, for success Intervention Orientation toward Improvement?

Reasons For Doing This Internal Clients/Stakeholders The President The Dean Your Chairman Your students You! External Clients/Stakeholders Societal Middle States Professional Granting agencies

Reasons For Doing This Have you considered the client’s “corporate culture” when constructing this project/design? Are you sure that you have focused on the client’s needs for ALL relevant clients and stakeholders?

Placement of Evaluation/assessment Think of evaluation/assessment before the program, not after. Integrate evaluation/assessment into the grant proposal, plan or program. Use evaluative data “for all it’s worth!” Consider evaluation/assessment as constructive criticism, rather than “doing time!”

The Dynamic Fit Between Planning, Evaluation/assessment and Doing In the Improvement Process HERE!! Based on E. Deming’s work

“Plan” and Define it.

What is it? Intervention Definitions

Example: A Distance Learning Business Plan

Goals and Objectives... implement

“Check” Some Approaches to “Checking” Keeping track... is it operating (not necessarily successfully!) Forming something, improving something Sum up, impact… did it work? Based on Michael Scriven’s Formative & Summative Evaluation

Some Tools for Tracking, Forming and Summing Frequency of something Rates or percentages of something Level of Need Opinions, behaviors, feelings Indicators, Estimates and Forecasts Survey Use patterns Focus groups

A Plan Level Example #1 Wells, Kepner, Barnes, Steele AACC

Toward an Evaluation/assessment Plan

Scroll

Measures….

A Plan Level Example #2 Barnes, Warner, Steele AACC

Some Tools for Tracking, Forming and Summing Frequency of something Rates or percentages of something Level of Need Opinions, behaviors, feelings Indicators, Estimates and Forecasts Survey Use patterns Focus groups

On Line Enrollment

A Course Level Example #3 Steele AACC

Some Formative Checks on Soc111 Online

How Many Links on the Module Did You Visit (all modules, N=188 response)?

Rate this Module's Learning Value (all modules, N=188 response)?

Rate this Module's Interest Value (all modules, N=188 responses)?

Summary: Strengths and Weaknesses Strengths Opportunity for continuos growth Possibility for continuous improvement Makes planning useful Improves program and course design Weaknesses Resource and infrastructure commitment Requires a variety of skills Difficult to stay “tangent” Demands attention