Program Assessment Kelly Aune Office of the Vice Chancellor for Academic Affairs, UH-Mānoa

Initial Questions  What have we done to our students as a result of their exposure to our program?  Problem – You should know what you did to them, or at least what you intended to do to them.  Did we do to our students what we intended to do to them?  Advantage – Requires learning outcomes.  Problem – All or nothing approach.

Better Questions  To what extent have we changed our students in the ways we intended to change them?  Advantage – Allows for range and variability of expected changes, i.e., more sensitive to change.  To what extent have we changed our students on X, Y, and Z?  Advantage – Sensitive and more specific.

Obvious First Step  If you have not done so yet, sit down as a faculty and articulate specific learning outcomes or objectives, specific effects your program is designed to produce in students.  Make certain your learning objectives lend themselves to measurement. “Be a better person” is tough to measure.

Nature of Effects  Cognitive effects – changes in:  Knowledge & beliefs  Understandings  Skills and Capabilities (e.g., calculate; introspect; evaluate; analyze; synthesize; ability to apply)  Attitudinal effects – changes in:  Predispositions; preferences; opinions.

Nature of Effects  Behavioral effects – changes in:  Various written communication skills  Various oral communication skills  Social skills  Organizational skills

Assessment  “Cheap” assessment -- e.g., paper/pencil surveys and testing.  Advantages – low dollar cost; minimal labor requirements; fast & efficient.  Can be used for low-level cognitive data.  Disadvantages – often limited to student self-report and/or attitudinal data.

Assessment  “Expensive” assessment  Disadvantages – can require significant investment to cover expenses such as incentives and labor costs.  Advantages – can provide data on higher order cognitive effects and behavioral performance indicators.

Cheap Assessment  Attitudinal surveys  Instructor evaluations  Course content evaluations  Advantages – fast & easy  Disadvantages – tells us what students think/believe about a course or instructor but not what the class has done for them.

Cheap Assessment  Self report surveys  Perceptions of capabilities  Perceptions of understandings  Perceptions of experiences – e.g., communication apprehension  Advantages – fast & easy; perceptions of experiences can be very useful.  Disadvantages – self reports of one’s capabilities and understandings are always suspect.

Better Cheap Assessment  Knowledge based assessment (a.k.a. multiple choice test).  Create a comprehensive “test” consisting of sample questions from all core or required courses.  Administer to selected groups at selected times.  Advantages – fast & easy; actually provides an indication of learning.  Disadvantages – provides a most superficial aspect of learning, a narrow bandwidth of cognitive change.

Expensive Assessment  Recording/Assessment of behavior  Recording of speech/presentation for subsequent analysis by coders.  Written responses to stimuli materials for subsequent analysis by coders.  Portfolio collection for subsequent analysis by judges/experts.  Recording of performance for subsequent assessment by judges/experts.  Track your alumni

Expensive Assessment  Advantages – provides actual performance indicators.  Disadvantages – time intensive and labor intensive; costs may be generated for incentives (for students); pay for coders/judges; coders need to be trained to ensure reliability and validity of assessments.

Design  Single Data Set  Advantages – fast & easy; yields one set of numbers.  Disadvantages – what can that single set of numbers tell you? How do you interpret those numbers?  Comparison Data Sets – Time 1 vs. Time 2 or comparison group.  Disadvantages – more labor intensive; analysis more complicated.  Advantages – allows comparison of data relative to learning stimuli.

What To Do With Your Data  Report Descriptive Statistics  Advantages – fast & easy; requires little expertise to compile.  Disadvantages – how do you argue for effectiveness?  Report Statistical Comparisons  Advantages – allows for strong arguments concerning learning outcomes.  Disadvantages – requires some expertise.

Closing the Loop  Interpreting the results  What do the data tell you about the effectiveness of your program?  Altering your program  Adjust curriculum content, sequencing of courses, prerequisites, etc., in response to results.  Test Again!

Final Thoughts  Forget WASC  Yes, really truly forget WASC!  Do this for yourselves. The only way to establish the “culture of evidence” that WASC is looking for is to find value in the assessment process. Do it thoughtfully, with scholarly curiosity, with an expectation of value and you will find this a rewarding process.