Making Student Services Assessment Meaningful Joanna M. Oxendine, MEd

Overview  Intentionality: The Meaningful-Assessment Foundation  Outcome Articulation: An Assessment Tipping Point  Authentic—and Manageable!—Assessment  Responding Intentionally: Meaningful Change and Continual Improvement

The Intentional Assessment Cycle Articulate Intended Outcomes Work Toward Intended Outcomes Gather Evidence Determine Extent to Which Outcomes Were Met Plan for Intentional Improvements

The Culture of Intentionality  Is student-centered  Seeks information about how well students are learning and/or how well various areas of the college are supporting the student experience  Reflects on what we teach or do and how we teach or do it  Accepts (some) responsibility for student learning and the student experience  Experiments with new strategies for student success Students become the primary focus of the assessment process

A Process of Questions 1. Why are you here? What need do you meet and for whom? (Mission) 2. What do you hope to accomplish? (Goals) 3. What measurable or observable thing(s) do you expect students to do after interacting with your office/department? (Student Learning Outcomes) What measurable or observable thing(s) does your office/department do to support the institution’s learning environment? (Service Area Outcomes) 4. Where/how will students they learn it? How will you do it? (Mapping of Activities to Outcomes) 5. How will you know if and how well students learned what you expected them to learn? How will you know if and how well you supported the learning environment? (Alignment of Measures to Outcomes) 6. What does your evidence tell you? How can you use it to make appropriate decisions for continual improvement? (Closing the Loop/Making Meaningful Change) The Culture of Intentionality opens a clear approach to program design and assessment.

It all begins with intentionality… and mutuality… and hinges upon well-articulated outcomes.

Outcome Articulation: AN ASSESSMENT TIPPING POINT

Articulating Clear Outcomes  Begin with the end in mind.  What is it, exactly, you want your students to be able to do?  What is it, exactly, you want your office, department, or program to do with regards to administrative process or procedure?  Experiment with articulation…  SUBJECT - VERB - OBJECT Who will do what?  …until you get it right.  Does your outcome accurately reflect the end you have in mind?  Does it utilize active verbs that clearly articulate the expected action?  Is it specific?  Is it measurable or observable?

Features of Effective Outcomes  Clear and succinct  Under the control of the responsible unit  Central to the unit’s mission and goals  Lend themselves to improvement  Focus on meaningful results  Focus on measuring levels of effectiveness  Utilize active verbs

Common Outcome Pitfalls  Wordy  Vague  Stacked or double-barreled  Unmeasurable  Too many outcomes/unmanageable

A Reminder Outcome articulation:  Directs efforts to evaluate how successful activities have been.  Helps to identify what kind of information to collect. Outcomes drive measures. The clearer, more descriptive the outcome (read “the verb used in the outcome”), the easier it becomes to authentically assess the outcome. The easier it is to authentically assess the outcome, the more useful the data becomes in making meaningful change.

Operational Verbs Verbs describing student acquisition and preparation of tools, materials, and texts of various types (including digital and archival) Verbs indicating what students do to certify information, materials, texts, etc. Verbs indicating the modes of student characterization of the objects of knowledge or materials of production, performance, exhibit Verbs describing what students do in processing data and allied information access, acquire, collect, accumulate, extract, gather, locate, obtain, retrieve cite, document, record, reference, source categorize, classify, define, describe, determine, frame, identify, prioritize, specify calculate, determine, estimate, manipulate, measure, solve, test Verbs further describing the ways in which students format data, information, materials Verbs describing what students do in explaining a position, creation, set of observations, or a text Verbs falling under the cognitive activities we group under “analyze” Verbs describing what students do when they “inquire” arrange, assemble, collate, organize, sort articulate, clarify, explicate, illustrate, interpret, outline, translate, elaborate, elucidate compare, contrast, differentiate, distinguish, formulate, map, match, equate examine, experiment, explore, hypothesize, investigate, research, test Verbs describing what students do when they combine ideas, materials, observations Verbs that describe what students do in various forms of “making” Verbs that describe the various ways in which students utilize the materials of learning Verbs that describe various executive functions students perform assimilate, consolidate, merge, connect, integrate, link, synthesize, summarize build, compose, construct, craft, create, design, develop, generate, model, shape, simulate apply, carry out, conduct, demonstrate, employ, implement, perform, produce, use operate, administer, control, coordinate, engage, lead, maintain, manage, navigate, optimize, plan Verbs that describe forms of deliberative activity in which students engage Verbs that indicate how students valuate objects, experiences, texts, productions, etc. Verbs that reference the types of communication in which we ask students to engage Verbs that describe what students do in rethinking or reconstructing argue, challenge, debate, defend, justify, resolve, dispute, advocate, persuade audit, appraise, assess, evaluate, judge, rank report, edit, encode/decode, pantomime (v), map, display, draw/ diagram, collaborate, contribute, negotiate, feed back accommodate, adapt, adjust, improve, modify, refine, reflect, review Adapted from Adelman, C. (2015). To imagine a verb: The language and syntax of learning outcomes statements. (Occasional Paper No. 24). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment.

Authentic—and Manageable!— Assessment

Identifying Measures to Gather Evidence In general, assessment should be:  Meaningful  Measurable  Manageable Clear and careful outcome articulation ensures it’s measureable. The critical analysis and use of evidence for continual improvement strategies ensures it’s meaningful. But how can we make assessment more manageable? Let your outcomes be your guide…

Identifying Measures to Gather Evidence What, exactly, do you intend students be able to do? What, exactly, do you intend to do? How do you know they’re able to do it (and how well)? How do you know you did it (and how well)? Take inventory! What are you already doing that provides evidence as to whether or not your articulated outcomes are being met? Can you tweak any of your existing tools to serve as meaningful measures? Let your outcomes be your guide…

It all begins with intentionality… and hinges upon well-articulated outcomes.

Self-Diagnostic  To what extent does your department/area have a mission statement that accurately reflects who you are and what you do?  To what extent do your outcomes accurately reflect that which you expect students to do or that which you do to support the learning environment?  To what extent do your outcomes help you identify what kind of information to collect or measures/tools to use?  To what extent do you know how well your students learned what you intended them to learn or how well your efforts support the learning environment?  To what extent does your assessment data provide feedback and evidence to inform change and guide your continual improvement efforts? Take 20 minutes to discuss with your colleagues the degree of your satisfaction in each of the areas of intentionality.

Example Let your outcomes be your guide…  What kind of information needs to be collected?  What kinds of measures/tools might be used?  Will said measure(s) provide information as to the extent to which students met the outcome?  Will the assessment data provide feedback and evidence to inform change and guide continual improvement efforts?

Let your outcomes be your guide…

Your Turn  Articulate an intended outcome.  Identify the various ways in which you might gather evidence for that outcome.  Determine whether or not said measure(s) will provide information as to the extent to which the outcome was met.  Evaluate the extent to which the measure(s) will provide feedback and evidence to inform change and guide continual improvement efforts

Responding Intentionally MEANINGFUL CHANGE AND CONTINUAL IMPROVEMENT

Responding to the Results If an outcome meets expectations at the standard:  Take no action  Identify this area as a program strength  Consider increasing expectations outlined in the outcome  Raise the standard of attainment  Consider surveying students about their experience of the program or service area Assessment may find that student learning or a program meets expectations at the determined standard for some outcomes.

Responding to the Results If results for an outcome do not meet expectations:  ASK QUESTIONS!!  Are our outcomes aligned with our expectations?  What aspect of our program or activities might be in need of revision?  Are our teaching and learning methods in need of revision?  With whom might we need to work to strengthen our outcomes?  What crazy idea might we come up with to tackle the problem? Assessment may find that student learning or a program does not meet expectations at the determined standard for some outcomes.

Not all data is created equally… Performance Percentages? Percentages of students meeting outcomes or not reveals overall performance but does not highlight HOW students do well or go wrong. Descriptions of Performance? Descriptions of patterns of strength and patterns of weakness can be the most revealing information, but percentages can help to define the extent of a particular problem. Surveys? Surveys often provide a snapshot or overview of satisfaction or awareness of services, but they rarely provide authentic or direct evidence as to whether learning outcomes have been met or highlight HOW students do well or go wrong.

Let your outcomes be your guide…and your measures provide actionable data to inform decisions and plans for intentional improvement.

Finding Some Takeaways 1.What (if anything) do these exercises reveal about your own program’s assessment practices and needs? 2.Identify one thing from this workshop that you think you might be able to take back to your program to make assessment more meaningful.

Responding to the Results Students benefit from an institution’s thoughtful response to an honestly undertaken attempt to determine a program’s strengths and weaknesses. Articulate Intended Outcomes Work Toward Intended Outcomes Gather Evidence Determine Extent to Which Outcomes Were Met Plan for Intentional Improvements