Introduction to Assessment Jennifer Lowman Coordinator, Student Persistence Research Credit for the content of this presentation should go to Marilee Bresciani and her many co-authors; & Gavin Henning & others with Academic Impressions

Outline Define Assessment Identify different Types of Assessment Review Cycle of Assessment Define Outcomes – Writing “Learning Outcomes” Practical Assessment Techniques Ethics Concluding Remarks

What is Assessment? Assessment, Evaluation, & Research Assessment – An effort to gather, analyze, and interpret evidence to understand effectiveness and improve programming Evaluation – Using assessment information to make an informed judgment of program quality or worth Research – Collection and analysis of data to build or confirm theory or conceptual foundations

Institutional Assessment Mission Goals or Objectives Program, Product, Service  Inputs Outcomes Improvement or Accountability Assessment Planning part of Programming – Embed Assessment Design, Measurement, & Analysis Reporting & Learning

Embedded Assessment Make assessment part of your program, event, service, or activity Decades of research shows that students respond to intentional activities that are linked to positive outcomes (Bresciani, Zelna, & Anderson, 2004). – If you tell students what you expect them to learn and how it will benefit them, they are more likely to learn

Embedded Assessment is… Continual assessment for continual improvement Track changes in program impact over time, as well as… – Improve measurement over time – Test new measures, while still using old Make it a habit – a state of mind!

Culture of Assessment Build shared trust: Begin by lowering social and interpersonal barriers to change. Build shared motivation: Collectively identify goals worth working toward and problems worth solving. Build a shared language: Develop a collective understanding of new concepts needed for transformation. Build shared guidelines: Develop a short list of research-based guidelines for using assessment to promote learning

Types of Assessment

Tracking Usage Needs Assessment Satisfaction Study Culture/Climate Assessment Outcomes Assessment Resource Effectiveness Study Benchmarking Program Review Strategic Planning

Forms of Assessment Operational or Descriptive Assessment Tracking Usage Needs Assessment Satisfaction Study Culture/Climate Assessment Formative Assessment Outcomes Assessment Resource Effectiveness Study Benchmarking Political Assessment Program Review Strategic Planning

Types of Assessment? (activity)

Cycle of Assessment

1.Define issue/problem 2.Identify Goals 3.Align mission & goals 4.Identify stakeholders 5.Identify theoretical/conceptual framework 6.Develop measurable outcomes from goals 7.Identify and measure inputs 8.Classify strategies from framework to reach goals 9.Isolate action steps to implement strategies Cycle of Assessment: External Perspective 10. Develop formative assessment plan to understand process 11. Measure Outcomes 12. Make sense of results 13. Report / Present findings 14. Review, Reallocate, Advocate for Resources 15. Retool Program 16. Plan for Iteration #2

Cycle of Assessment: Internal Perspective Mission Purpose Identify Outcomes Create Opportunity Gather, Interpret Data Improve Program

Outcomes-Based Assessment

Assessment & Planning Assessment and Planning are linked, intertwined, and interrelated “Backwards Design” or “Start with the end” Programs are designed this way too! Desired Outcome Program Inputs

From Goals to Outcomes Goals/Objectives – end result in broad terms (optimal outcomes) Outcomes – more specific statements derived from goals that are measurable – Operational Outcomes – Learning Outcomes – Program Outcomes

Learning v. Program Outcomes Learning Outcomes (aka Individual Outcomes) – a learning outcome is participant-centered and describes the desired effect of a program on a student’s cognitive or intellectual state. Statements of what students will value, do, or know as a result of participation. – ASK: attitude, skills, & knowledge Program Outcomes (aka Aggregate Outcomes) – the desired aggregate effect of a program.

Program or Learning Outcome? (worksheet)

Learning Outcomes Participant - centered – Focus on what students can do – Not on what you did! Purpose to reveal the impact of ‘Action Steps’ – Action Steps are the point of contact – May or may not be perceived by students – Outcomes may need to take variation in contact into account – Variation is key for identifying effectiveness!

Learning Outcomes Assessment What are students learning and why? How do we know? Are students learning what we intend for them to learn? How do we use assessment information to improve learning? Do the improvements we make work?

SMART Outcomes SMART  Specific  Measurable  Aggressive/Attainable  Results-oriented  Time-bound

Writing Learning Outcomes ABCD formula – Audience (Who) Who is the target? – Behavior (What) What should the target be able to know, do, value (ASK)? – Condition (How) What Action Steps will facilitate the learning? – Degree (How much) How much learning will be demonstrated?

Writing Learning Outcomes Condition + Audience + Behavior + Degree As a result of participating in the event + students + will demonstrate + 3 of 5 skills.

ABC v. ABCD… or why Degree is important! The degree of impact you expect to have on your students is important to… – Set criterion for success AHEAD of program implementation – Demonstrate Minimal Benefit Transparency & Accountability – Signal intention to students and stakeholders – Establish standard for data-driven decision-making

Degree of Impact Independent Indicators (Index) – Inter-changeable, equally important information – Differences between students at start/finish are not import. Ex. 3 of 5 study habits Ex. 5 of 7 coping strategies Continuous Scale – Cumulative or increasing attitude, skill, or knowledge – Expect differences between students at start/finish Increased leadership skills from baseline eval to next level (basic to intermed.; intermed. to adv.) Process – Student must master step 1 to move onto step 2…etc. – Describe steps in process within LO – Break process in to multiple LOs, but avoid yes/no outcomes

Degree of Impact ≠ Depth of Learning

Degree of Impact & Depth of Learning Influenced by the opportunity provided – Exposure – Relevance – Immediacy – Salience – Feedback – … L.O. reflects depth of learning in the “Action Verb”

Depth of Learning Bloom’s Taxonomy (revised) Remembering: recognizing, recalling, labeling Understanding: interpreting, summarizing, contrasting Applying: executing, implementing, classifying Analyzing: differentiating, organizing, attributing Evaluating: critiquing, integrating, generalizing Creating: generating, planning, producing (teaching)

List five ways identified in the readings that can help you reduce stress? (Remembering) Based on the floor meeting, discuss at least three reasons why we don’t allow alcohol in the residence halls? (Understanding) How can you use what you learned in these activities in your student organization? (Application) After participating in the retreat, compare the effectiveness of leadership tactics? (Analyzing) As you reflect on this past year as I have served as your organization’s advisor, in what areas have I been most effective and in what areas can I improve (Evaluating) Based on our conversation regarding career action steps, recommend four action steps to each of your group members? (Creating) The Depth of Learning expected is evident in the Action Verb!

Action Verbs (Observable Behaviors) Lower-Level List, Label, Name Define, Describe Identify, Interpret Contrast, Discuss Classify, Complete Relate, Modify Demonstrate Higher-Level Solve, Apply Analyze, Explain Infer, Integrate Plan, Create Design, Generalize Assess, Rank Recommend Compare, Teach Don’t Use – Appreciate – Become aware – Be familiar with – Know – Learn – Understand

Practice Writing Learning Outcomes (Activity)