Julia Williams, Ph.D University of Minnesota Duluth The Remarkable Half-naked Rubric: Creating Collaboratively Developed Tools to Measure Results of Education as Intervention AEA 2013 Take this word by word. What would have attracted you to this workshop? Give some examples of collaboratively created tools used in participatory evaluation Julia Williams, Ph.D University of Minnesota Duluth

Education as intervention? Training to perform or better perform tasks Awareness Skill building Knowledge creation Understanding Growth or increase in proficiency Increase affinity or proclivity Can you think of a time that you used education as intervention? Education as intervention?

If education is intervention….. How is it measured? How do you know if the training, teaching, was effective? How do you know what the learner learned? How do you know what the learner is expected to learn? If education is intervention…..

What is it that is hoped happens? Knowledge? Skill? Affinity? Awareness? Growth in proficiency? Understanding? What is it that is hoped happens?

Types of Assessment Selected Response Constructed Response Personal communication/ observation Performance Portfolio/longitudinal Affective Types of Assessment

So.. What’s the match? Types of assessment Types of targets Selected response Constructed response Personal communication Observation Performance Portfolio Affective Knowledge Understanding Skills Growth Dispositions So.. What’s the match?

If you measure understanding Wiggins and McTighe’s Facets Explanation Interpretation Application Perspective Empathy Self-Knowledge If you measure understanding

Compare and Contrast or the rest is futile Directions Checklist Matrix Rating Scale Rubric

What is it? Rubric

Rubrics and Scoring Criteria can be… Holistic Trait-analytic Weighted Critical Domain

Rubric Geography Rubric Scoring Criteria Empirical Descriptors Domains

Rubric Language…. Anchors Empirical Descriptors Comparative Descriptors Parallel construction Exemplars Levels Rubric Language….

Dr. Wiggins says….. Good Rubrics…. Discriminate among performances validly Do not combine criteria in domains Provide useful description Are valid Are reliable Rely on descriptive language Dr. Wiggins says…..

Dr. Wiggins also says…. Good rubrics Do not mandate Process Format Methods or approach Good rubrics do not rely on orthodoxy or taste, and they are not built based on mediocrity, and they do not reward “good faith/effort.” Dr. Wiggins also says….

Good Rubrics Clear to user Distinct differences between criteria Fair and Accurate Do not define what should not be done Never use negative terms…not quite, not always, only part of the time… Good Rubrics

Bad Rubrics Count Use irrelevant criteria Dictate format Use good, better, best… Aim to enable scoring regardless of levels of quality Bad Rubrics

Building Rubrics Decide what is important Decide what standard-level looks like Decide what acceptable looks like

What would break your heart if they moved on not knowing What would break your heart if they moved on not knowing? What are the enduring understandings? What are additional items that are important? What is nice to know? Rubric Groundwork

Blanket Making Essential Goals: The idea of continually working hard at something to create a larger goal. Little steps at a time. The idea that the learners can create something, start something and finish something designed from their own ideas. The idea of giving something important from themselves. An act of selflessness. Important Goals: The learners will learn the basic skills to be able to sew and create a blanket. The learners will become familiar with their sewing machine so they can do simple tasks such as threading the machine and bobbin. The learners will effectively know how to use mathematical reasoning to create the pattern and design of the blanket. Nice To Know Goals: The Learners would be able to alter their patterns or learn different styles of blankets. The learners would be able to use multiple sewing techniques such as appliqué, quilting or zig zig stitch. The learners would be able to fix basic problems with their sewing machines. The learners would be able to assist their peers with the creation of their blankets.

Blanket Making Rubric Patience Learner completes tasks despite frustration and difficulty. The learner completes tasks and asks for help when overwhelmed with frustration or confusion. The learner takes breaks before he/she is frustrated. Problem Solving The learner records major problems in his/her journal and fixes major mistakes before completion. The learner records most of their problems in their journal and goes back and fixes his/her mistakes before completion. Effective Use of Materials and Tools Utilizes cloth according to plan. Demonstrates mastery of the basic sewing skills. (See list) Wastes little to no cloth (less than 3”x3”). Demonstrates mastery of all of the basic sewing skills.

Another example

Use the items you have decided to measure to make your rubric Determine what it looks like when your students “nail it.” Determine what it looks like when your learners achieve minimal acceptable levels. Let’s try…….

Farmers hoping to receive loans must first complete a course in budgeting, long-range planning and basic accounting to reduce defaults Training consists of three weekends face to face and on-line in-between Youth in a large school district will all receive lessons in social emotional learning to reduce bullying Training is developed and delivered by homeroom teachers in advisory during the first two months of year Tribal youth (14-16, from 3 tribes) attend summer camp focused on The Circle of Courage to become leaders in their own communities Circle of Courage teachings are ½ of the curriculum of a week long camp over three years. Tenure-track assistant professors are required to attend training in effective instruction and assessment to increase student achievement at a large university Training consists of once-a-month ½ day direct group instruction and faculty observation until tenured Let’s Try These?

Fractal Missions, Visions, Beliefs, Standards and Goals Targets Determined by Stakeholder Consensus Benchmarks Placed Appropriately Curriculum and Measurement Aligned Appropriate Evidence Gathered and Scored 24

Mission Statement Some School is a public charter school serving grades nine through twelve. We provide a small, learner-directed community that encourages investigative learning, global citizenship and nurtures a sense of belonging. Our purpose is to graduate students who are knowledgeable, discerning, passionate, creative, and reflective.

Reflective Graduates will Target Rubric Level 1 Level 3 Accurately engage in self-evaluation Describes performance based on evidence Systematically and insightfully reviews performance to set clear goals Seek and effectively utilize critique from others Incorporates change based on feedback Establishes and utilizes systems for critical feedback to incorporate into performance Provide accurate and effective feedback to others Reviews work based on established criteria Consistently, effectively and sensitively analyzes the work of others

What does the mission/belief/standards look like? Building a rubric: What does it look like at standard? (Level 3) What does it look like at acceptable? (Level 1) 1 2 3 4

The Rest is up to you…… The Rest is up to you……

References Musial, D., Nieminen, G., Thomas, J & Burke, K. (2009). Foundations of meaningful assessment. Boston: McGraw- Hill. Popham, W.J.(2011). Classroom assessment: What teachers need to know 6th ed. Boston: Pearson. Stiggins, R.J., & Chappuis, J. (2012). An introduction to student-involved assessment for learning 6th ed. Boston: Pearson. Wiggens, G. & McTighe. (2005). Understanding by design. Upper Saddle River, NJ: Merrill Prentice Hall