1. Julia Williams, Ph.D University of Minnesota Duluth
The Remarkable Half-naked Rubric: Creating Collaboratively Developed Tools to Measure Results of Education as Intervention AEA 2013
Take this word by word. What would have attracted you to this workshop?
Give some examples of collaboratively created tools used in participatory evaluation
Julia Williams, Ph.D
University of Minnesota Duluth

2. Education as intervention?
Training to perform or better perform tasks
Awareness
Skill building
Knowledge creation
Understanding
Growth or increase in proficiency
Increase affinity or proclivity
Can you think of a time that you used education as intervention?
Education as intervention?

3. If education is intervention…..
How is it measured? How do you know if the training, teaching, was effective? How do you know what the learner learned? How do you know what the learner is expected to learn?
If education is intervention…..

4. What is it that is hoped happens?
Knowledge? Skill? Affinity? Awareness? Growth in proficiency? Understanding?
What is it that is hoped happens?

5. Types of Assessment Selected Response Constructed Response
Personal communication/ observation
Performance
Portfolio/longitudinal
Affective
Types of Assessment

6. So.. What’s the match? Types of assessment Types of targets
Selected response Constructed response Personal communication Observation Performance Portfolio Affective
Knowledge Understanding Skills Growth Dispositions
So.. What’s the match?

7. If you measure understanding
Wiggins and McTighe’s Facets
Explanation
Interpretation
Application
Perspective
Empathy
Self-Knowledge
If you measure understanding

8. Compare and Contrast or the rest is futile
Directions
Checklist
Matrix
Rating Scale
Rubric

9. What is it?
Rubric

10. Rubrics and Scoring Criteria can be…
Holistic
Trait-analytic
Weighted
Critical Domain

11. Rubric Geography
Rubric
Scoring
Criteria
Empirical
Descriptors
Domains

12. Rubric Language…. Anchors Empirical Descriptors
Comparative Descriptors
Parallel construction
Exemplars
Levels
Rubric Language….

13. Dr. Wiggins says….. Good Rubrics….
Discriminate among performances validly
Do not combine criteria in domains
Provide useful description
Are valid
Are reliable
Rely on descriptive language
Dr. Wiggins says…..

14. Dr. Wiggins also says…. Good rubrics
Do not mandate
Process
Format
Methods or approach
Good rubrics do not rely on orthodoxy or taste, and they are not built based on mediocrity, and they do not reward “good faith/effort.”
Dr. Wiggins also says….

15. Good Rubrics Clear to user Distinct differences between criteria
Fair and Accurate
Do not define what should not be done
Never use negative terms…not quite, not always, only part of the time…
Good Rubrics

16. Bad Rubrics Count Use irrelevant criteria Dictate format
Use good, better, best…
Aim to enable scoring regardless of levels of quality
Bad Rubrics

17. Building Rubrics Decide what is important
Decide what standard-level looks like
Decide what acceptable looks like

18. What would break your heart if they moved on not knowing
What would break your heart if they moved on not knowing? What are the enduring understandings?
What are additional items that are important?
What is nice to know?
Rubric Groundwork

19. Blanket Making
Essential Goals:
The idea of continually working hard at something to create a larger goal. Little steps at a time.
The idea that the learners can create something, start something and finish something designed from their own ideas.
The idea of giving something important from themselves. An act of selflessness.
Important Goals:
The learners will learn the basic skills to be able to sew and create a blanket.
The learners will become familiar with their sewing machine so they can do simple tasks such as threading the machine and bobbin.
The learners will effectively know how to use mathematical reasoning to create the pattern and design of the blanket.
Nice To Know Goals:
The Learners would be able to alter their patterns or learn different styles of blankets.
The learners would be able to use multiple sewing techniques such as appliqué, quilting or zig zig stitch.
The learners would be able to fix basic problems with their sewing machines.
The learners would be able to assist their peers with the creation of their blankets.

20. Blanket Making Rubric
Patience
Learner completes tasks despite frustration and difficulty.
The learner completes tasks and asks for help when overwhelmed with frustration or confusion. The learner takes breaks before he/she is frustrated.
Problem Solving
The learner records major problems in his/her journal and fixes major mistakes before completion.
The learner records most of their problems in their journal and goes back and fixes his/her mistakes before completion.
Effective Use of Materials and Tools
Utilizes cloth according to plan. Demonstrates mastery of the basic sewing skills. (See list)
Wastes little to no cloth (less than 3”x3”). Demonstrates mastery of all of the basic sewing skills.

21. Another example

22. Use the items you have decided to measure to make your rubric
Determine what it looks like when your students “nail it.”
Determine what it looks like when your learners achieve minimal acceptable levels.
Let’s try…….

23. Farmers hoping to receive loans must first complete a course in budgeting, long-range planning and basic accounting to reduce defaults
Training consists of three weekends face to face and on-line in-between
Youth in a large school district will all receive lessons in social emotional learning to reduce bullying
Training is developed and delivered by homeroom teachers in advisory during the first two months of year
Tribal youth (14-16, from 3 tribes) attend summer camp focused on The Circle of Courage to become leaders in their own communities
Circle of Courage teachings are ½ of the curriculum of a week long camp over three years.
Tenure-track assistant professors are required to attend training in effective instruction and assessment to increase student achievement at a large university
Training consists of once-a-month ½ day direct group instruction and faculty observation until tenured
Let’s Try These?

24. Fractal Missions, Visions, Beliefs, Standards and Goals
Targets Determined by Stakeholder Consensus
Benchmarks Placed Appropriately
Curriculum and Measurement Aligned
Appropriate Evidence Gathered and Scored 24

25. Mission Statement
Some School is a public charter school serving grades nine through twelve. We provide a small, learner-directed community that encourages investigative learning, global citizenship and nurtures a sense of belonging. Our purpose is to graduate students who are knowledgeable, discerning, passionate, creative, and reflective.

26. Reflective Graduates will
Target Rubric
Level 1
Level 3
Accurately engage in self-evaluation
Describes performance based on evidence
Systematically and insightfully reviews performance to set clear goals
Seek and effectively utilize critique from others
Incorporates change based on feedback
Establishes and utilizes systems for critical feedback to incorporate into performance
Provide accurate and effective feedback to others
Reviews work based on established criteria
Consistently, effectively and sensitively analyzes the work of others

27. What does the mission/belief/standards look like?
Building a rubric:
What does it look like at standard? (Level 3)
What does it look like at acceptable? (Level 1) 1 2 3 4

28. The Rest is up to you……
The Rest is up to you……

29. References
Musial, D., Nieminen, G., Thomas, J & Burke, K. (2009). Foundations of meaningful assessment. Boston: McGraw- Hill.
Popham, W.J.(2011). Classroom assessment: What teachers need to know 6th ed. Boston: Pearson.
Stiggins, R.J., & Chappuis, J. (2012). An introduction to student-involved assessment for learning 6th ed. Boston: Pearson.
Wiggens, G. & McTighe. (2005). Understanding by design. Upper Saddle River, NJ: Merrill Prentice Hall