Measuring the Assessment Literacy of Educators ---- Using the MAC’s Assessment Literacy Self-Assessments to Build Assessment Literacy Edward Roeber – Michigan Assessment Consortium Denise Brady – Shiawassee RESD Paul Stemmer – Michigan Department of Education

What is the MAC? The Michigan Assessment Consortium (MAC) is a non-profit professional organization of educators who believe quality education depends on accurate, balanced, meaningful assessment. MAC members work to advance assessment literacy and advocate for greater understanding about and use of assessment in Michigan. The goal of the MAC is to help educators use assessment to improve their teaching and students’ learning. A variety of MAC resources can be accessed at www.michiganassessmentconsortium.org

MAC Members Are individuals and organizations interested in quality assessment practices Work in public and private schools, local and regional districts, government agencies, professional associations, and institutes of higher learning Embrace high standards of excellence Understand that quality assessment is key to quality learning and student achievement Value professional learning and continuous improvement

Assessment Literacy Defined An assessment literate individual is one who understands how student assessment can enable them to better carry out their role in education, believes that assessment can improve teaching and learning, and puts into place activities and behaviors to act on these beliefs

Who Needs to be Assessment Literate? Everyone with a stake in education needs to be assessment literate, including: Students Parents/guardians Teachers Specialists at the district and building levels Building administrators Central office administrators Policymakers at the local, state, and national levels Public

Why is Assessment Literacy Needed? Assessment literacy is needed for several reasons Lack of understanding about assessment by those who adopt policy and laws, and govern our schools Lack of understanding by those who teach our students or lead our schools, due to continued lack of pre-service preparation for educators Misunderstanding by parents and students about how student assessment can promote high quality student learning Increased volume and changes in types of student assessments Increased stakes for students, educators, and school Why is Assessment Literacy Needed?

Why is Assessment Literacy Needed? Research has shown that students who are more involved in their own learning – and assessment – achieve more Effective use of formative assessment practices requires teachers to understand how on-going instructionally-embedded assessment can help all student achieve at higher levels Administrator involvement in school improvement activity is also related to higher student achievement Why is Assessment Literacy Needed?

Standards Development Process The work was begun by the MAC Board A review of the literature was carried out Other sets of standards were examined An initial draft of the ALS was created Another draft of the ALS was produced and reviewed externally by Susan Brookhart, Carol Commodore, Margaret Heritage, Ken O’Connor, Jim Popham, and Rick Stiggins An in-state review packet/survey was used to gather input from MI educators Standards Development Process

Standards Development Separate standards were developed for: Students and Their Parents Teachers Building administrators District administrators Local and state policymakers The MAC Assessment Literacy standards include: Dispositions (i.e., beliefs) Knowledge Performance (i.e., skills) Goal—provide a common basis for work in increasing assessment literacy Standards Development

Sample Disposition Standards Building-Level Administrators should believe that: An effective assessment system must balance different purposes for different users and use appropriate assessment methods to measure different learning targets. Multiple measures can provide a more balanced picture of a student or a school. Quality assessments are a critical attribute of effective teaching and learning. http://michiganassessmentconsortium.org/sites/default/files/mac_AssessLitStds_mobile.pdf Sample Disposition Standards

Sample Knowledge Standards Building-Level Administrators should know: A balanced assessment system consists of both of the following: Different users have different assessment purposes. Different assessment purposes may require different assessment methods. The different types of assessment methods and when teachers should use each Selected response Constructed response Performance Personal communication Sample Knowledge Standards

Sample Performance Standards Building-Level Administrators should promote a culture of appropriate assessment practice by: Promoting assessment literacy for self and staff Building-Administrators should promote the use of assessment data to improve student learning through the alignment of curriculum, instruction and assessment by: Using assessment results, including subgroup performance, to influence the school’s curriculum and instructional program. Using multiple sources of data over time to identify trends in learning. Sample Performance Standards

Sample Self-Assessment Question - Disposition Which of the following is the most important characteristic of balanced assessment system? (I-B)   A. Each type of assessment is given the same number of times during the school year. B. School accountability assessments are given the highest priority in the assessment system. C. Classroom assessments are given the highest priority in the assessment system. D. The information needs of all users are considered equally important and are met. Sample Self-Assessment Question - Disposition May not need this slide if sample assessment is given

Sample Self-Assessment Question - Knowledge What is the definition of “formative assessment strategies?” (II-C3)   A. Short tests given multiple times during the school year. B. Sets of test items available in an item bank. C. Activities used during instruction to determine whether students learned what they were just taught. D. An assessment that is given at the end of each marking period for grading students. Sample Self-Assessment Question - Knowledge

Sample Self-Assessment Question - Performance What is the most effective way for a building administrator to interpret achievement results and create improvement goals with staff? (III-F)   A. Demand teachers improve instructional practices B. Lead dialogues with staff in each of the content areas assessed C. Tell teachers what they need to improve D. Publically announce the high and low performing classrooms in the school Sample Self-Assessment Question - Performance

www.socrative.com Student Log-in Room Name: MIALS Take a Sample Assessment of Your Assessment Literacy www.socrative.com Student Log-in Room Name: MIALS Join the Room (You may put any name you choose!)

Three Important Questions The MAC thinks about three questions: If educators achieve these standards, what would be different in our students? If policymakers achieve these standards, what would be different about assessment policies and legislation? If this is a preferred future, what needs to be done to push this vision forward? This set of questions drives the work of the MAC Keep for after assessment

Assessment Development Separate Assessment Literacy Self Assessments have been or will be created for: Students/parents Teachers Building administrators District administrators Local policymakers State policymakers Talk about which have been developed at this point

Assessment Literacy Initiative: A New Frontier for NAEP Cornelia Orr, National Assessment Governing Board (NAGB) Holly Spurlock, National Center for Education Statistics (NCES) Assessment Literacy Workshop March 18, 2015

The Charge December 2012: David Driscoll challenged the Board to use NAEP to make a difference May 2013: Board member Jim Popham proposed increasing understanding of assessment literacy August 2013: The Assessment Literacy Work Group was created to develop a communications plan to educate parents, policymakers, and students about assessments Initial work group meetings determined that while this initiative would include NAEP, the campaign would be intentionally broader

NAGB Summit for Parent Leaders This pre-planned event was a great opportunity to get feedback from one of the initiative’s target audiences The Parent Summit collected feedback from parents on educational assessments Key takeaways: Parents want to know how to interpret their child’s test results There is strong mistrust towards testing Parents want to know how testing relates to learning Parents are confused about different tests and purposes

NCSA 2014 Governing Board session at the National Conference on Student Assessment (NCSA) Participants (including NAEP State Coordinators) provided feedback on the “Understandings” Key takeaways: Communication about achievement levels is important Talk about the assessment quality and precision Communicate the differences between a group and an individual assessment Word understandings more positively

Target Audience Primary audiences for the assessment literacy initiative are: Parents Policymakers Students Work group focuses on parents first Important to educate parents on the value of student assessment Governing Board’s existing plans to reach parents Future focus on policymakers and students

Reaching Parents: Intermediaries and Influencers

Parent “Understandings” “Understandings” are a set of concepts for the target audiences to acquire “Understandings” will serve as content framework for the initiative Five “Understandings” Evidence for Interpretations and Uses Accuracy of Test Scores Fairness Variations in Test Quality Assessment to Improve Teaching and Learning

Focus Groups: Objectives Gather feedback focused on clarity, relevance, appeal, and usefulness Explore knowledge and attitudes about assessments Explore parent preferences in communication channels Use parent feedback to revise content for communication materials

Focus Groups: Topics Topic Areas The Purpose of Educational Assessments Types of Information from Educational Assessments Using Assessment to Improve Teaching and Learning Test Scores Are Not Perfect Estimates of Student Performance Appropriate Use of Assessments Quality Variation in Assessments

Focus Groups: Lessons Learned “Understandings” Findings Evidence for interpretations and uses Parents want more information about the specific purpose of each assessment and how the results will be used Accuracy of test scores Parents support having multiple measures to assess student performance Fairness & Accuracy Parents are apprehensive about the fairness and accuracy of “high stakes” assessments Variations in test quality Parents rely on educators to determine if a test is high quality Assessments to improve teaching and learning Parents are not familiar with the term, but they value the concept of formative assessments

Strategies for Communicating with Parents Core messaging principles to guide final message development and prioritization: Assessment information must use plain, accessible language Assessment information for parents must center on benefits for their children Leading assessment messages must align with parent priorities

Prototypes: Objectives Assessment literacy materials will: Establish the Governing Board’s and NCES’ thought leadership on assessment literacy Engage parents and their influencers Serve a specific purpose to meet the various needs and interests of the diverse audience Incorporate “plain language” messaging Reach parents directly and indirectly through a variety of channels

Assessment Literacy 101 Video Prototypes: Examples Parent Toolkit Assessment Literacy 101 Video Microsite

Prototypes: Parent Toolkit A toolkit of materials will allow parents, educators, and collaborators to use materials flexibly Print materials can be adapted to accommodate a wide range of situations: Distribute at school meetings Featured at conferences Could include fact sheet introducing the campaign and one-pagers that expand on the five “Understandings”

Prototypes: Video Parent focus groups are interested in receiving assessment literacy information through videos In response, the work group has: Proposed a video—“Assessment Literacy 101”—with parent and teacher testimonials Discussed a video that could spotlight each “Understanding” to provide deeper content Storyboard in development on the topic of formative assessment

Prototypes: Microsite Website will promote specific actions: Learn about the “Understandings” Download materials such as fact sheets or informational videos; review a sample test Explore testing of all kinds in their community with an interactive map

Prototypes: Next Steps Additional focus groups with parents are planned. Their feedback will be used to guide the refinement of the communication materials being produced Continue to work collaboratively with the work group. This includes providing feedback from the focus groups Determine the calls to action for parents. Clearly define our expectations of parents and the actions they should take  

MAC Resources Supporting Assessment Literacy To access the resources of the MAC, visit the following webpage: www.michiganassessmentconsortium.org Resources include Common Assessment Development modules, white papers on assessment and other topics, access to a variety of archived videoconferences, and other material

Michigan Learns Assessment! First Steps on the Assessment Literacy Pathway Creation of a comprehensive assessment-learning program designed for individuals who want or need to know more about how student assessments are developed and used. The goal in creating the modules included in this introductory assessment-learning program is provide foundational knowledge about assessment dispositions, knowledge, and performances to teachers, administrators, and others, with the hope that the standards and the learning program will encourage individuals to seek to learn even more about assessment and therefore will use assessment more effectively

Michigan Learns Assessment! Vision: Development of a more comprehensive assessment-learning system A more advanced program of assessment learning might lead to: An assessment specialty endorsement for administrators MAC certification for teachers Assessment specialists for each district and school in the state Make Michigan the State of Assessment Literacy!

Michigan Learns Assessment! Modules: Currently under development Structured so that learning may occur: Short or longer sessions - one or more days Together or over time Each module will contain: A presentation including built-in individual and group learning activities A discussion guide for use at the conclusion of each module to guide group discussion and deeper assessment understanding

Michigan Learns Assessment! Timeline: Complete the drafts of the modules by January 2016 Conduct in-state reviews by April 2016 Revise and focus the presentations by June 2016 Field test the modules in schools/districts summer/fall 2016 Revise the modules by December 2016 Produce attractive final versions of the modules by May 2017 Use the modules to offer introductory assessment literacy efforts across the state by June 2017

Michigan Learns Assessment! Proposed Eight MAC Introductory Assessment-Learning Modules: 1–Introductory Module – What is Assessment Literacy and Why is it Important? Goal: To provide an overview of what assessment literacy means, why it is important, and how one can become assessment literate. 2–Standards and Assessment Goal: Understand how different definitions of learning, types of standards, and instructional designs affect student assessment choices and designs.

Michigan Learns Assessment! 3–Developing a High Quality, Balanced Assessment Systems Goal: Recognize the components of a balanced assessment system and the different types of measures used in it. 4–Criteria for High Quality Assessment Goal: Understand (in non-technical terms) key metrics related to assessment quality 5–Developing or Selecting Appropriate Assessments Goal: Know how to use the criteria for high quality assessment to select or develop needed student assessments

Michigan Learns Assessment! 6–The Formative Assessment Process Goal: Describe the elements of the formative assessment process as well as how teachers would use them. 7–Making Meaning from Student Assessments Goal: Know how to translate student assessment data usefully for teachers, students and their parents, and administrators 8–Using and Communicating Assessment Results to Promote Student Learning Goal: Know how to effectively use and report assessment information to a variety of audiences in order to promote student learning

Developed a proposal for planning a larger grant proposal to create assessment literacy development strategies and resources This initial grant will network MI professional organizations to think about what their members need to become more assessment literate—short-, medium-, and long-term The work is envisioned as occurring over a number of years, on an on-going basis Current Work of the MAC

The work will begin with a review of available resources Assessment Literacy Self Assessments will be used to: Gather data for proposal writing Individual awareness about assessment literacy Promote organizational work on assessment literacy Goal is to provide resources to address the current state of assessment illiteracy The MAC is committed to seeing this through – to continue as it can with available resources to push for greater assessment literacy Current Work of the MAC

Use of the Assessment Literacy Standards The goal in creating the Assessment Literacy Standards was to define what those affected by student assessment (of all types and for all purposes) should believe, know, and do regarding assessment After defining these, it is the goal of the MAC (and an essential part of its mission) to help various users become more assessment literate Becoming assessment literate is important, for a variety of reasons The MAC’s work is important for both pre-service and in-service educators Use of the Assessment Literacy Standards

Benefits of MAC Membership Access to… A rich library of assessment resources Invitations to assessment professional learning opportunities Video conferences and digitized content designed to enhance assessment literacy An emerging catalogue of model, curriculum-embedded performance assessments

Edward Roeber Assessment Director Michigan Assessment Consortium (517) 614-4877 roeber@msu.edu www.michiganassessmentconsortium.org For More Information