C R E S S T / U C L A Usable Assessment Knowledge: A Design Problem Eva L. Baker International Congress for School Effectiveness and Improvement January 6, 2003 Sydney, Australia UCLA Graduate School of Education & Information Studies Center for the Study of Evaluation National Center for Research on Evaluation, Standards, and Student Testing With support from the Education Institute of Science, U.S. Department of Education

C R E S S T / U C L A Knowledge to Support Educational Improvement UnusablePlentiful UsableScarce UsefulRare

C R E S S T / U C L A Usable and Useful Knowledge Usable Knowledge In a form that can be understood In a form that can be applied Timed appropriately May cause rethinking of the problem Useful Knowledge Rethinking indicates a new solution path Adapted to situation Sufficient to guide solution Improved outcomes occur as a result

C R E S S T / U C L A Why Are Some Schools Successful in Using Knowledge?  Focus on learning (students and adults)  Constant use of appropriate information (formal and informal)  Focus on feedback and change  Public display and exchange  Pride in outcomes of students and place  Select knowledge to foster these ends

C R E S S T / U C L A Assessment Knowledge in the Service of Reform Usable? Useful?

C R E S S T / U C L A Assessment: Historical Transformations 1. Assessment = measurement and interpretation of a sample of performance in the desired area of learning 2. Assessment (test) score = learning Activities not contributing should be dropped Coincidence = causality 3. Assessment is the best intervention Cost/ Effectiveness (Useful) (Usable) (Unusable)

C R E S S T / U C L A Types of Assessment Knowledge  Purposes for assessment (Why? For whom?)  What to assess  Whom to assess  How to assess (design and procedures)  How to interpret and report results  How to determine if results are trustworthy (validity) for the purpose(s)

C R E S S T / U C L A Key Design Principles for Useful Assessment  Assessment systems that start with thinking skills and apply them to content domains support Coherent, sustained learning Spiral teaching Transfer (application to new or unforeseen situations)

C R E S S T / U C L A CRESST Assessment Models  Research-based  Focus on cognition and learning  Reusable and cost-sensitive  Operationalized in models and templates

C R E S S T / U C L A Model-Based Assessment Families of Cognitive Demands Content Understanding Problem Solving Teamwork and Collaboration MetacognitionCommunication Learning

C R E S S T / U C L A Usable Model-Based Assessment Design  Specifications and “Templates” for assessment protocols, including scoring  Templates that allow common design approaches to be used, e.g., primary sources  Two template examples for the model of deep understanding of content Explanation Graphical representation of relationships

C R E S S T / U C L A Specifications of Model for Content Understanding  Primary source materials in each domain  Student required to integrate prior knowledge and principles to succeed  Scored by using expert model in subject matter

C R E S S T / U C L A Content Understanding Template #1 Explanation  An array of primary source materials  A prompt that asks for an explanation in context  Constructed (written) answer  Evaluated by means of a scoring rubric

C R E S S T / U C L A Hawaiian History Writing Assignment: Bayonet Constitution Be sure to show the relationships among your ideas and facts. Your essay should be based on two major sources: 1. The general concepts and specific facts you know about Hawaiian history, and especially what you know about the period of the Bayonet Constitution. 2. What you have learned from the readings yesterday. Imagine you are in a class that has been studying Hawaiian history. One of your friends, who is a new student in the class, has missed all the classes. Recently, your class began studying the Bayonet Constitution. Your friend is very interested in this topic and asks you to explain everything that you have learned about it. Write an essay explaining the most important ideas you want your friend to understand. Include what you have already learned in class about Hawaiian history, and what you have learned from the texts you have just read. While you write, think about what Thurston and Liliuokalani said about the Bayonet Constitution, and what is shown in the other materials.

C R E S S T / U C L A EXCERPTS from HAWAIIAN HISTORY PRIMARY SOURCE DOCUMENTS LILIUOKALANI For many years our sovereigns had welcomed the advice of American residents who had established industries on the Islands. As they became wealthy, their greed and their love of power increased. Although settled among us, and drawing their wealth from resources, they were alien to us in their customs and ideas, and desired above all things to secure their own personal benefit. Kalakaua valued the commercial and industrial prosperity of his kingdom highly. He sought honestly to secure it for every class of people, alien or native. Kalakaua’s highest desire was to be a true sovereign, the chief servant of a happy, prosperous, and progressive people. And now, without any provocation on the part of the king, having matured their plans in secret, the men of foreign birth rose one day en masse, called a public meeting, and forced the king to sign a constitution of their own preparation, a document which deprived [him] of all power and practically took away the franchise from the Hawaiian race.

C R E S S T / U C L A History Explanation Scoring Rubric  General Impression of Content Quality  Principles or Concepts  Prior Knowledge  Use of Available Resources  Misconceptions (negative)  Argumentation (domain appropriate)  English Mechanics

C R E S S T / U C L A Mathematics Explanation Task (9-year-olds) Imagine a person from a television station has asked you to give a demonstration on TV. You will be on a show to help other students learn about math. You are asked to explain everything students your age should know about fractions. Below are some questions you should try to answer. These are questions that students in the TV audience will ask you. For each question you should draw as many pictures as you can to show what you mean. Then write down what you would say about your pictures on TV. Use as many words and pictures as you need. What is a fraction? Why are there two numbers in a fraction? How many fractions are there between 0 and 1? How many fractions are equal to 1/2? What other important ideas should students know about fractions? Show how you would explain these ideas. Use as many pictures and words as you need.

C R E S S T / U C L A Template #2 Knowledge Representation  Key aspects of ideas, supporting facts and views and their relationships  Relationship is explicit  Organizational options Core and peripheral Hierarchical Cause and effect Chronological  Expert scoring

C R E S S T / U C L A History

C R E S S T / U C L A Genetics

C R E S S T / U C L A Bicycle Pump

C R E S S T / U C L A Key Design Principles from Knowledge Requirements  Knowing why  Knowing what to assess: content plus cognitive demands (problem solving, communication, learning to learn, teamwork, content knowledge)  Knowing how: to make tests that measure and support transfer (application to other topics and situations)

C R E S S T / U C L A To Make Assessment Knowledge Useful for Educators  Timing  Having sufficient content knowledge  Having sufficient knowledge of students  Knowing how to combine results and other sources of information  Knowing where to find help and resources  Knowing what to do (more than one option)

C R E S S T / U C L A Data Interpretation—Design Principles  How to interpret results (in comparison to what?)  How to understand data from sources other than classroom testing  How to integrate sources of test results and form ideas about what to do next

C R E S S T / U C L A Decision Support System :allows the integration of coherent information :allows the identification of conflicting or discordant data :requires the judgment of the user to assign value

C R E S S T / U C L A

QSP  Creates longitudinal records from external sources  Easy to read icons  Grade books for teachers  Digitized examples of student work  Easy to use queries about relationships  Parent conferencing material

C R E S S T / U C L A Groups - for disaggregation Reports - for displaying information graphically Goals - for monitoring improvement Gradebook - for keeping tabs on student progress Digital Portfolio - for examples of student work Resource Kit - to gather locally important information (bottom-up) Functions

C R E S S T / U C L A Communication The teacher decided to send a QSP Progress Report to the student’s parents.

C R E S S T / U C L A Using Assessments and Reflecting on Data Is Hard  Teachers: Fundamental shift from chronological organization to functional organization From what am I doing? To what should each learner be doing now? Time and collaboration  Administration: Taking a chance on change

C R E S S T / U C L A Context for Success of Knowledge-Based Reform  Local ownership of knowledge  Infrastructure and stability  Capacity to investigate  Learning  Congruence or peace with external mandates

C R E S S T / U C L A Usable Knowledge and Support May Get to Useful Knowledge  For assessment knowledge to be useful, it depends upon the context, capacity, and communication of the teaching system  For assessment knowledge to be useful to students, it must go to the heart of why, what, and how they learn