1 Language, Learning, and Assessment: Improving Validity Eva L. Baker UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing (CRESST) 26 th Annual Language Testing Research Colloquium Of the International Language Testing Association Temecula, CA March 26, 2004

2 Today’s Focus  Language and identity—a personal note  Quick assessment policy tour  Need for cognitive model-based assessment to link learning to external outcome measures  Examples  Research agenda involving language and assessment interplay

3 Assessment Policy Tour © 1965 Fantasy Records

4 Assessment Policy Tour (Cont’d)  Accountability is pushing testing practice  Soon to have greater role in postsecondary  Assessment and testing high priority in K-12, language certification, and adult and professional learning  Students’ language background a major area

5 K-12 Accountability Requirements  Goals (standards) to be linked to instruction, classroom assessments  External examinations are to be lined up with goals and instruction  Few examples of alignment, unless goals are restricted  Lack procedures and measures to document level of practice  Early growth likely not sustainable  Transfer of learning a major issue

6 Description: Extra comfort for senior dogs. Our popular orthopedic pet bed, made extra thick for aging dogs. A full 4" of medical grade convoluted foam supports bones and joints, and the elevated headrest provides proper neck and spine alignment.

7 Joyful manifestation of the heart’s desire.

8 Cognitive Model-Based Assessment  Cross-curricular cognitive families define core constructs  Embed in content, like linguistic rules in AI language understanding systems  Derives from work in writing assessment and AI systems  Evidence that it works across subject matters, types of learning, and students

9 Intellectual Capital Cognitive Families Content Understanding Problem Solving Teamwork and Collaboration Metacognition Learning to Learn Communication Learning

10 Cognitive Demands Examples  Factual knowledge  Definitions, propositions (e.g., statements about events, causes, relations; “math facts”; formulas), etc.  Terms and symbols (beyond arithmetic)  Identify domain by letter: A for algebra, G for geometry, etc.  Conceptual knowledge  Knowledge of “more complex, organized knowledge forms,” including classification schemes, principles and generalizations, theories, models, and knowledge structures  Procedural knowledge  Knowledge about how to do something, including mathematical skills and algorithms, techniques and methods Knowledge Categories

11 Cognitive Demands Examples (Cont.)  Recall information  Apply information  Interpret figures, charts, or tables  Transform representations  Change or move from one representation to another (e.g., language to diagram, text to equation, graph to equation)  Multi-step reasoning  Requires “students to generate an intermediate image, construct, or sub-problem before solving the original problem” Process Categories

12 From Science to Models to Templates DOMAIN- INDEPENDENT COGNITIVE PRINCIPLES CONTENT C B A TEMPLATE MODEL SCIENTIFIC FINDINGS COGNITIVE DEMANDS SCIENTIFIC FINDINGS SUBJECT MATTER SPECIFIC MODELS CONDITIONS TOPICS OF TASKS SPECIFIC SCORING FEATURES

13 From Templates to Tasks CB A TEMPLATE TASK

14 Templates for Model of Content Understanding  Primary source materials in each domain  Student required to integrate prior knowledge and principles to succeed  Scored by using expert model in subject matter

15 Content Understanding Template #1 Explanation  An array of primary source materials  A prompt that asks for an explanation in context  Constructed (written) answer  Evaluated by means of a scoring rubric

16 Excerpts from U.S. History Primary Source Documents Judge Douglas made two points upon my recent speech at Springfield. He says they are to be the issues of this campaign. The first one of these points he bases upon the language in a speech which I delivered at Springfield, which I believe I can quote correctly from memory. I said there that “we are now far into the fifth year since a policy was instituted for the avowed object, and with the confident promise, of putting an end to slavery agitation; under the operation of that policy, that agitation had not only not ceased, but had constantly augmented.” “I believe it will not cease until a crisis shall have been reached and passed. ‘A house divided against itself cannot stand.’ I believe this Government cannot endure permanently, half slave and half free.” “I do not expect the Union to be dissolved”—I am quoting from my speech— “I do not expect the house to fall, but I do expect it will cease to be divided. It will become all one thing or the other. Either the opponents of slavery will arrest the spread of it and place it where the public mind shall rest, in the belief that it is in the course of ultimate extinction, or its advocates will push it forward until it shall become alike lawful in all the States, North as well as South.” [text continues] ABRAHAM LINCOLN

17 U.S. History Writing Assignment: Civil War Era Be sure to show the relationships among your ideas and facts. Your essay should be based on two major sources: 1. The general concepts and specific facts you know about United States history, and especially what you know about the history of the Civil War. 2. What you have learned from the readings yesterday. Imagine that it is 1858 and you are an educated citizen living in Illinois. Because you are interested in politics and always keep yourself well- informed, you make a special trip to hear Abraham Lincoln and Stephen A. Douglas debating during their campaigns for the Senate seat representing Illinois. After the debates you return home, where your cousin asks you about some of the problems that are facing the nation at this time. Write an essay in which you explain the most important ideas and issues your cousin should understand.

18 Excerpts from Hawaiian History Primary Source Documents LILIUOKALANI For many years our sovereigns had welcomed the advice of American residents who had established industries on the Islands. As they became wealthy, their greed and their love of power increased. Although settled among us, and drawing their wealth from resources, they were alien to us in their customs and ideas, and desired above all things to secure their own personal benefit. Kalakaua valued the commercial and industrial prosperity of his kingdom highly. He sought honestly to secure it for every class of people, alien or native. Kalakaua’s highest desire was to be a true sovereign, the chief servant of a happy, prosperous, and progressive people. And now, without any provocation on the part of the king, having matured their plans in secret, the men of foreign birth rose one day en masse, called a public meeting, and forced the king to sign a constitution of their own preparation, a document which deprived [him] of all power and practically took away the franchise from the Hawaiian race.

19 Content Knowledge Prompt: Hawaiian History Writing Assignment—Bayonet Constitution Be sure to show the relationships among your ideas and facts. Your essay should be based on two major sources: 1. The general concepts and specific facts you know about Hawaiian history, and especially what you know about the period of the Bayonet Constitution. 2. What you have learned from the readings yesterday. Imagine you are in a class that has been studying Hawaiian history. One of your friends, who is a new student in the class, has missed all the classes. Recently, your class began studying the Bayonet Constitution. Your friend is very interested in this topic and asks you to explain everything that you have learned about it. Write an essay explaining the most important ideas you want your friend to understand. Include what you have already learned in class about Hawaiian history, and what you have learned from the texts you have just read. While you write, think about what Thurston and Liliuokalani said about the Bayonet Constitution, and what is shown in the other materials.

20 Content Knowledge Prompt (Cont’d) *From Hawaii’s Story by Hawaii’s Queen, Liliuokalani (Boston: Lee and Shepard Publishers, 1898). It may be asked, “Why did the king give them his signature?” I answer without hesitation, because he had discovered traitors among his most trusted friends and because the conspirators were ripe for revolution, and had taken measures to have him assassinated if he refused. It has been known ever since that day as “The Bayonet Constitution,” and the name is well-chosen; for the cruel treatment received by the king from the military companies. [text continues] Explain to your friend who missed class the reasons and differences for the Queen and the Senator’s approach to Hawaii’s future. Scoring Rubric General impression (on task) Principles and themes Prior knowledge Relevant concrete examples Avoidance of misconceptions

21 History Template

22 Benefits  Architecture for design of classroom instruction, assessments, and external tests, not multipurpose test, but multipurpose architecture  Transfer of learning  Flexibility of teaching  Vertical coherence  Specific procedures  Evidence base

23 Measuring Transfer of Learning  Not teach x, measure y: but describe variations in conditions, task content, criteria that permit students to demonstrate their flexibility  Learning to learn measurement  Metacognition and self-regulation  Time to learn new domains or tasks  Safeguard against teaching to the test as a preferred intervention

24 How Language Testing Influences Achievement Testing  Rubrics and procedures for assessing open ended performance  Linguistic analysis as accommodations  Linguistic descriptors as key task dimension  Arguments about language, content, and context

25 Toward Cognitively Oriented Constructs for English Language Development Exams (K-12)  Ironies  Expedience and science  Social and academic constructs  Measure of beginning skill or transfer  Relationships to other external exams not clear  Characterizing text  Readability  Lexiles  LEARNOME

26 Towards Common Language Across Domains  Genome: “Complete set of instructions”  LEARNOME: “Complete map of features in learning domain”

27 Preliminary Exploration of the LEARNOME  Fundamental and generalizable research on identification of components  Cognitive demands, e.g., problem solving  Linguistic requirements, i.e., discourse, syntax, lexicon  Task, e.g., context, constraints  Content (ontologies)  Descriptive language—granularity  Modeling and representation of results  Empirical verification

28 Research Interactions: Language Testing and Assessment  Model-based R&D  EFF Adult Literacy  Questions about differential validity and fairness

29 EFF Assessment Consortium: SRI International & Center for Literacy Studies. (2004, January). EFF standards and performance level descriptors for: Reading, writing, math, speaking, and Listening. Washington, DC: National Institute for Literacy.

30 EFF Assessment Consortium: SRI International & Center for Literacy Studies. (2004, January). EFF standards and performance level descriptors for: Reading, writing, math, speaking, and Listening. Washington, DC: National Institute for Literacy.

31 EFF Assessment Consortium: SRI International & Center for Literacy Studies. (2004, January). EFF standards and performance level Descriptors for: Reading, writing, math, speaking, and Listening. Washington, DC: National Institute for Literacy.

32 EFF Assessment Consortium: SRI International & Center for Literacy Studies. (2004, January). EFF standards and performance level descriptors for: Reading, writing, math, speaking, and Listening. Washington, DC: National Institute for Literacy.

33 EFF Assessment Consortium: SRI International & Center for Literacy Studies. (2004, January). EFF standards and performance level descriptors for: Reading, writing, math, speaking, and Listening. Washington, DC: National Institute for Literacy.

34 EFF Assessment Consortium: SRI International & Center for Literacy Studies. (2004, January). EFF standards and performance level descriptors for: Reading, writing, math, speaking, and Listening. Washington, DC: National Institute for Literacy.

35 EFF Assessment Consortium: SRI International & Center for Literacy Studies. (2004, January). EFF standards and performance level descriptors for: Reading, writing, math, speaking, and Listening. Washington, DC: National Institute for Literacy.

36 EFF Assessment Consortium: SRI International & Center for Literacy Studies. (2004, January). EFF standards and performance level descriptors for: Reading, writing, math, speaking, and Listening. Washington, DC: National Institute for Literacy.

37 EFF Assessment Consortium: SRI International & Center for Literacy Studies. (2004, January). EFF standards and performance level descriptors for: Reading, writing, math, speaking, and Listening. Washington, DC: National Institute for Literacy.

38 EFF Assessment Consortium: SRI International & Center for Literacy Studies. (2004, January). EFF standards and performance level descriptors for: Reading, writing, math, speaking, and Listening. Washington, DC: National Institute for Literacy.

39 EFF Assessment Consortium: SRI International & Center for Literacy Studies. (2004, January). EFF standards and performance level descriptors for: Reading, writing, math, speaking, and Listening. Washington, DC: National Institute for Literacy.

40 Research Interactions: Language Testing and Assessment (Cont’d)  Technology  TBALL project—NSF education, linguistics, electrical engineering—automated testing of ELL children  Essay scoring—towards propositional rather than regression solutions  Authoring systems with smarts  Simulation architecture and narrative structures

41

42 Research Interactions: Language Testing and Assessment (Cont’d)  Validity inferences for different groups  Comparability among tasks, paths  Alternative to norms for comparisons among groups

43 Priorities  Linking context, purposes, and evidence to an appropriate grain-sized framework  Document growth patterns associated with different instructional paths  Explore validity interpretations for ELL groups

44 CRESST Web Site