– Plutarch, On Listening to Lectures “The mind is not a vessel to be filled, but a fire to be kindled.” – Plutarch, On Listening to Lectures

Understanding Depth of Knowledge (DOK) for Use in Creating Classroom Assessments Presented by Rita Geiger, Educational Consultant Eugene Earsom, OKAGE Program Director Kelly Curtright, Director, Social Studies, OSDE

Just What Really is “Depth of Knowledge”? Depth of Knowledge (DOK) is a scale of cognitive demand.

Why DOK IS Important! Teachers of all subjects at all grade levels need to understand all DOK levels.

No items written at DOK 4 will appear on the World Geography CRT. Why DOK IS Important! The range of cognitive demand for objectives within each grade spans from DOK 1 to DOK 4. No items written at DOK 4 will appear on the World Geography CRT.

Why DOK IS Important! Instruction, assignments, and classroom assessments must incorporate the expectation of rigor for students associated with the DOK levels of all standards and objectives for World Geography.

Depth of Knowledge Level 1 Recall Recall of a fact, information, or procedure. Level 2 Skill/Concept Use information or conceptual knowledge, two or more steps, etc. Level 3 Strategic Thinking Requires reasoning, developing plan or a sequence of steps, some complexity, more than one possible answer. Level 4 Extended Thinking Requires an investigation, time to think and process multiple conditions of the problem.

Beginning at the Beginning: What DOK is can best be explained by saying first what DOK is NOT. DOK is not a verb. DOK is not about the “difficulty” of the task. DOK is not a grade-level indicator.

DOK is Not a Verb Describe . . . how many apples you see in the picture. the process of photosynthesis. how the two political parties are alike. your analysis of the literary elements in Snow Falling on Cedars as the novel reflected the effect of WWII on America.

DOK is About What Follows the Verb What comes after the verb is more important than the verb itself. “Analyze this sentence to decide if the commas have been used correctly” does not meet the criteria for high cognitive processing. Rationale: The student who has been taught the rule for using commas is merely using the rule.

DOK is NOT About Difficulty Who was the 16th president of the United States? If all of you know the answer, this question is an easy question. Who was the 14th president of the United States? If most of you do not know the answer, this question is a difficult question.

DOK is NOT About Difficulty What is the capital of the Iraq? If all of you know the answer, this question is an easy question. What is the capital of the Guyana? If most of you do not know the answer, this question is a difficult question.

DOK is About Intended Outcome, NOT Difficulty! DOK is a reference to the complexity of mental processing that must occur to answer a question, perform a task, or generate a product.

DOK IS About Complexity Every standard in Geography PASS has a DOK level. Instruction and classroom assessments should reflect the DOK level of the objective.

When Assigning the DOK Level, Consider . . . the level of work students are most commonly required to perform. the complexity of the task, rather than its difficulty. The DOK level describes the kind of thinking involved in a task, not the likelihood that the task will be completed correctly.

DOK 1 Activities Recall details of a story (events, character, plot, setting, etc.). Identify specific information contained in graphics. Conduct basic mathematical calculations. Label locations on a map. Represent in words or diagrams a scientific concept or relationship. Perform routine procedures like measuring length or using punctuation marks correctly. Describe the features of a place or people group. Identify who, what, where, when of a particular event or issue, list attributes, or define the meaning of terms.

DOK 2 Activities Identify and summarize the major events of a narrative. Use context cues to identify the meaning of unfamiliar words. Solve routine multi-step problems. Describe the cause and effect of a particular event or issue. Identify patterns in events or behavior. Compare/contrast people, places, events, and concepts. Convert information from one form to another form. Formulate a routine problem/issue given data and conditions. Organize, represent, and interpret data.

DOK 3 Activities Support ideas, thesis, or predictions with specific evidence, details, and examples. Use voice appropriate to purpose and audience. Identify research questions and design investigations for a scientific problem. Develop a scientific model for a complex situation. Determine the author’s purpose and describe how it affects the interpretation of a reading selection. Apply a concept in another context. Draw conclusions from a variety of sources of information. Make connections across time and place to explain a concept or “big idea.”

DOK 4 Activities Conduct a project that requires specifying a problem, designing and conducting an experiment, analyzing its data, and reporting results and/or solutions. Analyze and synthesize information from multiple sources. Describe and illustrate how common themes are found across texts from different cultures. Design a mathematical model to inform and solve a practical or an abstract situation. Design a thesis, conduct an investigation using multiple sources, analyze and synthesize the evidence in a written report (essay) or multimedia presentation, and present to an audience.

Sample Question 1

Sample Question 2

What Does DOK Look Like in the Classroom? Level One (Recall) – Name the presidents of the United States in order.

What Does DOK Look Like in the Classroom? Level Two (Skill/Concept) – Using the right and left political spectrum categorize the presidents of the 20th and 21st centuries according to their political standing.

What Does DOK Look Like in the Classroom? Level Three (Strategic Thinking) – Hypothesize how President Dwight D. Eisenhower would react to today’s political situation.

What Does DOK Look Like in the Classroom? Level Four (Extended Thinking) - Analyze the strategies and effectiveness of George H. W. Bush’s war strategies in the Persian Gulf with the war strategies of George W. Bush in Iraq.

What Does DOK Look Like in the Classroom? Level One (Recall) – Name two crops commonly grown in Argentina.

What Does DOK Look Like in the Classroom? Level Two (Skill/Concept) – Make a graph showing the annual production of the five largest crops grown Argentina.

What Does DOK Look Like in the Classroom? Level Three (Strategic Thinking) – Develop a logical argument for planting a particular crop in the Pampas of Argentina, taking into account soils, weather, and other variables.

What Does DOK Look Like in the Classroom? Level Four (Extended Thinking) – Design a three year crop rotation system for an estancia of 500 hectares, using as little chemical fertilizer as possible. Justify your system. Project the expected costs and revenues.

Examining the Three Levels of DOK Used on the Geography Criterion Referenced Test (CRT)

Level 1 - Recall of Information Recall facts, terms, concepts, and trends or recognize or identify specific information contained in graphics.

Level 1 - Recall of Information Requires students to identify, list, or define.

Level 1 - Recall of Information Recall who, what, when, and where.

Level 1 - Recall of Information Recognize or identify specific information contained in documents, excerpts, quotations, maps, charts, tables, graphs, or illustrations.

Level 2 - Basic Reasoning Engagement of some mental processing beyond recalling or reproducing a response.

Level 2 - Basic Reasoning Requires students to contrast or compare people, places, events, and concepts.

Level 2 - Basic Reasoning Convert information from one form to another.

Level 2 - Basic Reasoning Classify or sort items into meaningful categories.

Level 2 - Basic Reasoning Draw simple conclusions. http://www.srh.noaa.gov/jetstream/synoptic/ll_analyze_app.htm

Level 2 - Basic Reasoning Describe, interpret, or explain issues and problems, patterns, reasons, cause and effect, significance or impact, relationships, points of view, or processes.

Level 2 - Basic Reasoning A Level 2 “describe and/or explain” would require students to go beyond a description or explanation of recalled information to describe and/or explain a result or “how” or “why.”

Level 3 - Complex Reasoning Requires reasoning, using evidence, and a higher level of thinking than Level 1 and Level 2.

Level 3 - Complex Reasoning Draw conclusions from multiple or complex stimuli.

Level 3 - Complex Reasoning Use concepts to solve problems.

Level 3 - Complex Reasoning Analyze similarities and differences.

Level 3 - Complex Reasoning Evaluate solutions to problems.

Level 3 - Complex Reasoning Recognize and explain misconceptions.

Level 3 - Complex Reasoning Make connections across time and place to explain a concept or “big idea.”

Level 3 - Complex Reasoning Analyze and synthesize information from multiple sources.

Level 3 - Complex Reasoning Make predictions with evidence as support.

OVERVIEW OF ITEM SPECIFICATIONS For each PASS standard, item specifications are organized under the following headings: PASS Standard and PASS Objective Item Specifications a. Emphasis b. Stimulus Attributes c. Format d. Content Limits e. Distractor Domain f. Sample Test Items

GENERAL CONSIDERATIONS FOR ITEM WRITING Use the Item Specifications to help create items for your classroom assessments.

GENERAL CONSIDERATIONS FOR ITEM WRITING Use the language and terms from the World Geography PASS both for instruction and assessment.

GENERAL CONSIDERATIONS FOR ITEM WRITING Test items are varied and address all PASS standards and objectives in World Geography.

DOK Distribution on the CRT

GENERAL CONSIDERATIONS FOR ITEM WRITING As much as possible, no item or response choice clues the answer to any other item. Example: What term describes the two times during the year when the hours of daylight and darkness in a day are nearly equal? (Equinox)

GENERAL CONSIDERATIONS FOR ITEM WRITING Test items are tied closely and particularly to the stimuli from which they derive, so that the impact of outside (prior) knowledge, while never wholly avoidable, is minimized.

GENERAL CONSIDERATIONS FOR ITEM WRITING Each multiple-choice item contains a question and four answer options, only one of which is correct. Correct answers are approximately equally distributed among As, Bs, Cs, and Ds.

GENERAL CONSIDERATIONS FOR ITEM WRITING The four choices are approximately the same length, have the same format, and are syntactically and semantically parallel; students should not be able to rule out a wrong answer or identify a correct response simply by virtue of its looking or sounding different.

GENERAL CONSIDERATIONS FOR ITEM WRITING Distractors adopt the language and sense of the material in the stimuli so that students must think their way to the correct answer rather than simply identify incorrect responses by virtue of a distractor’s obviously inappropriate nature.

GENERAL CONSIDERATIONS FOR ITEM WRITING Distractors should always be plausible (but, of course, incorrect) in the context of the stimulus.

GENERAL CONSIDERATIONS FOR ITEM WRITING Items are focused on what all children should know and be able to do as as they exit seventh grade.

GENERAL CONSIDERATIONS FOR ITEM WRITING Distractors are developed based on the types of errors students are most likely to make.

GENERAL CONSIDERATIONS FOR ITEM WRITING The responses “Both of the above,” “All of the above,” “None of the above,” and “Neither of the above” are not used.

GENERAL CONSIDERATIONS FOR ITEM WRITING The material presented is balanced, culturally diverse, well-written, and of interest to seventh-grade students. The stimuli and items are presented fairly in order to gain a true picture of students’ skills.

GENERAL CONSIDERATIONS FOR ITEM WRITING All items are reviewed to eliminate language or implication that shows bias or is otherwise likely to disadvantage a particular group of students. That is, items do not display unfair representations of gender, race, ethnicity, disability, culture, or religion; nor do items contain elements that are offensive to any such groups.

GENERAL CONSIDERATIONS FOR ITEM WRITING Each item begins with a stem that asks a question or poses a clear problem.

GENERAL CONSIDERATIONS FOR ITEM WRITING Most stems are positively worded—avoiding the use of the word not. If a negative is required, the format is “All of the following . . . except. . . .”

Stimulus Materials At least 50% of the items have appropriate pictorial and graphical representations. Graphs, tables, or figures are clearly associated with their intended items.

Survey of the Enacted Curriculum Web Sites http://facstaff.wcer.wisc.edu/normw/ Alignment Tool http://www.wcer.wisc.edu/WAT/index.aspx Survey of the Enacted Curriculum http://www.SECsurvey.org

Contact Information Kelly Curtright Phone: (405) 522-3523 E-mail: Kelly_Curtright@sde.state.ok.us Eugene Earsom Phone: (405) 325-5832 E-mail: earsom@ou.edu

“If you want to be successful it’s just this simple “If you want to be successful it’s just this simple. Know what you are doing. Love what you are doing. And believe in what you are doing.” – Will Rogers “If you want to be successful it’s just this simple. Know what you are doing. Love what you are doing. And believe in what you are doing.” – Will Rogers