1. – Plutarch, On Listening to Lectures
“The mind is not a vessel to be filled, but a fire to be kindled.”
– Plutarch, On Listening to Lectures

2. Understanding Depth of Knowledge (DOK) for Use in Creating Classroom Assessments
Presented by
Rita Geiger, Educational Consultant
Eugene Earsom, OKAGE Program Director
Kelly Curtright, Director, Social Studies, OSDE

3. Just What Really is “Depth of Knowledge”?
Depth of Knowledge (DOK) is a scale of cognitive demand.

4. Why DOK IS Important!
Teachers of all subjects at all grade levels need to understand all DOK levels.

5. No items written at DOK 4 will appear on the World Geography CRT.
Why DOK IS Important!
The range of cognitive demand for objectives within each grade spans from DOK 1 to DOK 4.
No items written at DOK 4 will appear on the World Geography CRT.

6. Why DOK IS Important!
Instruction, assignments, and classroom assessments must incorporate the expectation of rigor for students associated with the DOK levels of all standards and objectives for World Geography.

7. Depth of Knowledge Level 1 Recall
Recall of a fact, information, or procedure.
Level 2 Skill/Concept
Use information or conceptual knowledge, two or more steps, etc.
Level 3 Strategic Thinking
Requires reasoning, developing plan or a sequence of steps, some complexity, more than one possible answer.
Level 4 Extended Thinking
Requires an investigation, time to think and process multiple conditions of the problem.

8. Beginning at the Beginning:
What DOK is can best be explained by saying first what DOK is NOT.
DOK is not a verb.
DOK is not about the “difficulty” of the task.
DOK is not a grade-level indicator.

9. DOK is Not a Verb Describe . . .
how many apples you see in the picture.
the process of photosynthesis.
how the two political parties are alike.
your analysis of the literary elements in Snow Falling on Cedars as the novel reflected the effect of WWII on America.

10. DOK is About What Follows the Verb
What comes after the verb is more important than the verb itself. “Analyze this sentence to decide if the commas have been used correctly” does not meet the criteria for high cognitive processing. Rationale: The student who has been taught the rule for using commas is merely using the rule.

11. DOK is NOT About Difficulty
Who was the 16th president of the United States? If all of you know the answer, this question is an easy question. Who was the 14th president of the United States? If most of you do not know the answer, this question is a difficult question.

12. DOK is NOT About Difficulty
What is the capital of the Iraq? If all of you know the answer, this question is an easy question. What is the capital of the Guyana? If most of you do not know the answer, this question is a difficult question.

13. DOK is About Intended Outcome, NOT Difficulty!
DOK is a reference to the complexity of mental processing that must occur to answer a question, perform a task, or generate a product.

14. DOK IS About Complexity
Every standard in Geography PASS has a DOK level.
Instruction and classroom assessments should reflect the DOK level of the objective.

15. When Assigning the DOK Level, Consider . . .
the level of work students are most commonly required to perform.
the complexity of the task, rather than its difficulty.
The DOK level describes the kind of thinking involved in a task, not the likelihood that the task will be completed correctly.

17. DOK 1 Activities
Recall details of a story (events, character, plot, setting, etc.).
Identify specific information contained in graphics.
Conduct basic mathematical calculations.
Label locations on a map.
Represent in words or diagrams a scientific concept or
relationship.
Perform routine procedures like measuring length or using
punctuation marks correctly.
Describe the features of a place or people group.
Identify who, what, where, when of a particular event or issue,
list attributes, or define the meaning of terms.

18. DOK 2 Activities
Identify and summarize the major events of a narrative.
Use context cues to identify the meaning of unfamiliar words.
Solve routine multi-step problems.
Describe the cause and effect of a particular event or issue.
Identify patterns in events or behavior.
Compare/contrast people, places, events, and concepts.
Convert information from one form to another form.
Formulate a routine problem/issue given data and conditions.
Organize, represent, and interpret data.

19. DOK 3 Activities
Support ideas, thesis, or predictions with specific evidence,
details, and examples.
Use voice appropriate to purpose and audience.
Identify research questions and design investigations for a
scientific problem.
Develop a scientific model for a complex situation.
Determine the author’s purpose and describe how it affects the
interpretation of a reading selection.
Apply a concept in another context.
Draw conclusions from a variety of sources of information.
Make connections across time and place to explain a concept or
“big idea.”

20. DOK 4 Activities Conduct a project that requires specifying a problem,
designing and conducting an experiment, analyzing its data,
and reporting results and/or solutions.
Analyze and synthesize information from multiple sources.
Describe and illustrate how common themes are found across
texts from different cultures.
Design a mathematical model to inform and solve a practical
or an abstract situation.
Design a thesis, conduct an investigation using multiple
sources, analyze and synthesize the evidence in a written
report (essay) or multimedia presentation, and present to
an audience.

22. Sample Question 1

23. Sample Question 2

24. What Does DOK Look Like in the Classroom?
Level One (Recall) – Name the presidents of the United States in order.

25. What Does DOK Look Like in the Classroom?
Level Two (Skill/Concept) – Using the right and left political spectrum categorize the presidents of the 20th and 21st centuries according to their political standing.

26. What Does DOK Look Like in the Classroom?
Level Three (Strategic Thinking) – Hypothesize how President Dwight D. Eisenhower would react to today’s political situation.

27. What Does DOK Look Like in the Classroom?
Level Four (Extended Thinking) - Analyze the strategies and effectiveness of George H. W. Bush’s war strategies in the Persian Gulf with the war strategies of George W. Bush in Iraq.

28. What Does DOK Look Like in the Classroom?
Level One (Recall) – Name two crops commonly grown in Argentina.

29. What Does DOK Look Like in the Classroom?
Level Two (Skill/Concept) – Make a graph showing the annual production of the five largest crops grown Argentina.

30. What Does DOK Look Like in the Classroom?
Level Three (Strategic Thinking) – Develop a logical argument for planting a particular crop in the Pampas of Argentina, taking into account soils, weather, and other variables.

31. What Does DOK Look Like in the Classroom?
Level Four (Extended Thinking) – Design a three year crop rotation system for an estancia of 500 hectares, using as little chemical fertilizer as possible. Justify your system. Project the expected costs and revenues.

32. Examining the Three Levels of DOK Used on the Geography Criterion Referenced Test (CRT)

33. Level 1 - Recall of Information
Recall facts, terms, concepts, and trends or recognize or identify specific information contained in graphics.

34. Level 1 - Recall of Information
Requires students to identify, list, or define.

35. Level 1 - Recall of Information
Recall who, what, when, and where.

36. Level 1 - Recall of Information
Recognize or identify specific information contained in documents, excerpts, quotations, maps, charts, tables, graphs, or illustrations.

37. Level 2 - Basic Reasoning
Engagement of some mental processing beyond recalling or reproducing a response.

38. Level 2 - Basic Reasoning
Requires students to contrast or compare people, places, events, and concepts.

39. Level 2 - Basic Reasoning
Convert information from one form to another.

40. Level 2 - Basic Reasoning
Classify or sort items into meaningful categories.

41. Level 2 - Basic Reasoning
Draw simple conclusions.

42. Level 2 - Basic Reasoning
Describe, interpret, or explain issues and problems, patterns, reasons, cause and effect, significance or impact, relationships, points of view, or processes.

43. Level 2 - Basic Reasoning
A Level 2 “describe and/or explain” would require students to go beyond a description or explanation of recalled information to describe and/or explain a result or “how” or “why.”

44. Level 3 - Complex Reasoning
Requires reasoning, using evidence, and a higher level of thinking than Level 1 and Level 2.

45. Level 3 - Complex Reasoning
Draw conclusions from multiple or complex stimuli.

46. Level 3 - Complex Reasoning
Use concepts to solve problems.

47. Level 3 - Complex Reasoning
Analyze similarities and differences.

48. Level 3 - Complex Reasoning
Evaluate solutions to problems.

49. Level 3 - Complex Reasoning
Recognize and explain misconceptions.

50. Level 3 - Complex Reasoning
Make connections across time and place to explain a concept or “big idea.”

51. Level 3 - Complex Reasoning
Analyze and synthesize information from multiple sources.

52. Level 3 - Complex Reasoning
Make predictions with evidence as support.

53. OVERVIEW OF ITEM SPECIFICATIONS
For each PASS standard, item specifications are organized under the following headings:
PASS Standard and PASS Objective
Item Specifications
a. Emphasis
b. Stimulus Attributes
c. Format
d. Content Limits
e. Distractor Domain
f. Sample Test Items

54. GENERAL CONSIDERATIONS FOR ITEM WRITING
Use the Item Specifications to help create items for your classroom assessments.

55. GENERAL CONSIDERATIONS FOR ITEM WRITING
Use the language and terms from the World Geography PASS both for instruction and assessment.

56. GENERAL CONSIDERATIONS FOR ITEM WRITING
Test items are varied and address all PASS standards and objectives in World Geography.

57. DOK Distribution on the CRT

58. GENERAL CONSIDERATIONS FOR ITEM WRITING
As much as possible, no item or response choice clues the answer to any other item.
Example:
What term describes the two times during the year when the hours of daylight
and darkness in a day are nearly equal?
(Equinox)

59. GENERAL CONSIDERATIONS FOR ITEM WRITING
Test items are tied closely and particularly to the stimuli from which they derive, so that the impact of outside (prior) knowledge, while never wholly avoidable, is minimized.

60. GENERAL CONSIDERATIONS FOR ITEM WRITING
Each multiple-choice item contains a question and four answer options, only one of which is correct. Correct answers are approximately equally distributed among As, Bs, Cs, and Ds.

61. GENERAL CONSIDERATIONS FOR ITEM WRITING
The four choices are approximately the same length, have the same format, and are syntactically and semantically parallel; students should not be able to rule out a wrong answer or identify a correct response simply by virtue of its looking or sounding different.

62. GENERAL CONSIDERATIONS FOR ITEM WRITING
Distractors adopt the language and sense of the material in the stimuli so that students must think their way to the correct answer rather than simply identify incorrect responses by virtue of a distractor’s obviously inappropriate nature.

63. GENERAL CONSIDERATIONS FOR ITEM WRITING
Distractors should always be plausible (but, of course, incorrect) in the context of the stimulus.

64. GENERAL CONSIDERATIONS FOR ITEM WRITING
Items are focused on what all children should know and be able to do as as they exit seventh grade.

65. GENERAL CONSIDERATIONS FOR ITEM WRITING
Distractors are developed based on the types of errors students are most likely to make.

66. GENERAL CONSIDERATIONS FOR ITEM WRITING
The responses “Both of the above,” “All of the above,” “None of the above,” and “Neither of the above” are not used.

67. GENERAL CONSIDERATIONS FOR ITEM WRITING
The material presented is balanced, culturally diverse, well-written, and of interest to seventh-grade students. The stimuli and items are presented fairly in order to gain a true picture of students’ skills.

68. GENERAL CONSIDERATIONS FOR ITEM WRITING
All items are reviewed to eliminate language or implication that shows bias or is otherwise likely to disadvantage a particular group of students. That is, items do not display unfair representations of gender, race, ethnicity, disability, culture, or religion; nor do items contain elements that are offensive to any such groups.

69. GENERAL CONSIDERATIONS FOR ITEM WRITING
Each item begins with a stem that asks a question or poses a clear problem.

70. GENERAL CONSIDERATIONS FOR ITEM WRITING
Most stems are positively worded—avoiding the use of the word not. If a negative is required, the format is “All of the following except ”

71. Stimulus Materials
At least 50% of the items have appropriate pictorial and graphical representations. Graphs, tables, or figures are clearly associated with their intended items.

72. Survey of the Enacted Curriculum
Web Sites
Alignment Tool
Survey of the Enacted Curriculum

73. Contact Information
Kelly Curtright
Phone: (405)
Eugene Earsom
Phone: (405)

74. “If you want to be successful it’s just this simple
“If you want to be successful it’s just this simple. Know what you are doing. Love what you are doing. And believe in what you are doing.”
– Will Rogers
“If you want to be successful it’s just this simple. Know what you are doing. Love what you are doing. And believe in what you are doing.”
– Will Rogers