Practical Strategies for Student Metacognition Technology, Innovations, & Pedagogy Conference CSU Fresno August 18, 2014 Ed Nuhfer – Director of Faculty Development, Director of Educational Assessment and Professor of Geology (University of Wisconsin, University of Colorado, Idaho State University, California State University (retired) enuhfer@earthlink.net Phone 208-241-5029

“High level teaching" is "…teachers teaching the students something, instructing them in how to produce something, and giving them instruction as to the processes of learning.” Hattie, J. (2009). Visible Learning: A Synthesis of over 800 Meta-Analyses Relating to Achievement. New York, New York: Routledge. P. 11.

CCER, BIAC, OECD/CERI Colloquium Metacurriculum for Metacognition 14-15 October 2013 Instruments Knowledge or Skill Learning-across-curriculum modules Strategies and developing a learning philosophy Reading Reflections Reflection & Monitoring Exam Wrappers Reflection, Monitoring & Evaluation Knowledge Surveys Goal-setting, Monitoring & Evaluation Developing self-assessment & self-efficacy Learning Journals Evaluation & Goal-setting Developing self-assessment & self-efficacy Metadisciplinary Concept Inventories (Assessment) Reasoning, respect for diverse ways of knowing, overarching philosophy for becoming educated

Available Learning-Across-the-Curriculum Modules From CSU Faculty Developers

Used by permission of S. E. Petersen Used by permission of S. E. Petersen. Positron emission tomographic studies of the processing of single words.

Tools and Scales of Learning Challenges

“Exam Wrappers” What did I do well on and why? 95% for the exam plus 5% for submitting the following metacognitive reflection: What did I do well on and why? What did I do less well and why? When I encounter a similar challenge, what strategy will I use to perform better?

Reflective Learning Journals (1) Summarize your assumptions, beliefs, understanding that you had when you first encountered the topic assigned here, before you started to engage it during learning it in this course Assessing what one thinks at the start of a task is a common theme in metacognition. A free writing exercise given at the introduction of a new topic is ideal for helping students capture their baseline knowledge and archive that for later use in a journal entry.

Reflective Learning Journals (2) Disclose new vocabulary that you acquired during your study of this topic. Include vocabulary that you may have acquired as part of your process in items 4-6 below. Building vocabulary increases reading comprehension and personal ability to access information.

Reflective Learning Journals (3) Discover and describe connections that you can make between the topic and several other class sessions, exercises, readings, homework. Note in SPECIFICS what you found from making these connections that informed or expanded your understanding of the topic. Be sure to note how this learning is relevant to one or more of the published learning outcomes for the course. Refer to the specific outcome(s). Metacognitive reflection uncovers connections that are otherwise difficult to access. Practice in seeking connections between lessons and larger ideas is immensely valuable.

Reflective Learning Journals (4) So as avoid depending on materials given to you in class, take action on your own behalf by looking to other resources. To provide evidence of this learning, describe IN SPECIFICS what you learned and from what source(s). Assigning students to go beyond what authority (instructors) offers them moves beginning students to higher-level stages of thinking. Metacognition trains independent thinking.

Reflective Learning Journals (5) “Close the loop” by comparing what you knew in #1 against what you know now. Seeing one's growth affirmed builds self-efficacy.

Journal template furnished as “blueline helpsheet”

Journal template furnished as “blueline helpsheet” Download it from http://profcamp.tripod.com/rljhelpsheet.rtf.

Knowledge surveys Metacognitive self-assessments If I think I can do this, how well can I do this? Can I look at the items and start to distinguish which emphasize knowledge, skills or reasoning? For construction, consult the tutorials at http://elixr.merlot.org/assessment-evaluation/knowledge-surveys?noCache=191:1306864598

Knowledge surveys…

Our challenge is to keep students in constant contact with their metacognition… KS ITEM – COURSE CONTENT: 0 1 2 I can explain why it was necessary to first understand the principles of original horizontality, superposition, and crosscutting relationships before scientists could begin to derive the theory of plate tectonics. KS ITEM — METADISCPLINARY OUTCOME: 0 1 2 I can employ examples to illustrate key differences between the ways of knowing of science and of technology. KS ITEM — STAGE IN DEVELOPMENT OF THINKING: 0 1 2 My friend tells me: "Because everyone has a right to his or her opinion, we respect others by accepting all opinions as equally valid.” I can recognize the stage of intellectual development revealed by that statement.

Hattie (2009, p 39 - 43) found present understanding of level of achievement as the most significant attribute of students for meeting their future goals.

Self-Efficacy “If I have the belief that I can do it, I shall surely acquire the capacity to do it, even if I may not have it at the beginning.” Mahatma Gandhi

KSs as an aid to teaching self-efficacy Self-efficacy—realistic belief in self regarding the proficiency that one can attain with instruction and support. Alfred Bandura established strong self-efficacy as one of the most important predictors of student success (www.des.emory.edu/mfp/self-efficacy.htm) PRE-COURSE: “Good heavens! Will we be able to learn ALL this?" POST COURSE: "I really did learn that much.”

Correlations between performance and several metacognitive self-assessments (n = 1011) First Impression Global Self-assessment Knowledge Survey item-by-item Self-Assessment Post-Knowledge Survey Overall Self- Assessment Post-SLCI Overall Self-assessment (*n=590) SLCI Actual Score r = 0.29 r = 0.58 r = 0.45 r = 0.62* Average 72 % Average 75% Average 79% Average 76% Average* 76%

Metadisciplinary Ways of Knowing You have a handout with outcomes for six metadisciplines If not, grab it from http://profcamp.tripod.com/metadoutcomeslist.pdf

Example of the reasoning component of “Science literacy” Insert your own metadiscipline’s outcomes in place of “science” as you consider what follows. If a citizen understands science’s way of knowing, how can we recognize this? How can students recognize when they are “getting it?”

Concepts for Citizen Literacy in the Metadiscipline of Science 1. Science explains physical phenomena based upon testable information about the physical world. 2. In modern life, science literacy is important to both personal and collective decisions that involve science content and reasoning. 3. Doubt plays necessary roles in advancing science. 4. Scientists use evidence-based reasoning to select which among several competing working hypotheses best explains a physical phenomenon. 5. A theory in science is a unifying explanation for observations that result from testing several hypotheses. 6. Peer review generally leads to better understanding of physical phenomena than can the unquestioned conclusions of involved investigators. 7. Science can test certain kinds of hypotheses through controlled experiments. 8. All science rests on fundamental assumptions about the physical world. 9. Science differs from technology. 10. Scientific knowledge is discovered, and some discoveries require an important history. 11. Science employs modeling as a method for understanding the physical world. 12. Scientific knowledge imparts power that must be used ethically.

Concepts restated as 12 Outcomes for Science Literacy Students will be able to… 1. Define the domain of science and determine whether a statement constitutes a hypothesis that can be resolved within that domain. 2. Describe through example how science literacy is important in everyday life to an educated person. 3. Explain why the attribute of doubt has value in science. 4. Explain how scientists select which among several competing working hypotheses best explains a physical phenomenon. 5. Explain how "theory" as used and understood in science differs from "theory "as commonly used and understood by the general public. 6. Explain why peer review generally improves our quality of knowing within science. 7. Explain how science employs the method of reproducible experiments to understand and explain the physical world. 8. Articulate how science’s way of knowing rests on some assumptions. 9. Distinguish between science and technology by examples of how these are different frameworks of reasoning. 10. Cite a single major theory from one of the science disciplines and explain its historical development. 11. Explain and provide an example of how modeling is used in science. 12. Explain why ethical decision-making becomes increasingly important to a society as it becomes increasingly advanced in science.

OK…Can we assess this stuff?

The Instrument For each outcome In addition… Construct several concept inventory items. Use established methods for drafting items that have been developed in other concept inventories. In addition… Test reasoning, not factual knowledge. Administer the inventory under the conditions in which a citizen will use common information.

…initial instrument constructed 2008-2010 through the collegial efforts of Edward Nuhfer, Faculty Development & Geology, Channel Islands Jerry Clifford, Physics, Channel Islands Christopher Cogan, Environmental Sciences & Resource Management, Channel Islands Anya Goodman, Biochemistry, San Luis Obispo Carl Kloock, Biology, Bakersfield Beth Stoeckly, Physics, Channel Islands Christopher Wheeler, Geology, Channel Islands Gregory Wood, Physics, Channel Islands Natalie Zayas, Science Education & Environmental Sciences, Monterey Bay Note: 9 people, 5 disciplines, and 4 CSU campuses

Science Literacy Concept Inventory Incorporates 25 validated items that map to the twelve concepts Reliability of .85 Tested on over 16,000 students in about 30 institutions

Outcome. Student can define the domain of science and determine whether a statement constitutes a hypothesis that can be resolved within that domain. Concept Some Misconceptions Science explains physical phenomena based upon testable information about the physical world. Science is on a mission to refute religion; scientists study the paranormal; untestable statements are like scientific hypotheses.

Which of the following statements presents a hypothesis that science can now easily resolve? A. Warts can be cured by holding quartz crystals on them daily for a week. B. A classmate sitting in the room can see the auras of other students. C. Radio City Music Hall in New York is haunted by several spirits. D. People with chronic illnesses have them as punishment for past misdeeds. Correct answer is: A This is testable. Answer B: False. Not testable. Answer C: False. Not testable. Answer D: False. Not testable.

Which of the following statements presents a hypothesis that science can now easily resolve? A. Warts can be cured by holding quartz crystals on them daily for a week. B. A classmate sitting in the room can see the auras of other students. C. Radio City Music Hall in New York is haunted by several spirits. D. People with chronic illnesses have them as punishment for past misdeeds. Correct answer is: A This is testable. Answer B: False. Not testable. Answer C: False. Not testable. Answer D: False. Not testable.

What did we learn that made this so interesting? Starting with…Do experts (professors) outscore novices (students)?

YES! They do! Also, students on average do come to us with some science literacy: zero literacy = about 25% (random guessing).

Does knowing “stuff” advantage a person?

Professors in Metadisciplines by Mean Score %

We can learn about our students’ science literacy

Our current GE science courses don’t produce science literacy (N = 16,623)…

Development seems more apparent long term through ranks (N = 16,599)… The major gains are breaks that may result from attrition as well as growth in ability to reason.

We can learn much about our students

First Generation Students: 65% vs 70% (N = 16,203) Students from families privileged with higher education generally achieve higher.

English as a First Language: 62% vs 70% (N = 16,182) Students whose first language is English generally achieve significantly higher scores.

Commitment to Interest in Science: 66% vs 72% (N = 16,190) Differences are already significant in freshman year, and becomes increasingly so with each rank.

Gender-equality (n = 16,617) And this holds true so far in our study within every ethnic group except one.

Middle Eastern (n = 373)

Ethnicities Show Some Profound Differences

They Differ Across Three Factors

Regression Equation to Account for Socio-economic advantages/disadvantages based on 16,263 undergraduates Corrected Score = 0.62424 - 0.032575 (First Generation) +0.06206 (English as Native Language) + 0.05323 (Science Interest).

Ethnicity adjusted for Socio-economic advantages

SLCI by Institutions with Other Measures As Eric Gaze and the Quantitative Literacy test researchers have noted, these little literacy tests may be powerful predictors of student success.

Institutions

Given that our GE science courses don’t produce much increase in reasoning of science literacy…

For continued learning about metacognition, we recommend that participants enroll with the Improve with Metacognition forum at http://www.improvewithmetacognition.com/2014/07/