Metacognition - Outline

Causal Effects of Conscious Experience Broadest Case: Mental states  Behavior I want a beer, I think beer is in fridge,  I open fridge A Narrower Case (Metacognition) Knowledge about cognition  control cognition I believe I will not remember the name of the person I just met  I may take special measures to commit her name to memory (think that “Rita” rhymes with “pita”).

Metacognition Knowledge about cognition (a conscious experience). –Knowledge about tasks (e.g., ‘rote vs deep encoding’) –Knowledge about persons (e.g., ‘I’ am not good at this’) Control of cognitive processes. For example, –Which strategy to use –How much time to allocate to studying Monitoring of cognitive processes. For example, –Ease of Learning: “learning names is difficult” –Judgment of Learning (JoL): “I won’t remember that name” –Feeling of Knowing (FoK): “I feel that I know her name” –Confidence Judgment: “I’m not sure, it may be ‘Rita’”

Nelson & Narens, 1990 In Advance Of Learning EncodingRetrievalRetention Ongoing Learning Self-directed search Maintenance of Knowledge Output of Response Confidence in Retrieved answer Feeling of Knowledge Judgment of Learning Monitoring Select Search Strategy Allocate Study Time Select Encoding strategy Control Terminate Search Ease-of- Learning Judgment

Judgment of Learning (JoL) JoL: “I will I be able to remember this at a later time (at test)” Example : - Study this pair:Captain - Carbon - Make a JoL:“How likely that I will remember the target word that went with “Captain”? 25% 50% 75% 100% - Test: Captain ________ Findings: -JoL correlates with recall - if I think I know it, it’s likely I do know it -but this correlation is far from perfect (usually less than.50) - You can also have aggregate JoL at the end of the list.

Monitoring Effectiveness: JoL & recall Resolution (aka discrimination accuracy or relative accuracy) –The extent to which the subject is able to distinguish between answers that are more likely or less likely to be correct. Calibration –Whether there is overconfidence or underconfidence

How do we judge what we know and what we don’t? That is, what cues are used for JoL? A hint to this question comes from studies looking at immediate JoL vs. delayed JoL Immediate JoL –while seeing word pair (spoon – cuiller) –Correlation with recall was low (r =.35) Delayed JoL: –At the end of list –seeing only cue (spoon - ___) –Excellent correlation with recall (r =.90) –2 nd graders and old adults also provide good delayed JoL Nelson & Dunlosky, 1991

So it seems that we (mis)use the target to assess the cue effectiveness To directly test this idea –Water – ocean –Chicken – penguin (nelson & Koriat )

Mnemonic cues: cues that give rise to the ‘feeling’ of an item having been encoded. Two types of mnemonic cues: –Intrinsic cues: Properties of the items that (subjects believe to) influence memory (e.g., high vs. low frequency words) –Extrinsic cues: Conditions of the task that (subjects believe to) influence memory (e.g., repetition) Subjects underestimate some cues (e.g, repetition) Subjects incorrectly believe that low frequency words would be harder to recognize than high frequency ones

Relation between JoL & study time Can Judgment of Learning control how long we study? 1 - Study English - French pairs:- Pen - Stylo - Free allocation of study time - Make a JoL:- How likely will you recall “stylo” when presented with ‘Pen’? -Test: - Pen ________ Findings 1. students allocated more study time to items which were judged to be more difficult. 2. still remembered more of the easy ones (labor-in-vain). Did not compensate adequately for difficulty 1 what is the causal effect of conscious experience on behavior? (Nelson & Leonesio, 1988)

Relation between JoL & study choice Can we use Judgment of Learning to effectively choose what to study? Two types of essay (within-subject): - easy (e.g., why we need to take vitamins) - hard (post-modern interpretations of neo-classical fiction) Amount of study time available - One group was given a limited amount of time - The other group was given a great deal of time Results: Adaptive use of study time. When time limited: subjects focused on easy items when time unlimited: focus on hard items Original study with Ivy League students; replicated with Inner City Public schools in NY (6th grade) Son and Metcalfe (2000)

Feeling of Knowing (FoK) 1.Recall: What is the capital of Australia? ____ 2.FoK Judgment: I am __ % sure I will recognize its name 3.Recognition Is it: A)Sydney B) Melbourne C) Canberra D) Perth -FoK accuracy usually ranges from.35 to.60 -Similar range for 6-y olds & for older adults

Tip of the Tongue (ToT) ToT: “I know this word and I will retrieve it soon” - What is the last name of the first person to set foot on the moon? - What is the name of the large flightless bird from Africa? - What is the name of the religious group from Northern India whose men where large turbans wrapped around their head? ToT: -correlates with recall & recognition -correlates with partial information about the target -but these correlations are far from perfect -ToT seems to be caused by the familiarity of the topic and the accessibility of partial information -The same metaphor is used in many languages

Monitoring and control processes in ‘free- report’ memory performance Performance in memory tasks depends on: –Memory per se –Memory monitoring (i.e., the subjective assessment that the answer that comes to mind is correct) –Memory control (e.g., decide to report or withhold the answer based on confidence and consequences) Examples: - “answer 4 out of 5 questions”: the student with better metacognitive skills will do better (other things being equal). - multiple choice: is there penalty or not? - “memoirs” vs. “jury”: In a capital punishment case we may withhold memories that we deem acceptable to voice in a memoir, because in the former the potential cost of error is high

Input Question retrieve monitor Long-term memory Best candidate answer Assessed Probability (Pa) RetrievalMonitoringPerformanceControl Set Response criterion Probability (Prc) Situational demands/ Payoff Pa > Pcr ? - report option - accuracy incentive yes no report withhold Koriat & Goldsmith, 1996

Factors that contribute to free- report memory performance Overall retention (memory per se) Monitoring effectiveness: the extent to which the assessed probabilities successfully differentiate correct from incorrect candidate answers.(*) Control sensitivity: the extent to which withholding or volunteering is in fact based on monitoring output Response criterion setting: whether the probability threshold is set in accordance with the payoff schedule. * Being able to distinguish what you know from what you don’t know.

Testing the model (Koriat & Goldsmith, 1996, exp. 1) General knowledge questions –Phase 1: Forced-report (forced recall) Confidence rate –Phase 2: (same items) Free report (free recall) Half subjects with high accuracy incentive (penalty) Results –Good monitoring & control high correlation between accuracy in forced recall and confidence rate Very high correlation between confidence and report in phase 2 –Adaptive response criterion People withhold more items when penalty (but there was a quantity/accuracy trade-off)

Testing the model (Koriat & Goldsmith, 1996, exp. 2) Same method, except that Monitoring was manipulated with two types of items: –Typical items (good confidence-accuracy correlation) –Deceiving items (items people are usually sure and wrong) What is the capital of Australia? Results –Unlike the ‘good’ monitoring condition (typical items) for the ‘bad’ monitoring condition (deceiving items): free-report did little to increase accuracy, and the accuracy-quantity tradeoff was much larger than for typical items Monitoring might also be impaired: –in special populations: Children, Korsakoff, frontal patients. –Due to priming

Goldsmith & koriat toronto 05 Underconfidence with practice Water-ocean Study times

Witness testimony -While deliberating, jurors rely upon their memories of the trial (availability heuristic) -Jurors who are very confident about their memories have the largest impact during deliberation (Kassin & Wrightsman, 1988) -But is that likely to be more convincing - However, this is based on the assumption that confidence is correlated with accuracy, is it?

Witness testimony (cont’d) - -Open ended questioning and free recall are preferable to direct questioning and recognition tests, which tend to contaminate memories. -Witnesses should be reassured that “I don’t’ remember” is an acceptable answer. -See Hunt& Ellis textbook

EOL: inferential (performance predicition) –Prediction of one’s memory span –5 y-old are overconfident –May be due to unfamiliarity (much better in how far they can jump) –At grade 4, improve calibration due in second prediction (but not at grade 3) –May be kids confound predicition with wishful thinking (what I want to get), as they do better in predicting others’ performance

Cues to FoK

Metamemory affects memory performance Imagine a multiple choice test in which –There are 5 choices –1 point if answer is correct –0 point if there is no answer (omission) –-.25 point if error –Instructions “wild guesses will be penalized”, Problems : –Instructions are vague: what counts as ‘wild’ Cultural, gender, & personality biases in risk taking (control) –Even with precise instructions there may be individual differences in monitoring (e.g., overconfident)

Are most confident jurors also the most reliable ones? view videotape of actual murder trial make a global JoL (“I will remember __% of events in the trial”) Answer questions about the trial & provide a confidence judgment for each answer Results JoL did not correlate with accuracy of response More confident jurors speak up (even though their accuracy is no better than others’ accuracy) –This may explain why deliberation does not enhance accuracy –May also occur in study groups, in which the overconfident pre- med guy runs the show despite not having done the readings Pritchard & Keenan, 1999, JEP:Applied

Confidence judgments An example of a confidence judgment is when you mark a question in the MC exam so you can go back to it at a later time The typical procedure is to ask a general knowledge question in a multiple choice format –What is the capital of North Carolina? A. Charlotte, B. Raleigh –Provide a confidence judgment: 50% = guessing 100% = certain –Confidence is positively correlated with accuracy (resolution), but

Calibration of judgments Calibration = mean judgment – mean performance For confidence judgments –There is over-confidence in hard items and –There is under- confidence in easy ones –People are poorly calibrated, suggesting they are insensitive to how much they know, but why? –One possibility is confirmation bias: People might consider reasons why an answer may be correct and fail to consider why the answer might be wrong –When people are asked to provide reasons why their answer might be incorrect, their judgment becomes much better calibrated (koriat ’80)

JoL & FoK Are JoL & FoK based on the same info? Is r > 0? 20 paired associates, cued recall, learnt to criterion –Half the pairs needed to be recalled correctly once –The other half needed to be recalled 4 times (overlearnt) –After item reached criterion -> immediate JoL (affected by overlearning) –4 week retention interval –Recall test (better for overlearnt items) (relative acc of JoL =.3) –FoK to each non-recalled item (unaffected by overlearning; relative acc of FoK =.2) –Recognition test of non-recalled items –Correlation bt JoL & FoK =.17 (very low) Leonesio & Nelson, 1990

How do we come up with a metamemory judgment? The Direct-Access Hypothesis –Things that we store in our mind have a memory trace –Some things have stronger trace than others –Our JoL, FoK, & confidence judgments are based on those traces. –In the case of the FoK we cannot access a memory but even in those cases we can experience the trace.

Evidence against the Direct-Access Hypothesis general knowledge questions, followed by FoK, and recognition of unrecalled items Two types of questions –Standard items: relative accuracy of FoK was.35 In other words, FoK predicted recognition –Deceiving items (e.g. ‘name the capital of California ’ relative accuracy of FoK was 0 In other words, people were bad at predicting recognition Koriat, 1995

More evidence against the Direct-Access Hypothesis Trivia questions (semantic memory) –Answers were grouped based on how long it took subjects to respond (a proxy for difficulty) Easy items required little processing of the event (episodic memory) Hard items required more elaboration (deeper processing) Immediate JoL re: free recall in 20 mins Test: Free recall of answers (episodic recall) –Because Hard items required deeper processing, they are better recalled! –If subjects relied on direct-access, harder items should receive higher JoL (but they don’t!) –This suggests subjects relied in an effort heuristic (hard -> low) Benjamin, Bjork, & Schwartz, 1998

More evidence against the Direct-Access Hypothesis List of Words –High frequency words –Low frequency words Which type will be better recognized? Immediate JoL re: recognition Recognition Test –Low frequency words are easier to recognized –If subjects relied on direct-access, low frequency words should receive higher JoL (but they don’t!) –This suggests subjects relied on (incorrect) beliefs about the effect of word frequency Benjamin,, 1998

Cue-Familiarity Hypothesis of metacognitive judgments FoK are based on familiarity with the cue (question), rather than memory traces of the target (answer) If FoK is high, then people attempt target retrieval Since cue familiarity is likely to be related to target familiarity, it is somewhat predictive (thus the finding of FoK relative accuracy) Priming subjects with the cues leads to increased FoK (without increased in recognition) Reder, 1987

Accessibility Hypothesis of metacognitive judgments FoK are based on accessibility of the target It is different from direct access hypothesis in that –Information IS accessed –The accessed information need not to be accurate (tricky questions) –the fluency with which the information is accessed is important (e.g., trivia study, Benjamin et al.) Koriat, 1993

Cue-Familiarity vs. Target Accessibility hypotheses Proactive interference task with three conditions –AB AB: familiar cue; fluent access to target –AD AB: familiar cue; difficult access to target –CD AB: novel cue; modest access to target Test –Paired associate recall –FoK for incorrectly recalled items –Recognition test Results favored cue-familiarity hypotheses However, both hypotheses may be true as they are not mutually incompatible. Cue-familiarity acts first, if high then people attempt retrieval, point at which accessibility influences FoK Metcalfe, Schwartz, & Joaquim 1993

How well do students judge their learning? Study a paragraph Make a global JoL Recall Low correlation (.27)

Inputs for metamemory judgments Immediate JoL –Ease of processing during study (begg et al, 1989; Hertzog et al, 2003 Jep:lmc29, 22-34) –How related cue & target are (koriat, 1997) FoK –Cue familiarity (reder) –Accessibility of partial information –SEE VERY IMPORTANT FIGURE 8.5, WHICH INCLUDES ALL TYPES OF INPUTS

Inputs for JoL Aspects of study processes –Imagery vs. repetition –Ease of processing Item characteristics –Pair relatedness –Concrete or abstract words –Item frequency/familiarity Context of study –Number of study trials –Serial position of items –Luminance of items

Inputs for Confidence Judgment For recalled items –Latency of recall –Item characteristics –Pair relatedness For non-recalled items (also used in FoK) –Cue familiarity (reder) –Accessibility to partial information (koriat) For recognition task –Latency of recognition –Cue familiarity –Recollecting an episode in which the item has been learnt –Reasons given why answer may be wrong –Resons given why answer may be right

In sum, metamemory judgments are modestly related to performance (e.g, FoK relative accuracy. 35). However, this is not due to some misterious direct-access to trace Rather, it is due to use of inputs (cues) which have varying degrees of predictiveness about the target, from very low (e.g. deceptive items, false beliefs on importance of word frequency) to more positive (true relation between target and cue)

FoK is greatly impaired in Korsakoff (Shimamura

JoL & study time Intuitively, it makes sense that people will spend more time studying those items that they believed have not yet mastered (i.e., low JoL items) (see Son& Metcalfe 2000 for an exhaustive review) Discrepancy-Reduction Hypothesis: –There is a goal state and a current state, and people try to minimize the distance between current and goal states. –But people also use context. So if you are short for time, you will devote that time to easier items than to harder ones (Thiede & Dunlosky, 1999). –People also sometimes focus on items of intermediate difficulty (Metcalfe, 2002)

I think that in our study with Aaron what we need to do is to find an input that we can predict will be used for the JoL (e.g., ease of reading, see Hertzog 2003) or alternatively, look for a measure (delayed JoL) that we think will be affected by proactive interference. As it is now, why would PI affect immediate JoL?? Sounds silly