1. Measuring implicit and explicit knowledge by means of verbal reports and subjective measures of awareness Patrick Rebuschat (p.rebuschat@lancaster.ac.uk) IRIS Conference, York, September 2013

2. Outline How “implicit” is implicit knowledge? Verbal reports Retrospective Concurrent Subjective measures Example 1: Rebuschat & Williams (2012) Example 2: Rebuschat & Williams (in prep)

3. Definitions

4. What is implicit learning? Example: First language acquisition Learning process that results in unconscious (implicit) knowledge. Generally associated with incidental learning conditions. Learning process that results in unconscious (implicit) knowledge. Generally associated with incidental learning conditions.

5. What is explicit learning? Classic example of explicit knowledge: Pedagogical rules in language classroom Learning process that results in conscious (explicit) knowledge. Usually associated with intentional learning conditions. Learning process that results in conscious (explicit) knowledge. Usually associated with intentional learning conditions.

6. knowledge unconsciousconscious learning incidental implicit intentional explicit 6

7. How “implicit” is implicit knowledge?

8. How “implicit” is implicit L2 knowledge? Is the acquired knowledge really unconscious? Implicit-Explicit Interface Does explicit knowledge affect the acquisition of implicit knowledge? Implicit knowledge Implicit learning Explicit knowledge Explicit learning Input

9. “There is a broad consensus that the acquisition of an L2 entails the development of implicit knowledge. However, there is no consensus on how this is achieved; nor is there consensus on the role played by explicit knowledge.” (Rod Ellis, 2005, p. 143) Implicit knowledge Implicit learning Explicit knowledge Explicit learning Input

10. To decide on these questions, we need tests that determine whether knowledge is implicit and whether learning is implicit. There are many options... Focus in my talk: Two behavioral options. 1.Verbal reports 2.Subjective measures

11. Verbal reports

12. Prompt subjects to verbalize any rules, patterns, etc. they might have noticed while performing the experimental tasks. Classic procedure for measuring whether subjects have acquired implicit or explicit knowledge.  Reber (1967)

13. When is knowledge “implicit”? Knowledge is considered to be unconscious when subjects show an effect of training, despite being unable to describe the knowledge that underlies their performance. Example: Subjects perform above chance on grammaticality judgment task but they cannot describe the system that they acquired.

14. Two types of verbal reports: 1.Retrospective reports 2.Concurrent reports (think-aloud protocols)

15. Retrospective reports

16. There is little doubt that exposure can result in unconscious knowledge if lack of verbalization is used as a criterion for implicitness. Many studies have provided evidence for a dissociation between task performance and verbalizable knowledge.

17. Concurrent reports

18. 2. Concurrent verbal reports: Advocated by Ron Leow. Used to measure awareness at the time of encoding. Ask learners to perform on a task and think aloud at the same time.

19. If learners provide no evidence of having noticed a specific structure while performing on the task, they are classified as “unaware”. We can then compare the performance of “aware” und “unaware” subjects in our classification task (e.g. grammaticality judgments, etc.). If there is a learning effect in the “unaware” subjects, then this counts as evidence for implicit knowledge/learning.

20. Problems with verbal reports as measures of awareness?

21. Problem with using verbal reports to assess awareness Verbal reports (retrospective and concurrent) are not sensitive enough to serve as measures of awareness. Problems: Low confidence knowledge might not be reported Fabrication Unwillingness to report Lack of appropriate terminology Problem specific to concurrent reports: Issue of reactivity (see Rebuschat et al., 2013, in prep)

22. Subjective measures

23. Guessing Criterion (Cheesman & Merikle, 1984) Knowledge is unconscious if subjects believe to be guessing even though their performance on task (e.g. grammaticality judgments) is significantly above chance.

24. Zero Correlation Criterion (Chan, 1992; Dienes et al., 1995) Knowledge is unconscious if there is no correlation between confidence and accuracy. In unaware participants, confidence and accuracy are not correlated. In aware participants, confidence and accuracy are correlated.

25. How can we apply these two criteria? Collect confidence ratings and source attributions (Dienes & Scott, 2005). For each judgment, we can ask subjects to report: how confident they are in their decision (50-100%, etc.) What the basis of their judgment was (e.g. guess, intuition, memory, rule knowledge)

26. Example: Rebuschat & Williams (2012)

27. Artificial language study Training phase: Experimental subjects exposed under incidental learning conditions Testing phase: Measure of learning: Grammaticality judgment task Measure of awareness: 1.Subjective measures: Confidence ratings and source attributions 2.Verbal reports (retrospective) 27

28. Testing phase (experimental and control groups) Listen to sentence Judge grammaticality of sentence Report confidence level Report basis for grammaticality decision 28 60 new sentences 30 grammatical 30 ungrammatical 60 new sentences 30 grammatical 30 ungrammatical 0 = guess 5 = somewhat confident 9 = very confident 0 = guess 5 = somewhat confident 9 = very confident V2, V2-VF, VF-V1 *V1, * V3, * V4, * VF, * VF-V2, * V1-VF V2, V2-VF, VF-V1 *V1, * V3, * V4, * VF, * VF-V2, * V1-VF G= guess I = intuition M = memory R = rule knowledge G= guess I = intuition M = memory R = rule knowledge

29. Results: Performance on grammaticality judgment task suggest a very clear learning effect. No explicit knowledge according to retrospective verbal reports. This suggests: Exposure resulted in implicit knowledge! 29

30. But subjective measures of awareness indicate that the picture is much more interesting: 30

31. Confidence ratings 1.Zero Correlation Criterion: Subjects were more confident in correct decisions than in incorrect ones.  They were partially aware of having acquired knowledge. 2.Guessing Criterion: When subjects claimed to be guessing, their performance was at chance level.  No unconscious (judgment) knowledge. Experimental group: Accuracy and Proportions (in Percent) across Confidence Ratings GuessSomewhat confidentVery confident Accuracy5360*65* Proportion125434 Significance from chance: * p <.05. 31

32. Source attributions 1.When attributing grammaticality decisions to intuition and rule knowledge, subjects perform significantly above chance.  Evidence for both conscious and unconscious structural knowledge. 2.Native-speaker like intuition: Knowing that a sentence is grammatical or ungrammatical, with high levels of confidence, but with basis on intuition. 32 Experimental group: Accuracy and Proportions (in Percent) across Source Attributions GuessIntuitionMemoryRule Accuracy5659*5765** Proportion10321543 Significance from chance: * p <.05. ** p <.001.

33. Rebuschat and Williams (2012): Summary Adult learners are able to acquire L2 morphosyntax: – very rapidly (30 mins of exposure) – without intending to – without feedback – while processing sentences for meaning. Subjects were aware of having acquired knowledge, but they did not know what this knowledge was. Replicated by: Tagarelli, Borges Mota & Rebuschat (2011) Kim & Rebuschat (in prep) Rebuschat & Kissling (in prep) Replicated by: Tagarelli, Borges Mota & Rebuschat (2011) Kim & Rebuschat (in prep) Rebuschat & Kissling (in prep)

34. Example: Rebuschat & Williams (in prep)

35. What happens if we ask subjects to consciously try to figure out the word order rules? When we compare incidental learners to intentional learners, we find…

36. Results: Grammaticality judgments Instructing subjects to look for ‘rules’ promotes learning. Interestingly, this holds even subjects are not actually able to figure out the rules! Only 4 subjects figured out the system. * *  Improved performance under rule-search conditions. 36

37. Results: Subjective measures of awareness Intentional group: Accuracy and Proportions (in Percent) across Confidence Ratings GuessSomewhat confidentVery confident Accuracy80*74*81* Proportion65638 Significance from chance: * p <.05. 37 Intentional group: Accuracy and Proportions (in Percent) across Source Attributions GuessIntuitionMemoryRule Accuracy65.9 + 68.9**61.9*83.6** Proportion4.621.813.160.6 Significance from chance: + p <.06 * p <.05. ** p <.001. Evidence for both conscious and unconscious knowledge.

38. knowledge unconsciousconscious learning incidental implicit intentional explicit 38 Subjective measures suggest that both incidental and intentional learners develop implicit and explicit knowledge. But how much of each they develop depends on learning context!

39. Summary

40. Adult learners are able to acquire L2 morphosyntax without intending to and without becoming aware of the knowledge they have acquired. The relationship between learning condition and the acquired knowledge is complex. Subjects often acquire both implicit and explicit knowledge. Verbal reports would not have allowed us to make this observation! See also: Hamrick & Rebuschat (2012, 2013), Grey, Williams, & Rebuschat (in press), Tagarelli, Mota, & Rebuschat (2011), Rodgers, Rebuschat, & Revesz (in prep), Rebuschat & Kissling (in prep), Kim & Rebuschat (in prep), Dabrowska (in press) 40

41. Methodological observations: Reliance on verbal reports is clearly inadequate in order to distinguish implicit and explicit knowledge. Verbal reports lack sensitivity: Explicit knowledge might not be detected, which then leads to the erroneous assumption that learning in a given study resulted primarily unconscious knowledge. In the case of think-aloud procedure: This can have detrimental impact on learning (see Rebuschat et al., 2013, in prep). 41

42. If a fundamental objective of SLA research is to investigate the development of implicit L2 knowledge (and of how we can influence this by means of pedagogical intervention), then... we need to make sure that our tests are sophisticated enough to measure implicit and explicit knowledge (Ellis, 2005). Subjective measures of awareness useful tool. But there are many other good options! 42

43. Thank you! Elizabeth Kissling Georgetown Linguistics Sarah Grey Georgetown Spanish and Brain and Language Luke Amoroso Katie Kim Georgetown PsyLab John Williams Cambridge Phillip Hamrick Julie Lake Kaitlyn Tagarelli

44. Measuring implicit and explicit knowledge by means of subjective measures of awareness Patrick Rebuschat (p.rebuschat@lancaster.ac.uk) IRIS Conference, York, September 2013

45. Today BOUGHT John the apple in the supermarket. Some time ago CLAIMED Jim that his mother Melbourne LIKED. After his wife a sandwich CRAVED, RUSHED Mike to Japan. [V2] [V2-VF] [VF-V1] Three syntactic patterns English words, German syntax

46. Exp Performance across grammaticality Endorsement rates 46 Experimental group is significantly more likely than control group to endorse grammatical items.

47. Exp Performance across grammaticality Endorsement rates 47 No significant difference between the two groups on ungrammatical items.

48. Exp Performance across grammaticality Endorsement rates 48 Overall performance driven by correct endorsement of previously encountered patterns.

49. What do subjects learn? Subjects were able to classify strings they had never encountered before. Classification performance: Above-chance performance only on grammatical patterns. 49  Subjects acquired abstract linguistic knowledge as a result of exposure.  Exposure results in memory for syntactic patterns, not rules in the traditional sense.

50. Adult learners are able to acquire L2 syntax without intending to and without becoming aware of the knowledge they have acquired. The acquired knowledge is abstract: Subjects are able to rate completely novel sequences, and changing modality from exposure to test does not affect this ability. Subjects do not acquire ‘rules’ in the traditional sense, but rather syntactic patterns. 50

51. The relationship between learning condition and the acquired knowledge is complex. Subjects often acquire both implicit and explicit knowledge. Working memory capacity plays a role when subjects are exposed under explicit learning conditions, but not under implicit learning conditions. Adult L2 acquisition as an excellent example of implicit/explicit learning “out of the lab”! 51

52. Summary: How to distinguish implicit and explicit knowledge The different measures are not equally sensitive. Use at least one, ideally more than one. Examples: 1.Prompt participants to verbalize the knowledge they have acquired. 2.Compare relationship between confidence and accuracy (Zero Correlation Criterion).

53. 3.Observe accuracy in those judgments where subjects believe to be guessing (Guessing Criterion). 4.Use online measures such as RT in sentence reading task, ERP, gaze.

54. Key Questions in Implicit and Explicit L2 Learning Research

55. How “implicit” is implicit L2 learning / knowledge? Is there learning without “noticing”? Is the acquired knowledge really unconscious? How is implicit L2 knowledge represented? Do we acquire rules, constructions, chunks? Implicit knowledge Implicit learning Explicit knowledge Explicit learning Input

56. What aspects of the L2 can be acquired implicitly? Sounds, words, grammar, meaning? What aspects of grammar? How does prior knowledge affect the acquisition of implicit L2 knowledge? What is the role of the L1? What is the role of (learned) attention? Implicit knowledge Implicit learning Explicit knowledge Explicit learning Input

57. Individual differences in implicit /explicit learning Do individual differences affect implicit and explicit learning differently? Implicit-Explicit Interface Does explicit knowledge affect the acquisition of implicit knowledge? Implicit knowledge Implicit learning Explicit knowledge Explicit learning Input

58. “There is a broad consensus that the acquisition of an L2 entails the development of implicit knowledge. However, there is no consensus on how this is achieved; nor is there consensus on the role played by explicit knowledge.” (Rod Ellis, 2005, p. 143) Implicit knowledge Implicit learning Explicit knowledge Explicit learning Input

59. Debate in psychology: Is the knowledge acquired in AGL expts is really implicit (unconscious). Debate in SLA research: Can learning be really implicit, i.e. is there learning without awareness (= focal attention)? Implicit knowledge Implicit learning Explicit knowledge Explicit learning Input

60. Stimuli from Altmann, Dienes, & Goode (1995)

61. Squire L, Zola S PNAS 1996;93:13515-13522 Example: Sequence learning

62. 62 NewPartially- trained Trained TA Rebuschat et al. (in prep) Think-aloud group failed to generalize to novel structures. Learning effect driven by memory for trained patterns.

63. 63 NewPartially- trained Trained No TA TA Rebuschat et al. (in prep) Think-aloud group failed to generalize to novel structures. Learning effect driven by memory for trained patterns.