1. Neural Correlates of Degraded Picture Perception Tom Busey, Rob Goldstone and Bethany Knapp

5. General Method Record brain activity during perception of degraded pictures. Change knowledge by sometimes showing undegraded picture. Always record during presentation of degraded picture. Also change experience by showing primes.

6. (4 repetitions)

7. What was the picture? 1 Raccoon 2 Baseball player 3 Chairs 4 Two women talking 5 Bird facing right 6 Trees 7 Monkey 8 Don’t know (no clue)

8. (flip between degraded and undegraded images)

10. (4 repetitions)

11. Experimental Design 4 Repetitions degraded pict Questions Flipping but blank screen instead of undegraded picture Flipping with undegraded picture 4 Repetitions degraded pict Pre-exposurePost-exposure Process repeated 60 times 30 Pictures in each condition. Each condition is replicated 120 times per subject x 3 subjects. One Trial

12. (4 repetitions)

13. What was the picture? 1 Dog 2 Woman with hands 3 Camel facing left 4 Horse 5 Raccoon 6 Bird facing right 7 Baseball player 8 Don’t know (no clue)

14. (flip between degraded image and gray screen)

16. (4 repetitions)

17. Experimental Design 4 Repetitions degraded pict Questions Flipping but blank screen instead of undegraded picture Flipping with undegraded picture 4 Repetitions degraded pict Pre-exposurePost-exposure Process repeated 60 times 30 Pictures in each condition. Each condition is replicated 120 times per subject x 3 subjects. One Trial

18. Front of Head ERP Data From Experiment 1 Back of Head Left SideRight Side

19. Pz (Back Middle of Head)Cz (Middle of Head) ERP Data From Experiment 1 Back of Head Left SideRight Side Front of Head

20. ERP Data From Experiment 1 Knowledge No Knowledge Prior to Flipping BlueGreen After Flipping RedCyan Pz Cz

21. ERP Data From Experiment 1 Knowledge No Knowledge Prior to Flipping BlueGreen After Flipping RedCyan Pz Cz

23. ICA Decomposition Goal: Recover a set of independent signals (components) that were mixed together in the EEG electrodes. Recovered components can be considered as latent variables or factors like those in Factor Analysis. Not the same as dipole analysis, but dipoles can be fit to ICA components.

25. ICA Decomposition Independence: –Knowing something about one component tells you nothing about the state of the other components. –Joint density equals the product of the marginal densities: p(y 1,y 2 )= p(y 1 )p(y 2 ) –Independent components are uncorrelated, but uncorrelated factors (from PCA) need not be independent. Decomposition is an interative process –Knows nothing about time or conditions –Adjusts weight values assigned to each electrode to find components with distributions that are as independent as possible. –Decomposition is not unique, but we see good convergence over repeated simulations.

26. Component Maps from Experiment 1

27. ICA Decomposition Visualization –Compute components (which are defined by their weights) from concatenated individual subject data. –Visualize using grand average data. –Look for components that differentiate between the conditions –Back project each component to voltage, which simulates what we would have recorded if this was the only active component. Statistical Issues –Statistical analysis of ICA components is still relatively new. –Stress replication across experiments over hypothesis testing.

28. ICA Component From Experiment 1 Knowledge No Knowledge Prior to Flipping BlueGreen After Flipping RedCyan Pz Cz

29. ICA Component From Experiment 1 Knowledge No Knowledge Prior to Flipping BlueGreen After Flipping RedCyan O2O2 O1O1

30. ICA Component From Experiment 1 Knowledge No Knowledge Prior to Flipping BlueGreen After Flipping RedCyan O2O2 O1O1

31. ICA Component From Experiment 1 Knowledge No Knowledge Prior to Flipping BlueGreen After Flipping RedCyan Pz Cz

32. ICA Component From Experiment 1 Knowledge No Knowledge Prior to Flipping BlueGreen After Flipping RedCyan Pz Cz

33. Experiment 1 Conclusions Experience with the real image produces large centrally-located changes in the ERP beginning around 400 ms. Also see an ICA component that separates out this condition in perceptual regions as early as 250 ms. Knowledge of the picture's gist but not its interpretation produces an ICA component that separates at 400 ms and is localized to the occipital portion of the head.

34. Experiment 2 How does prior knowledge about the content of the picture help you interpret the degraded image? Precede the degraded image with a text description of the contents, called a prime. How are the components identified by ICA affected by the prime?

35. Experiment 2 Design Changes Pre-expose half of the pictures at the start of the experiment, along with their primes and degraded versions.

36. Prior to experiment bird facing right

37. Prior to experiment

38. Experiment 2 Design Changes Pre-expose half of the pictures at the start of the experiment, along with their primes and degraded versions. The other half of the pictures are shown in their degraded form only.

39. Prior to experiment baseball player

40. Prior to experiment baseball player

41. Experimental Design Incorrect Description (comes from other pictures) DelayDegraded Picture 30 Pictures in each condition Each condition is replicated 120 times per subject x 4 subjects. Prime One trial Correct Description 1000 ms

42. (One Trial) baseball player

45. Experimental Design-End Allow flipping with description (prime) Ask subject: How well did you figure this picture out?

46. End of experiment baseball player

47. End of experiment baseball player

48. How well did you interpret this picture? No Clue: I never figured it out. Partial: I got some of the details, or I figured it out midway through the experiment. Knew: I figured this picture out almost immediately (or it was shown to me at the beginning of the experiment). Exclude data from pictures in the No-Knowledge condition that subjects figure out.

49. Experimental Design Prior Knowledge? KnowledgeNo-KnowledgePrime veracity Correct Incorrect Blue RedCyan Green Incorrect primes come from other Knowledge or No-Knowledge pictures. No information in the prime as to whether a Knowledge or No- Knowledge picture would appear. But, some No-Knowledge pictures will have primes associated with known pictures.

50. Bias Model Prime makes it more likely that picture will be interpreted in a manner consistent with the prime. –Bias or preference effect. Verbal/semantic in nature. –Doesn't involve recall of image from memory. –Look at processing of prime to see if get differences that may reflect recall of memory.

51. Picture Retrieval Model Prime causes a retrieval of undegraded picture from memory. Association between word and interpreted picture important. –Affects only pictures that had undegraded versions presented, or those that the observer figured out. –Predicts no effect of prime veracity on no-clue pictures (unless observer can recall and use uninterpreted blobs from degraded picture).

52. Perceptual Facilitation Prime affects the perceptual processing of the picture –Helps bind feature elements, separate figure from ground. e.g. outdoor scenes might be interpreted differently –Evidence from Yu and Blake (1992). Degraded images from real scenes dominated binocular rivalry even though scene could not be interpreted.

53. Which image comes from a real Dalmatian picture?

54. ERP Data From Experiment 2- Eight Subjects Knowledge Correct Prime Blue Incorrect Prime Red Pz Cz

55. ERP Data From Experiment 2- Eight Subjects Knowledge No Knowledge Correct Prime BlueGreen Incorrect Prime RedCyan Pz Cz

56. ICA Component From Experiment 2 Knowledge No Knowledge Correct Prime BlueGreen Incorrect Prime RedCyan Pz Cz

57. ICA Component From Experiment 2 Knowledge No Knowledge Correct Prime BlueGreen Incorrect Prime RedCyan O2O2 O1O1

58. Experiment 2 Conclusions Preceeding a known picture with a correct prime produces differences in the onset and peak latency of a centrally-located source. –Differences begin at about 350-400 ms. –Is the correctly primed picture advanced, or the incorrectly primed picture delayed? A single ICA component captures both correctly and incorrectly primed known images. Suggests that the neural processes are similar, but offset in time.

59. Experiment 2 Conclusions ICA decomposition also reveals a separation of the correctly-primed known condition from the other conditions that is located near the perceptual regions. –Differences begin at about 300 ms. –Consistent with a facilitation model that involves perceptual regions of the brain. Unknown pictures show an effect of the prime!

60. Experiment 3 Design Changes Included a neutral prime condition Primes for unknown pictures come only from pictures they never see –Can't form mental picture from these primes, because never see the picture. Forced-choice test at the end of the experiment with only the degraded versions shown. –Again exclude pictures that subjects figure out by themselves. 17 Subjects (twice as many as Experiment 2)

61. Experimental Design Incorrect Description (comes from never-seen pictures) DelayDegraded Picture 30 Pictures in each condition Each condition is replicated 120 times per subject x 4 subjects. Prime One trial Correct Description 1000 ms *******

62. Experimental Design Prior Knowledge? KnowledgeNo-KnowledgePrime Type Correct Neutral Blue GreenMagenta Cyan Incorrect primes come from pictures they never see. Netural prime was '********' A given picture always had same incorrect prime for the 4 trials in which it was incorrectly primed. IncorrectRedBlack

63. ERP Data From Experiment 3 PrimeKnowledge CorrectBlue NeutralGreen IncorrectRed Pz Cz

64. ERP Data From Experiment 3 PrimeKnowledgeNo Knowledge CorrectBlueCyan NeutralGreenMagenta IncorrectRedBlack Pz Cz

65. ICA Component From Experiment 3 PrimeKnowledgeNo Knowledge CorrectBlueCyan NeutralGreenMagenta IncorrectRedBlack Pz Cz

66. ICA Component From Experiment 3 PrimeKnowledgeNo Knowledge CorrectBlueCyan NeutralGreenMagenta IncorrectRedBlack Pz Cz

67. ICA Component From Experiment 3 PrimeKnowledgeNo Knowledge CorrectBlueCyan NeutralGreenMagenta IncorrectRedBlack O2O2 O1O1

68. ICA Component From Experiment 3 PrimeKnowledgeNo Knowledge CorrectBlueCyan NeutralGreenMagenta IncorrectRedBlack O2O2 O1O1

69. ICA Component From Experiment 3 Knowledge No Knowledge Correct Prime BlueGreen Incorrect Prime RedCyan O2O2 O1O1

70. ICA Component From Experiment 2 using Experiment 3 weights Knowledge No Knowledge Correct Prime BlueGreen Incorrect Prime RedCyan O2O2 O1O1

71. ICA Component From Experiment 2 using Experiments 2 and 3 weights Knowledge No Knowledge Correct Prime BlueGreen Incorrect Prime RedCyan O2O2 O1O1

72. Experiment 3 Conclusions Correctly-primed pictures show a component with an earlier onset and peak latency than either the incorrect or netural primed conditions. –One possibilitiy: A correct prime may speed up processing of the degraded picture. Again we see a component in the ICA decomposition that has the correctly-primed picture processing separating out at about 250-300 ms in channel locations associated with perceptual regions. –Involvement of perceptual regions in facilitation.

73. Experiment 3 Conclusions Unknown pictures showed no effect of the prime veracity. No ICA component separated the correctly-primed unknown pictures from the neutral and incorrectly-primed unknown pictures. –Experiment 2 difference between correctly and incorrectly primed unknown pictures is likely due to a mismatch between the picture engendered by the prime and the uninterpretable degraded image. –No real evidence for perceptual facilitation of uninterpreted pictures.

74. General Conclusions Knowledge of the pictures intepretation produces large changes in the ERP in central recording sites. –Consistently revealed by ICA. Some evidence for changes in perceptual regions as well. A correct prime appears to speed up interpreation of the picture. May also produce changes in the perceptual regions of the brain.

75. How Does ICA Help? Allows for comparison between conditions while removing irrelevant activity (e.g. neutral primes). Demonstrates that two sets of activity have the same distributions on the scalp but differ in time (e.g. known pictures and 3 types of primes). Highlights perceptual activity. Allows comparisons across experiments even if they don't match in designs. The potential for overinterpretation is vast, and replication is important.

76. EEG Data From Experiment 3 PrimeKnowledgeNo Knowledge CorrectBlueCyan NeutralGreenMagenta IncorrectRedBlack

77. EEG Data From Experiment 3 PrimeKnowledgeNo Knowledge CorrectBlueCyan NeutralGreenMagenta IncorrectRedBlack

78. EEG Data From Experiment 2 Knowledge No Knowledge Correct Prime BlueGreen Incorrect Prime RedCyan