Volume 26, Issue 7, Pages 928-934 (April 2016) Crossmodal Association of Visual and Haptic Material Properties of Objects in the Monkey Ventral Visual Cortex  Naokazu Goda, Isao Yokoi, Atsumichi Tachibana, Takafumi Minamimoto, Hidehiko Komatsu  Current Biology  Volume 26, Issue 7, Pages 928-934 (April 2016) DOI: 10.1016/j.cub.2016.02.003 Copyright © 2016 Elsevier Ltd Terms and Conditions

Figure 1 Experimental Design (A) Rod-shaped, real-object stimuli used for the monkey behavioral task. The surfaces of the objects (diameter 17.7–28.0 mm) were made from nine categories of materials (four exemplars per category). The inset shows an example scene in which a monkey is grasping a glass object. The monkeys first had to fixate on the top of the object and then to reach for, grasp, and pull the object with their hands. They were allowed to see the whole object after the first fixation until the end of the pulling. Each monkey performed the task with nine objects (one exemplar from each of the nine categories; typically 30 trials for each object) in one daily session, and a total of 32–44 sessions across 2 months. See the Supplemental Experimental Procedures. (B) Visual images used for the monkey fMRI experiments. Set A contains images of the objects used for the behavioral task (shown in A), and set B contains the images of four other exemplars from each of the nine categories. These images were presented during fMRI scans in a central 9.2° × 9.2° visual field within which the object subtended 2.2°–3.5° in width and 9.2° in height on a gray background (duration 500 ms, inter-stimulus interval >1,000 ms, twice per exemplar in each category block). See also Figure S1. Current Biology 2016 26, 928-934DOI: (10.1016/j.cub.2016.02.003) Copyright © 2016 Elsevier Ltd Terms and Conditions

Figure 2 Comparison between Neural Representations before and after Experience (A) Distributions of voxels used to analyze the neural representation of set A after experience (V1: green; V2: cyan; V3: blue; V4: purple; PIT: red). Each of the five ROIs contained the 500 most visually responsive voxels per hemisphere, determined using an independent dataset (i.e., a dataset excluding the data for set A after experience), within an areal boundary defined previously (see the Supplemental Experimental Procedures). Each color scale denotes the number of overlaps across four hemispheres (voxels in the left hemispheres are flipped). For clarity, only voxels overlapping across at least two hemispheres are shown. A, anterior; D, dorsal; IOS, inferior occipital sulcus; IPS, intraparietal sulcus; LuS, lunate sulcus; P, posterior; STS, superior temporal sulcus; V, ventral. (B) Representational dissimilarity matrix obtained from the V1 activity pattern (neural RDM) for set A after experience. The activity pattern was estimated for each material category from the fMRI data (40 scanning runs per image set, both before and after experience) using a voxel-wise general linear model analysis (see the Supplemental Experimental Procedures). The color scale indicates the dissimilarity between category pairs (Euclidean distance of the activity patterns [beta values]). Lighter colors indicate the pair is more dissimilar. See Figure 1A for abbreviations of materials. (C) Scatterplot showing the relationship between the neural RDMs from V1 for set A before and after experience. Each point represents dissimilarity between a category pair. The Spearman correlation coefficient (r) is indicated. Error bars indicate the SEM across four hemispheres. See also Figures S2A and S2B. (D) Correlation between the neural RDMs before and after experience for each of five ROIs (set A: yellow; set B: blue). ∗∗p < 0.01, ∗∗∗p < 0.001 (one-tailed permutation test, uncorrected). Current Biology 2016 26, 928-934DOI: (10.1016/j.cub.2016.02.003) Copyright © 2016 Elsevier Ltd Terms and Conditions

Figure 3 Representational Similarities between Neural Activities and the Material Properties of Objects (A) Scatterplot for 72 images (set A: filled symbols; set B: open symbols) in a two-dimensional space derived from MDS analysis (nonmetric, stress 0.096 for the two dimensions) of visual ratings of material properties by human participants. See Figure 1A for abbreviations of materials. (B) Representational dissimilarity matrix of the material properties (material RDM) obtained from humans’ visual rating data for set A. The color scale indicates the dissimilarity between category pairs (Euclidean distance of the rating scores averaged for samples within each category). (C) Scatterplots showing the relationship between the neural RDM obtained from the PIT and the material RDM (B) for set A (before experience: left; after experience: right). Each point represents dissimilarity between a category pair. The Spearman correlation coefficient is indicated. Error bars indicate the SEM across four hemispheres. See also Figures S2B and S2C. (D) Correlation between the neural RDMs obtained from the five ROIs and the material RDM (B) for set A (before experience: green; after experience: red). See also Figure S3. ∗p < 0.05, ∗∗∗p < 0.001 (one-tailed permutation test, uncorrected). Current Biology 2016 26, 928-934DOI: (10.1016/j.cub.2016.02.003) Copyright © 2016 Elsevier Ltd Terms and Conditions

Figure 4 Representation of Visual and Non-visual Material Properties in V4 and the PIT (A) Visual and non-visual material RDMs estimated from humans’ visual rating data for set A (left and middle), and the haptically estimated non-visual material RDM for set A (right). The format is the same as in Figure 3B. The haptically estimated RDM was derived from haptic rating experiments, in which 12 blindfolded human participants were asked to touch 36 objects and rate their impressions of the materials using nine bipolar adjective scales (excluding three visual adjective pairs; Table S1). (B) Correlation between the neural RDMs and each of the three material RDMs shown in (A) (visual: light blue bars; non-visual: green; haptically estimated: yellow) for set A. The correlation with the original material RDM evaluated using all adjective pairs (Figure 3B) is also shown for comparison (dark blue bars: re-plots of Figure 3D). See also Figure S4A. (C) The same analysis as in (B) for set B. The correlation with the haptically estimated material RDM is not shown because the haptic ratings were conducted only for the objects in set A. See also Figure S4B. ∗p < 0.05, ∗∗p < 0.01, ∗∗∗p < 0.001 (one-tailed permutation test, uncorrected). Current Biology 2016 26, 928-934DOI: (10.1016/j.cub.2016.02.003) Copyright © 2016 Elsevier Ltd Terms and Conditions