Large-Scale Organization of Semantic Networks Mark Steyvers Josh Tenenbaum Stanford University

Graph theoretic analyses: Collaboration network of film actors, scientists Watts & Strogatz (1998); Newman (2001) Neural network of worm: C. elegans Watts & Strogatz (1998) WWW Barabasi & Albert (1999)

Link structure of semantic networks: Small-world & scale free What produces such link structures? Semantic growth Relation to age-of-acquisition effects Behavioral effects of link structure Overview

Word Association Nelson et al. (1999) n words = 5,000+

Roget’s Thesaurus Categories 1,000 Word forms 29,000+

Wordnet George Miller Word forms 122,000+ Word senses 99,000+

1. Short path lengthsL = average length of shortest path between two nodes One class of Small World Networks: Word Association Roget WordNet 3 x number of triangles number of connected triples of vertices C=0C=1 2. Local clustering C = Power-law  = exponent in power-law  degree distribution distribution Random Graphs n = number of nodes , ,000+

Power law tail Exponential tail e.g., random graphs (Erdös-Réyni) or Watts & Strogatz (1998) model Exponential: Power law: HUBS

 =3.01  =3.19  =3.11

Zipf’s (1949) “Law of Meaning” Word frequency rank #meanings Slope in rank plot a=.466 Slope in distribution plot  = 3.15 Adamic (2000):  =1+1/a

Link structure of semantic networks: Small-world & scale free What produces such link structures? Semantic growth Relation to age-of-acquisition effects Behavioral effects of link structure Overview

H.A. Simon (1955). Power laws in distributions: –Scientists by number of papers published –Cities by population –Income by size -> “rich get richer” growth-like stochastic process Barabasi et al. (1999). Power laws in WWW –in-degree & out-degree -> growth processes

Proposal: Power-law degree distributions in semantic networks are signature of semantic growth within individual; lexical development across speakers; language evolution Disclaimer: We will not describe in detail any specific psychological mechanism

Growing Network Model Representation: Nodes represent words or concepts Edges represent semantic relations or associations Variables: k i = degree of node i u i = utility of node i based on word frequency:

Start with small fully connected network with M nodes A new node is inserted: 1)Choose a local neighborhood i (a neighborhood i of a node is formed by node i and its neighbors) 2)Make M connections into neighborhood repeat n times until network is large enough

new node Preferentially choose large neighborhoods: 1 new node Preferentially make M connections to nodes with high utility:

Word Association n Path LengthL (.04) ClusteringC (.007) coefficient Power-Law  coefficient Growing Network Model Barabasi & Albert (1999) Model

Power-laws in non-growing semantic representations?

LSA: Latent Semantic Analysis e.g., Landauer & Dumais (1997) Analyzed co-occurrence statistics in a large corpus Placed 60,000+ words in 300-dimensional space Good semantic neighbors volcano Hawaii ache relax soothe lava Convert LSA space to graph by variable thresholding on similarity measure

Tversky & Hutchinson (1986) Low dimensional geometric models are not suitable for representing conceptual similarity relations; upper bound on the number of points that can share the nearest neighbor

Ferrer & Solé (submitted): Connect two words if they co-occur within a small contextual window Slide window over large corpus No good semantic neighborhoods volcano -> was -> head -> ache (word association: volcano->hawaii->relax->soothe->ache) or tick -> tock -> made -> wonderful -> universe (word association: tick -> dog -> master -> universe)

Link structure of semantic networks: Small-world & scale free What produces such link structures? Semantic growth Relation to age-of-acquisition effects Behavioral effects of link structure Overview

Age of acquisition (AoA) effects Naming and lexical decision tasks Carroll & White (1973); Brysbaert et al. (2000) Locus of AoA effects? Brown & Watson (1987); Lambon Ralph et al. (1998) AoA is really cumulative frequency effect? Lewis, Gerhand & Ellis (1999) Need framework to understand AoA effects.

t=1…15 t=16…50 t= Prediction of model: early acquired nodes have more connections. Do words acquired early in life have more connections?

Language Evolution Words acquired early in English language are words with high degree (work in progress)

Link structure of semantic networks: Small-world & scale free What produces such link structures? Semantic growth Relation to age-of-acquisition effects Behavioral effects of link structure Overview

Behavioral effects of structural variables centrality Degree-centrality Authority (Eigenvector-centrality) Proposal: In cognitive system, search is biased toward facts, concepts or words with high centrality Naming and lexical decision latencies

Semantic Dementia Hodges & Patterson (1995)

Conclusion Link structure of semantic networks: a)shows non-trivial patterns b)shows signature of growth processes “rich get richer” respecting local neighborhoods c)is relevant for search strategies central words might be searched first. Paper will be available at www-psych.stanford.edu/~msteyver

But… Early acquired words become more central in your model but maybe Words that are more central are acquired earlier

Earliest year of quotation (in OED) vs. k (connectivity)