Research Introspection “ICML does ICML” Andrew McCallum Computer Science Department University of Massachusetts Amherst

Relational Modeling of the Research Literature & other Entities Better understand structure of our own research area. Tools to help us learn a new sub-field. Aid collaboration Map how ideas travel through social networks of researchers. Aids for hiring and finding reviewers! Many opportunities for rich relational learning... in a domain we understand well.

Previous Systems

Research Paper Cites Previous Systems

Research Paper Cites Person UniversityVenue Grant Groups Expertise More Entities and Relations

Our Status So Far Over 1.6 million research papers, gathered as part of Rexa.info portal. Cross linked papers / people / grants / topics.

Rexa System Overview Reference resolution (of papers, authors & grants) Spider Web for PDFs Convert to text (with layout & format) Extract metadata (title, authors, abstract, venue, citations; 14 fields in total) Browsable Web Interface Topic Analysis & other Data Mining WWW Home-grown Java+MySQL (~1m PDF/day) Enhanced ps2text (better word stiching, plus layout in XML) Conditional Random Fields (99% word accuracy) NSF grant DB Discriminatively trained graph partitioning (competition-winning accuracy)

From Text to Actionable Knowledge Segment Classify Associate Cluster Filter Prediction Outlier detection Decision support IE Document collection Database Discover patterns - entity types - links / relations - events Data Mining Spider Actionable knowledge

Segment Classify Associate Cluster Filter Prediction Outlier detection Decision support IE Document collection Database Discover patterns - entity types - links / relations - events Data Mining Spider Actionable knowledge Uncertainty Info Emerging Patterns Joint Inference

Segment Classify Associate Cluster Filter Prediction Outlier detection Decision support IE Document collection Probabilistic Model Discover patterns - entity types - links / relations - events Data Mining Spider Actionable knowledge Conditional Random Fields [Lafferty, McCallum, Pereira] Conditional PRMs [Koller…], [Jensen…], [Geetor…], [Domingos…] Discriminatively-trained undirected graphical models Complex Inference and Learning Just what we researchers like to sink our teeth into! Unified Model

Information Extraction Markov dependencies...and long-range & KB dependencies?

IE from Research Papers [McCallum et al kaelbling96reinforcement, author = "Leslie Pack Kaelbling and Michael L. Littman and Andrew P. Moore", title = "Reinforcement Learning: A Survey", journal = "Journal of Artificial Intelligence Research", volume = "4", pages = " ", year = "1996",

(Linear Chain) Conditional Random Fields y t-1 y t x t y t+1 x t +1 x t - 1 Finite state modelGraphical model Undirected graphical model, trained to maximize conditional probability of output sequence given input sequence... FSM states observations y t+2 x t +2 y t+3 x t +3 said Jones a Microsoft VP … OTHER PERSON OTHER ORG TITLE … output seq input seq Asian word segmentation [COLING’04], [ACL’04] IE from Research papers [HTL’04] Object classification in images [CVPR ‘04] Wide-spread interest, positive experimental results in many applications. Noun phrase, Named entity [HLT’03], [CoNLL’03] Protein structure prediction [ICML’04] IE from Bioinformatics text [Bioinformatics ‘04],… [Lafferty, McCallum, Pereira 2001] where

Entity Resolution Joint inference among all pairwise coref...models of entities, attributes, first-order...

Y/N Joint Co-reference Decisions, Discriminative Model Stuart Russell [Culotta & McCallum 2005] S. Russel People

Y/N Co-reference for Multiple Entity Types Stuart Russell University of California at Berkeley [Culotta & McCallum 2005] S. Russel Berkeley PeopleOrganizations

Y/N Joint Co-reference of Multiple Entity Types Stuart Russell University of California at Berkeley [Culotta & McCallum 2005] S. Russel Berkeley PeopleOrganizations Reduces error by 22%

Dean Martin Howard Dean Howard Martin SamePerson(Howard Dean, Howard Martin, Dean Martin)? First-Order Features  x 1,x 2 StringMatch(x 1,x 2 )  x 1,x 2 ¬StringMatch(x 1,x 2 )  x 1,x 2 EditDistance>.5(x 1,x 2 ) ThreeDistinctStrings(x 1,x 2, x 3 ) Toward High-Order Representations Identity Uncertainty

Structured Topic Models Discovering latent structure in jointly modeling words, time, relations...

Topical N-gram Model z1z1 z2z2 z3z3 z4z4 w1w1 w2w2 w3w3 w4w4 y1y1 y2y2 y3y3 y4y4  11 T D...  W T W  11 22  22 [Wang, McCallum 2005]

Finding Topics with TNG Traditional unigram LDA run on 1.6 million titles / abstracts (200 topics)...select ~300k papers on ML, NLP, robotics, vision... Find 200 TNG topics among those papers.

Topical Transfer Citation counts from one topic to another. Map “producers and consumers”

Trends in 17 years of NIPS proceedings

Topic Distributions Conditioned on Time time topic mass (in vertical height)

Topical Transfer Through Time Can we predict which research topics will be “hot” at ICML next year?...based on –the hot topics in “neighboring” venues last year –learned “neighborhood” distances for venue pairs

How do Ideas Progress Through Social Networks? COLT “ADA Boost” ICML ACL (NLP) ICCV (Vision) SIGIR (Info. Retrieval) Hypothetical Example:

How do Ideas Progress Through Social Networks? COLT “ADA Boost” ICML ACL (NLP) ICCV (Vision) SIGIR (Info. Retrieval) Hypothetical Example:

How do Ideas Progress Through Social Networks? COLT “ADA Boost” ICML ACL (NLP) ICCV (Vision) SIGIR (Info. Retrieval) Hypothetical Example:

How do Conferences Influence Each Other? Run an LDA on research papers. For each year, create an agglomerated topic distribution for a particular conference Model the topic distribution of a conference by the topic distributions of related conferences

Topic Prediction Models Static Model Transfer Model Linear Regression and Ridge Regression Used for Coefficient Training.

Preliminary Results Mean Squared Prediction Error # Venues used for prediction Transfer Model with Ridge Regression is a good Predictor (Smaller Is better) Transfer Model

Estimated Neighborhood Distances ML.079 Neural Computation.023 UAI PAMI.0998 Theoretical CS.0955 AI.032 AAAI.082 Transfer into NIPS,

Other Relational Opportunities Categorizing citations. Map transfer of ideas through science. Rank CS departments by various criteria. What 10 papers tell the story of ASR research? Predicting when a student will graduate. Help me find the right postdoc. Suggest best collaborative opportunities. Who should chair the next ICML?