1. Bayesian topic models Tom Griffiths UC Berkeley

4. Latent structure Observed data probabilistic process

5. probabilistic process Latent structure (meaning) Observed data (words)

6. statistical inference Latent structure (meaning) Observed data (words)

7. Outline Topic models Latent Dirichlet allocation More complex models Conclusions

8. Outline Topic models Latent Dirichlet allocation More complex models Conclusions

9. Topic models Hierarchical Bayesian models –share parameters (topics) across densities Define a generative model for documents –each document is a mixture of topics –each word is chosen from a single topic An idea that is widely used in NLP –e.g., Bigi et al., 1997; Blei et al., 2003; Hofmann, 1999; Iyer & Ostendorf, 1996; Ueda & Saito, 2003 topics shared weights vary

10. A generative model for documents HEART0.2 LOVE0.2 SOUL0.2 TEARS0.2 JOY0.2 SCIENTIFIC 0.0 KNOWLEDGE 0.0 WORK 0.0 RESEARCH0.0 MATHEMATICS0.0 HEART0.0 LOVE0.0 SOUL0.0 TEARS0.0 JOY0.0 SCIENTIFIC 0.2 KNOWLEDGE 0.2 WORK 0.2 RESEARCH0.2 MATHEMATICS0.2 topic 1topic 2 w P(w|z = 1)w P(w|z = 2)

11. Choose mixture weights for each document, generate “bag of words” {P(z = 1), P(z = 2)} {0, 1} {0.25, 0.75} {0.5, 0.5} {0.75, 0.25} {1, 0} MATHEMATICS KNOWLEDGE RESEARCH WORK MATHEMATICS RESEARCH WORK SCIENTIFIC MATHEMATICS WORK SCIENTIFIC KNOWLEDGE MATHEMATICS SCIENTIFIC HEART LOVE TEARS KNOWLEDGE HEART MATHEMATICS HEART RESEARCH LOVE MATHEMATICS WORK TEARS SOUL KNOWLEDGE HEART WORK JOY SOUL TEARS MATHEMATICS TEARS LOVE LOVE LOVE SOUL TEARS LOVE JOY SOUL LOVE TEARS SOUL SOUL TEARS JOY

12. Geometric interpretation P(LOVE) P(RESEARCH) P(TEARS) P(RESEARCH) P(TEARS) 0 1 1 1 P(LOVE)

13. Geometric interpretation P(RESEARCH) 0 1 1 1 P(LOVE) P(TEARS) P(LOVE) P(TEARS)

14. Geometric interpretation P(RESEARCH) 0 1 1 1 TOPIC 1 TOPIC 2 P(LOVE) P(TEARS) P(LOVE) P(TEARS)

15. Geometric interpretation P(RESEARCH) 0 1 1 1 TOPIC 1 TOPIC 2 P(LOVE) P(TEARS) P(LOVE) P(TEARS)

16. P(w|z = 2) Geometric interpretation SWSW P(w|z = 3) P(w|z = 1) minimum KL projection P(w)P(w)try to minimize KL divergence (Hofmann, 1999)

17. words  documents words topics documents P(w|z)P(w|z) P(z)P(z) Matrix factorization interpretation Maximum-likelihood estimation is finding the factorization that minimizes KL divergence P(w)P(w) (Hofmann, 1999)

18. Three nice properties of topic models Interpretability of topics –useful in many applications

19. STORY STORIES TELL CHARACTER CHARACTERS AUTHOR READ TOLD SETTING TALES PLOT TELLING SHORT FICTION ACTION TRUE EVENTS TELLS TALE NOVEL MIND WORLD DREAM DREAMS THOUGHT IMAGINATION MOMENT THOUGHTS OWN REAL LIFE IMAGINE SENSE CONSCIOUSNESS STRANGE FEELING WHOLE BEING MIGHT HOPE WATER FISH SEA SWIM SWIMMING POOL LIKE SHELL SHARK TANK SHELLS SHARKS DIVING DOLPHINS SWAM LONG SEAL DIVE DOLPHIN UNDERWATER DISEASE BACTERIA DISEASES GERMS FEVER CAUSE CAUSED SPREAD VIRUSES INFECTION VIRUS MICROORGANISMS PERSON INFECTIOUS COMMON CAUSING SMALLPOX BODY INFECTIONS CERTAIN FIELD MAGNETIC MAGNET WIRE NEEDLE CURRENT COIL POLES IRON COMPASS LINES CORE ELECTRIC DIRECTION FORCE MAGNETS BE MAGNETISM POLE INDUCED SCIENCE STUDY SCIENTISTS SCIENTIFIC KNOWLEDGE WORK RESEARCH CHEMISTRY TECHNOLOGY MANY MATHEMATICS BIOLOGY FIELD PHYSICS LABORATORY STUDIES WORLD SCIENTIST STUDYING SCIENCES BALL GAME TEAM FOOTBALL BASEBALL PLAYERS PLAY FIELD PLAYER BASKETBALL COACH PLAYED PLAYING HIT TENNIS TEAMS GAMES SPORTS BAT TERRY JOB WORK JOBS CAREER EXPERIENCE EMPLOYMENT OPPORTUNITIES WORKING TRAINING SKILLS CAREERS POSITIONS FIND POSITION FIELD OCCUPATIONS REQUIRE OPPORTUNITY EARN ABLE Interpretable topics each column shows words from a single topic, ordered by P(w|z)

20. Three nice properties of topic models Interpretability of topics –useful in many applications Handling of multiple meanings or senses –better than spatial representations (e.g., LSI)

21. STORY STORIES TELL CHARACTER CHARACTERS AUTHOR READ TOLD SETTING TALES PLOT TELLING SHORT FICTION ACTION TRUE EVENTS TELLS TALE NOVEL MIND WORLD DREAM DREAMS THOUGHT IMAGINATION MOMENT THOUGHTS OWN REAL LIFE IMAGINE SENSE CONSCIOUSNESS STRANGE FEELING WHOLE BEING MIGHT HOPE WATER FISH SEA SWIM SWIMMING POOL LIKE SHELL SHARK TANK SHELLS SHARKS DIVING DOLPHINS SWAM LONG SEAL DIVE DOLPHIN UNDERWATER DISEASE BACTERIA DISEASES GERMS FEVER CAUSE CAUSED SPREAD VIRUSES INFECTION VIRUS MICROORGANISMS PERSON INFECTIOUS COMMON CAUSING SMALLPOX BODY INFECTIONS CERTAIN FIELD MAGNETIC MAGNET WIRE NEEDLE CURRENT COIL POLES IRON COMPASS LINES CORE ELECTRIC DIRECTION FORCE MAGNETS BE MAGNETISM POLE INDUCED SCIENCE STUDY SCIENTISTS SCIENTIFIC KNOWLEDGE WORK RESEARCH CHEMISTRY TECHNOLOGY MANY MATHEMATICS BIOLOGY FIELD PHYSICS LABORATORY STUDIES WORLD SCIENTIST STUDYING SCIENCES BALL GAME TEAM FOOTBALL BASEBALL PLAYERS PLAY FIELD PLAYER BASKETBALL COACH PLAYED PLAYING HIT TENNIS TEAMS GAMES SPORTS BAT TERRY JOB WORK JOBS CAREER EXPERIENCE EMPLOYMENT OPPORTUNITIES WORKING TRAINING SKILLS CAREERS POSITIONS FIND POSITION FIELD OCCUPATIONS REQUIRE OPPORTUNITY EARN ABLE Handling multiple senses each column shows words from a single topic, ordered by P(w|z)

22. Three nice properties of topic models Interpretability of topics –useful in many applications Handling of multiple meanings or senses –better than spatial representations (e.g., LSI) Well-formulated generative models –many options for inferring topics from documents –supports extensions of the basic model

23. Outline Topic models Latent Dirichlet allocation More complex models Conclusions

24. Latent Dirichlet allocation (Blei, Ng, & Jordan, 2001; 2003) NdNd D zizi wiwi  (d) (d)  (j) (j)    (d)  Dirichlet(  ) z i  Discrete(  (d) )  (j)  Dirichlet(  ) w i  Discrete(  (z i ) ) T distribution over topics for each document topic assignment for each word distribution over words for each topic word generated from assigned topic Dirichlet priors

25. Multivariate equivalent of Beta distribution Hyperparameters  determine form of the prior

26. Relationship to other models A fully generative aspect model (Hofmann, 1999) A multinomial version of PCA (Buntine, 2002) Poisson models: –non-negative matrix factorization (Lee & Seung, 1999) –GaP model (Canny, 2004) –discussed in detail in Buntine & Jakulin (2005) Statistics: grade of membership (GoM) models (Erosheva, 2002) Genetics: structure of populations (Pritchard, Stephens, & Donnelly, 2000)

27. Inverting the generative model Maximum likelihood estimation (EM) –e.g. Hofmann (1999) Deterministic approximate algorithms –variational EM; Blei, Ng & Jordan (2001; 2003) –expectation propagation; Minka & Lafferty (2002) Markov chain Monte Carlo –full Gibbs sampler; Pritchard et al. (2000) –collapsed Gibbs sampler; Griffiths & Steyvers (2004)

28. The collapsed Gibbs sampler Using conjugacy of Dirichlet and multinomial distributions, integrate out continuous parameters Defines a distribution on discrete ensembles z

29. The collapsed Gibbs sampler Sample each z i conditioned on z -i This is nicer than your average Gibbs sampler: –memory: counts can be cached in two sparse matrices –optimization: no special functions, simple arithmetic –the distributions on  and  are analytic given z and w, and can later be found for each sample

30. Gibbs sampling in LDA iteration 1

31. Gibbs sampling in LDA iteration 1 2

32. Gibbs sampling in LDA iteration 1 2

33. Gibbs sampling in LDA iteration 1 2

34. Gibbs sampling in LDA iteration 1 2

35. Gibbs sampling in LDA iteration 1 2

36. Gibbs sampling in LDA iteration 1 2

37. Gibbs sampling in LDA iteration 1 2

38. Gibbs sampling in LDA iteration 1 2 … 1000

39. pixel = word image = document sample each pixel from a mixture of topics A visual example: Bars

42. Effects of hyperparameters  and  control the relative sparsity of  and  –smaller , fewer topics per document –smaller , fewer words per topic Good assignments z compromise in sparsity log  (x) x

43. Analysis of PNAS abstracts Test topic models with a real database of scientific papers from PNAS All 28,154 abstracts from 1991-2001 All words occurring in at least five abstracts, not on “stop” list (20,551) Total of 3,026,970 tokens in corpus

44. FORCE SURFACE MOLECULES SOLUTION SURFACES MICROSCOPY WATER FORCES PARTICLES STRENGTH POLYMER IONIC ATOMIC AQUEOUS MOLECULAR PROPERTIES LIQUID SOLUTIONS BEADS MECHANICAL HIV VIRUS INFECTED IMMUNODEFICIENCY CD4 INFECTION HUMAN VIRAL TAT GP120 REPLICATION TYPE ENVELOPE AIDS REV BLOOD CCR5 INDIVIDUALS ENV PERIPHERAL MUSCLE CARDIAC HEART SKELETAL MYOCYTES VENTRICULAR MUSCLES SMOOTH HYPERTROPHY DYSTROPHIN HEARTS CONTRACTION FIBERS FUNCTION TISSUE RAT MYOCARDIAL ISOLATED MYOD FAILURE STRUCTURE ANGSTROM CRYSTAL RESIDUES STRUCTURES STRUCTURAL RESOLUTION HELIX THREE HELICES DETERMINED RAY CONFORMATION HELICAL HYDROPHOBIC SIDE DIMENSIONAL INTERACTIONS MOLECULE SURFACE NEURONS BRAIN CORTEX CORTICAL OLFACTORY NUCLEUS NEURONAL LAYER RAT NUCLEI CEREBELLUM CEREBELLAR LATERAL CEREBRAL LAYERS GRANULE LABELED HIPPOCAMPUS AREAS THALAMIC A selection of topics TUMOR CANCER TUMORS HUMAN CELLS BREAST MELANOMA GROWTH CARCINOMA PROSTATE NORMAL CELL METASTATIC MALIGNANT LUNG CANCERS MICE NUDE PRIMARY OVARIAN

45. Cold topicsHot topics

46. Cold topicsHot topics 2 SPECIES GLOBAL CLIMATE CO2 WATER ENVIRONMENTAL YEARS MARINE CARBON DIVERSITY OCEAN EXTINCTION TERRESTRIAL COMMUNITY ABUNDANCE 134 MICE DEFICIENT NORMAL GENE NULL MOUSE TYPE HOMOZYGOUS ROLE KNOCKOUT DEVELOPMENT GENERATED LACKING ANIMALS REDUCED 179 APOPTOSIS DEATH CELL INDUCED BCL CELLS APOPTOTIC CASPASE FAS SURVIVAL PROGRAMMED MEDIATED INDUCTION CERAMIDE EXPRESSION

47. Cold topicsHot topics 2 SPECIES GLOBAL CLIMATE CO2 WATER ENVIRONMENTAL YEARS MARINE CARBON DIVERSITY OCEAN EXTINCTION TERRESTRIAL COMMUNITY ABUNDANCE 134 MICE DEFICIENT NORMAL GENE NULL MOUSE TYPE HOMOZYGOUS ROLE KNOCKOUT DEVELOPMENT GENERATED LACKING ANIMALS REDUCED 179 APOPTOSIS DEATH CELL INDUCED BCL CELLS APOPTOTIC CASPASE FAS SURVIVAL PROGRAMMED MEDIATED INDUCTION CERAMIDE EXPRESSION 37 CDNA AMINO SEQUENCE ACID PROTEIN ISOLATED ENCODING CLONED ACIDS IDENTITY CLONE EXPRESSED ENCODES RAT HOMOLOGY 289 KDA PROTEIN PURIFIED MOLECULAR MASS CHROMATOGRAPHY POLYPEPTIDE GEL SDS BAND APPARENT LABELED IDENTIFIED FRACTION DETECTED 75 ANTIBODY ANTIBODIES MONOCLONAL ANTIGEN IGG MAB SPECIFIC EPITOPE HUMAN MABS RECOGNIZED SERA EPITOPES DIRECTED NEUTRALIZING

48. Outline Topic models Latent Dirichlet allocation More complex models Conclusions

49. Extensions to the basic model No need for a pre-segmented corpus –useful for modeling discussions, meetings

50. NuNu U zizi wiwi (u)(u)  (j) (j)    (u) |s u =0  Delta(  (u-1) )  (u) |s u =1  Dirichlet(  ) z i  Discrete(  (u) )  (j)  Dirichlet(  ) w i  Discrete(  (z i ) ) T A model for meetings susu  (u-1) …

51. Sample of ICSI meeting corpus (25 meetings) no it's o_k. it's it'll work. well i can do that. but then i have to end the presentation in the middle so i can go back to open up javabayes. o_k fine. here let's see if i can. alright. very nice. is that better. yeah. o_k. uh i'll also get rid of this click to add notes. o_k. perfect NEW TOPIC (not supplied to algorithm) so then the features we decided or we decided we were talked about. right. uh the the prosody the discourse verb choice. you know we had a list of things like to go and to visit and what not. the landmark-iness of uh. i knew you'd like that. nice coinage. thank you.

52. Topic segmentation applied to meetings Inferred Segmentation Inferred Topics

53. Comparison with human judgments Topics recovered are much more coherent than those found using random segmentation, no segmentation, or an HMM

54. Extensions to the basic model No need for a pre-segmented corpus Learning the number of topics

55. Can use standard Bayes factor methods to evaluate models of different dimensionality –e.g. importance sampling via MCMC Alternative: nonparametric Bayes –fixed number of topics per document, unbounded number of topics per corpus (Blei, Griffiths, Jordan, & Tenenbaum, 2004) –unbounded number of topics for both (the hierarchical Dirichlet process) (Teh, Jordan, Beal, & Blei, 2004)

56. Extensions to the basic model No need for a pre-segmented corpus Learning the number of topics Learning topic hierarchies

57. Fixed hierarchies: Hofmann & Puzicha (1998) Learning hierarchies: Blei et al. (2004) Topic 0 Topic 1.1 Topic 1.2 Topic 2.1 Topic 2.2 Topic 2.3

58. Learning topic hierarchies Topic 0 Topic 1.1 Topic 1.2 Topic 2.1 Topic 2.2 Topic 2.3 The topics in each document form a path from root to leaf Fixed hierarchies: Hofmann & Puzicha (1998) Learning hierarchies: Blei et al. (2004)

59. Twelve years of NIPS (Blei, Griffiths, Jordan, & Tenenbaum, 2004)

60. Extensions to the basic model No need for a pre-segmented corpus Learning the number of topics Learning topic hierarchies Modeling authors as well as documents

61. The Author-Topic model (Rosen-Zvi, Griffiths, Smyth, & Steyvers, 2004) NdNd D zizi wiwi  (a) (a)  (j) (j)    (a)  Dirichlet(  ) z i  Discrete(  (x i ) )  (j)  Dirichlet(  ) w i  Discrete(  (z i ) ) T xixi A x i  Uniform(A (d) ) each author has a distribution over topics the author of each word is chosen uniformly at random

62. Four example topics from NIPS

63. Who wrote what? A method 1 is described which like the kernel 1 trick 1 in support 1 vector 1 machines 1 SVMs 1 lets us generalize distance 1 based 2 algorithms to operate in feature 1 spaces usually nonlinearly related to the input 1 space This is done by identifying a class of kernels 1 which can be represented as norm 1 based 2 distances 1 in Hilbert spaces It turns 1 out that common kernel 1 algorithms such as SVMs 1 and kernel 1 PCA 1 are actually really distance 1 based 2 algorithms and can be run 2 with that class of kernels 1 too As well as providing 1 a useful new insight 1 into how these algorithms work the present 2 work can form the basis 1 for conceiving new algorithms This paper presents 2 a comprehensive approach for model 2 based 2 diagnosis 2 which includes proposals for characterizing and computing 2 preferred 2 diagnoses 2 assuming that the system 2 description 2 is augmented with a system 2 structure 2 a directed 2 graph 2 explicating the interconnections between system 2 components 2 Specifically we first introduce the notion of a consequence 2 which is a syntactically 2 unconstrained propositional 2 sentence 2 that characterizes all consistency 2 based 2 diagnoses 2 and show 2 that standard 2 characterizations of diagnoses 2 such as minimal conflicts 1 correspond to syntactic 2 variations 1 on a consequence 2 Second we propose a new syntactic 2 variation on the consequence 2 known as negation 2 normal form NNF and discuss its merits compared to standard variations Third we introduce a basic algorithm 2 for computing consequences in NNF given a structured system 2 description We show that if the system 2 structure 2 does not contain cycles 2 then there is always a linear size 2 consequence 2 in NNF which can be computed in linear time 2 For arbitrary 1 system 2 structures 2 we show a precise connection between the complexity 2 of computing 2 consequences and the topology of the underlying system 2 structure 2 Finally we present 2 an algorithm 2 that enumerates 2 the preferred 2 diagnoses 2 characterized by a consequence 2 The algorithm 2 is shown 1 to take linear time 2 in the size 2 of the consequence 2 if the preference criterion 1 satisfies some general conditions Written by (1) Scholkopf_B Written by (2) Darwiche_A

64. Extensions to the basic model No need for a pre-segmented corpus Learning the number of topics Learning topic hierarchies Modeling authors as well as documents Combining topics and syntax

65.  z w z z ww x x x semantics: probabilistic topics syntax: probabilistic regular grammar Factorization of language based on statistical dependency patterns: long-range, document specific, dependencies short-range dependencies constant across all documents (Griffiths, Steyvers, Blei, & Tenenbaum, 2005)

66. FOOD FOODS BODY NUTRIENTS DIET FAT SUGAR ENERGY MILK EATING FRUITS VEGETABLES WEIGHT FATS NEEDS CARBOHYDRATES VITAMINS CALORIES PROTEIN MINERALS MAP NORTH EARTH SOUTH POLE MAPS EQUATOR WEST LINES EAST AUSTRALIA GLOBE POLES HEMISPHERE LATITUDE PLACES LAND WORLD COMPASS CONTINENTS DOCTOR PATIENT HEALTH HOSPITAL MEDICAL CARE PATIENTS NURSE DOCTORS MEDICINE NURSING TREATMENT NURSES PHYSICIAN HOSPITALS DR SICK ASSISTANT EMERGENCY PRACTICE BOOK BOOKS READING INFORMATION LIBRARY REPORT PAGE TITLE SUBJECT PAGES GUIDE WORDS MATERIAL ARTICLE ARTICLES WORD FACTS AUTHOR REFERENCE NOTE GOLD IRON SILVER COPPER METAL METALS STEEL CLAY LEAD ADAM ORE ALUMINUM MINERAL MINE STONE MINERALS POT MINING MINERS TIN BEHAVIOR SELF INDIVIDUAL PERSONALITY RESPONSE SOCIAL EMOTIONAL LEARNING FEELINGS PSYCHOLOGISTS INDIVIDUALS PSYCHOLOGICAL EXPERIENCES ENVIRONMENT HUMAN RESPONSES BEHAVIORS ATTITUDES PSYCHOLOGY PERSON CELLS CELL ORGANISMS ALGAE BACTERIA MICROSCOPE MEMBRANE ORGANISM FOOD LIVING FUNGI MOLD MATERIALS NUCLEUS CELLED STRUCTURES MATERIAL STRUCTURE GREEN MOLDS Semantic topics PLANTS PLANT LEAVES SEEDS SOIL ROOTS FLOWERS WATER FOOD GREEN SEED STEMS FLOWER STEM LEAF ANIMALS ROOT POLLEN GROWING GROW

67. GOOD SMALL NEW IMPORTANT GREAT LITTLE LARGE * BIG LONG HIGH DIFFERENT SPECIAL OLD STRONG YOUNG COMMON WHITE SINGLE CERTAIN THE HIS THEIR YOUR HER ITS MY OUR THIS THESE A AN THAT NEW THOSE EACH MR ANY MRS ALL MORE SUCH LESS MUCH KNOWN JUST BETTER RATHER GREATER HIGHER LARGER LONGER FASTER EXACTLY SMALLER SOMETHING BIGGER FEWER LOWER ALMOST ON AT INTO FROM WITH THROUGH OVER AROUND AGAINST ACROSS UPON TOWARD UNDER ALONG NEAR BEHIND OFF ABOVE DOWN BEFORE SAID ASKED THOUGHT TOLD SAYS MEANS CALLED CRIED SHOWS ANSWERED TELLS REPLIED SHOUTED EXPLAINED LAUGHED MEANT WROTE SHOWED BELIEVED WHISPERED ONE SOME MANY TWO EACH ALL MOST ANY THREE THIS EVERY SEVERAL FOUR FIVE BOTH TEN SIX MUCH TWENTY EIGHT HE YOU THEY I SHE WE IT PEOPLE EVERYONE OTHERS SCIENTISTS SOMEONE WHO NOBODY ONE SOMETHING ANYONE EVERYBODY SOME THEN Syntactic classes BE MAKE GET HAVE GO TAKE DO FIND USE SEE HELP KEEP GIVE LOOK COME WORK MOVE LIVE EAT BECOME

68. Outline Topic models Latent Dirichlet allocation More complex models Conclusions

69. Topic models are a flexible class of models that can capture the content of documents Dirichlet priors can be used to assert a preference for sparsity in multinomials The collapsed Gibbs sampler makes it possible to exploit these priors, and is simple to use… …easy to extend to other aspects of language… …and applicable in a broad range of models

71. words documents  U D V  words dims vectors documents LSI words  documents words topics documents LDA P(w|z)P(w|z) P(z)P(z) P(w)P(w) (Dumais, Landauer)

74. NIPS: support vector topic

75. NIPS: neural network topic

76. MODEL ALGORITHM SYSTEM CASE PROBLEM NETWORK METHOD APPROACH PAPER PROCESS IS WAS HAS BECOMES DENOTES BEING REMAINS REPRESENTS EXISTS SEEMS SEE SHOW NOTE CONSIDER ASSUME PRESENT NEED PROPOSE DESCRIBE SUGGEST USED TRAINED OBTAINED DESCRIBED GIVEN FOUND PRESENTED DEFINED GENERATED SHOWN IN WITH FOR ON FROM AT USING INTO OVER WITHIN HOWEVER ALSO THEN THUS THEREFORE FIRST HERE NOW HENCE FINALLY #*IXTN-CFP#*IXTN-CFP EXPERTS EXPERT GATING HME ARCHITECTURE MIXTURE LEARNING MIXTURES FUNCTION GATE DATA GAUSSIAN MIXTURE LIKELIHOOD POSTERIOR PRIOR DISTRIBUTION EM BAYESIAN PARAMETERS STATE POLICY VALUE FUNCTION ACTION REINFORCEMENT LEARNING CLASSES OPTIMAL * MEMBRANE SYNAPTIC CELL * CURRENT DENDRITIC POTENTIAL NEURON CONDUCTANCE CHANNELS IMAGE IMAGES OBJECT OBJECTS FEATURE RECOGNITION VIEWS # PIXEL VISUAL KERNEL SUPPORT VECTOR SVM KERNELS # SPACE FUNCTION MACHINES SET NETWORK NEURAL NETWORKS OUPUT INPUT TRAINING INPUTS WEIGHTS # OUTPUTS NIPS Semantics NIPS Syntax (Griffiths, Steyvers, Blei, & Tenenbaum, 2005)

78. Varying   decreasing  increases sparsity

79. Varying  decreasing  increases sparsity ?

80. Number of words per topic Topic Number of words