1. Bayesian Generative Modeling
Jason Eisner
Summer School on Machine Learning
Lisbon, Portugal – July 2011 1

2. Bayesian Generative Modeling
what’s a model?
Jason Eisner
Summer School on Machine Learning
Lisbon, Portugal – July 2011 2 2

3. Bayesian Generative Modeling
what’s a generative model?
Jason Eisner
Summer School on Machine Learning
Lisbon, Portugal – July 2011 3 3

4. Bayesian Generative Modeling
what’s Bayesian?
Jason Eisner
Summer School on Machine Learning
Lisbon, Portugal – July 2011 4 4

5. Task-centric view of the worldx y
evaluation (loss function)
e.g., p(y|x) model and decoder

6. Task-centric view of the worldx y
loss function
p(y|x) model
Great way to track progress & compare systems
But may fracture us into subcommunities (our systems are incomparable & my semantics != your semantics)
Room for all of AI when solving any NLP task
Spelling correction could get some benefit from deep semantics, unsupervised grammar induction, active learning, discourse, etc.
But in practice, focus on raising a single performance number
Within strict, fixed assumptions about the type of available data
Do we want to build models & algs that are good for just one task?
Spelling correction: Humans do all kinds of reasoning about what was really meant in this context; they can even whether the misspelling was deliberate (pun, persona, implicit quotation, other metalinguistic behavior). But as an engineering matter, this doesn’t help much?
Do we want to build models & algs that are good for just one task? There are some tasks that are genuinely worth solving as formulated, at least for now. Others are set up as academic exercises to motivate research, and there they may motivate peculiar research directions (e.g., RTE using n-grams). Either way, when the task parameters change, a lot of effort falls on the floor. Of course it keeps us busy always building the next system; but we may be missing opportunities for fundamental progress and reuse. Surely humans do tune to particular tasks, for both speed and accuracy, and I’ll say more about that at the end. But that is perhaps on top of an architecture that also lets them puzzle out the answers new tasks without training from scratch.

7. Variable-centric view of the world
Focus on what variables might help us understand and produce language.
Some are token variables (utterance-specific), others are type variables (language-specific).
Might be observed or latent.
How those variables are represented, what the priors are, and (next slide) how they interrelate.
When we deeply understand language, what representations (type and token) does that understanding comprise?

8. Bayesian View of the World
observed data
probability distribution
hidden data

9. Different tasks merely change which variables are observed and which ones you care about inferring
comprehension
production
learning
sentence  ?
syntax tree
(?)
latent
()
semantics
facts about speaker/world
facts about the language

10. Different tasks merely change which variables are observed and which ones you care about inferring
comprehension
production
learning
surface form of word  ?
surface  underlying alignment
(?)
latent
()
underlying form of word
abstract morphemes in word
underlying form of morphemes (lexicon)
constraint ranking (grammar)

11. Different tasks merely change which variables are observed and which ones you care about inferring
MT decoding
MT training
cross-lingual projection
Chinese sentence 
Chinese parse
latent ?
English parse
English sentence
translation & language models

12. ∑latent p(observed, needed, latent)
All you need is “p”
Science = a descriptive theory of the world
Write down a formula for p(everything)
everything = observed  needed  latent
Given observed, what might needed be?
Most probable settings of needed are those that give comparatively large values of
∑latent p(observed, needed, latent)
Formally, we want p(needed | observed) = p(observed, needed) / p(observed)
Since observed is constant, the conditional probability of needed varies with p(observed, needed), which is given above
(What do we do then?)

13. All you need is “p” Science = a descriptive theory of the world
Write down a formula for p(everything)
everything = observed  needed  latent
p can be any non-negative function you care to design
(as long as it sums to 1)
(or another finite positive number: just rescale)
But it’s often convenient to use a graphical model
Flexible modeling technique
Well understood
We know how to (approximately) compute with them

14. Graphical model notation
slide thanks to Zoubin Ghahramani

15. slide thanks to Zoubin Ghahramani
Factor graphs
slide thanks to Zoubin Ghahramani

16. Rather basic NLP example
First, a familiar example
Conditional Random Field (CRF) for POS tagging
Possible tagging (i.e., assignment to remaining variables) … … v v v
find
preferred
tags
Observed input sentence (shaded) 16 16

17. Rather basic NLP example
First, a familiar example
Conditional Random Field (CRF) for POS tagging
Possible tagging (i.e., assignment to remaining variables)
Another possible tagging … … v a n
find
preferred
tags
Observed input sentence (shaded) 17 17

18. Conditional Random Field (CRF)
”Binary” factor that measures compatibility of 2 adjacent tags v n a 2 1 3 v n a 2 1 3
Model reuses same parameters
at this position … …
find
preferred
tags 18 18

19. Conditional Random Field (CRF)v n a 2 1 3 v n a 2 1 3 … … v
0.3 n
0.02 a v
0.3 n a
0.1 v
0.2 n a
find
preferred
can’t be adj
tags 19 19

20. Conditional Random Field (CRF)
p(v a n) is proportional to the product of all
factors’ values on v a n v n a 2 1 3 v n a 2 1 3 … … v a n v
0.3 n
0.02 a v
0.3 n a
0.1 v
0.2 n a
find
preferred
tags 20 20

21. Conditional Random Field (CRF)
p(v a n) is proportional to the product of all
factors’ values on v a n v n a 2 1 3 v n a 2 1 3
= … 1*3*0.3*0.1*0.2 … … … v a n v
0.3 n
0.02 a v
0.3 n a
0.1 v
0.2 n a
find
preferred
tags
MRF vs. CRF? 21 21

22. Inference: What do you know how to compute with this model?
p(v a n) is proportional to the product of all
factors’ values on v a n v n a 2 1 3 v n a 2 1 3
= … 1*3*0.3*0.1*0.2 … … … v a n v
0.3 n
0.02 a v
0.3 n a
0.1 v
0.2 n a
find
preferred
tags
Maximize, sample, sum … 22 22

23. Variable-centric view of the world
Focus on what variables might help us understand and produce language.
Some are token variables (utterance-specific), others are type variables (language-specific).
Might be observed or latent.
How those variables are represented, what the priors are, and (next slide) how they interrelate.
When we deeply understand language, what representations (type and token) does that understanding comprise?

24. To recover variables, model and exploit their correlations
semantics
lexicon (word types)
inflection
cognates
transliteration
abbreviation
neologism
language evolution
entailment
correlation
tokens
sentences N
translation
alignment
editing
quotation
resources
I don’t want to learn to do tasks but rather learn to understand what’s going on in the language I see;
happy to take help from direct or indirect supervision or other resources
speech
misspellings,typos
formatting
entanglement
annotation
discourse context
To recover variables, model and exploit their correlations

25. How do you design the factors?
It’s easy to connect “English sentence” to “Portuguese sentence” …
… but you have to design a specific function that measures how compatible a pair of sentences is.
Often, you can think of a generative story in which the individual factors are themselves probabilities.
May require some latent variables.

26. Directed graphical models (Bayes nets)
Under any model: p(A, B, C, D, E) = p(A)p(B|A)p(C|A,B)p(D|A,B,C)p(E|A,B,C,D)
Model above says:
slide thanks to Zoubin Ghahramani (modified)

27. Unigram model for generating text… w1 w2 w3
p(w1)  p(w2)  p(w3) …

28. Explicitly show model’s parameters 
“ is a vector that says which unigrams are likely” …  w1 w2 w3
p()  p(w1 | )  p(w2 | )  p(w3 | ) …

29. “Plate notation” simplifies diagram
“ is a vector that says which unigrams are likely”  w N1
p()  p(w1 | )  p(w2 | )  p(w3 | ) …

30. Learn  from observed words (rather than vice-versa)
p()  p(w1 | )  p(w2 | )  p(w3 | ) …

31. Explicitly show prior over  (e.g., Dirichlet)
“Even if we didn’t observe word 5, the prior says that 5 = 0 is a terrible guess”
 given
  Dirichlet()
wi     w N1
p()  p( | )  p(w1 | )  p(w2 | )  p(w3 | ) …

32. Dirichlet Distribution
Each point on a k dimensional simplex is a multinomial probability distribution: 1
dog
the
cat 1 1
dog
the
cat
slide thanks to Nigel Crook

33. Dirichlet Distribution
A Dirichlet Distribution is a distribution over multinomial distributions  in the simplex. 1 1 1 1 1
slide thanks to Nigel Crook

34. slide thanks to Percy Liang and Dan Klein

35. Dirichlet Distribution
Example draws from a Dirichlet Distribution over the 3-simplex:
Dirichlet(5,5,5)
Dirichlet(0.2, 5, 0.2) 1
Dirichlet(0.5,0.5,0.5)
slide thanks to Nigel Crook

36. Explicitly show prior over  (e.g., Dirichlet)
Posterior distribution p( | , w) is also a Dirichlet just like the prior p( | ).
“Even if we didn’t observe word 5, the prior says that 5 = 0 is a terrible guess”
prior = Dirichlet()  posterior = Dirichlet(+counts(w))
Mean of posterior is like the max-likelihood estimate of , but smooth the corpus counts by adding “pseudocounts” .
(But better to use whole posterior, not just the mean.)   w N1
p()  p( | )  p(w1 | )  p(w2 | )  p(w3 | ) …

37. Training and Test Documents
“Learn  from document 1, use it to predict document 2”
test
What do good configurations look like if N1 is large?
What if N1 is small? w N2
train   w N1

38. Many Documents 3 w 2 w  1 w
“Each document has its own unigram model” 3 w N3
Now does observing docs 1 and 3 help still predict doc 2?
Only if  learns that all the ’s are similar (low variance).
And in that case, why even have separate ’s? 2 w N2  1 w N1

39. “Each document has its own unigram model”
Many Documents
“Each document has its own unigram model”
or tuned to maximize training or dev set likelihood
 given
d  Dirichlet()
wdi  d   w ND D

40. Bayesian Text Categorization
“Each document chooses one of only K topics (unigram models)”
 given
k  Dirichlet()
wdi  k
but which k?   w K ND D

41. Bayesian Text Categorization
 given
  Dirichlet()
zd  
“Each document chooses one of only K topics (unigram models)” 
a distribution over topics 1…K
 given
k  Dirichlet()
wdi  zd 
Allows documents to differ considerably while some still share  parameters.
And, we can infer the probability that two documents have the same topic z.
Might observe some topics.
a topic in 1…K z   w K ND D

42. Latent Dirichlet Allocation (Blei, Ng & Jordan 2003)
“Each document chooses a mixture of all K topics; each word gets its own topic”   z   w K ND D

43. (Part of) one assignment to LDA’s variables
slide thanks to Dave Blei

44. (Part of) one assignment to LDA’s variables
slide thanks to Dave Blei

45. Latent Dirichlet Allocation: Inference?  … z1 z2 z3 …   w w1 w2 w3 K K D

46. Finite-State Dirichlet Allocation (Cui & Eisner 2006)
“A different HMM for each document”   … z1 z2 z3 …   w1 w2 w3 K D

47. Variants of Latent Dirichlet Allocation
Syntactic topic model: A word or its topic is influenced by its syntactic position.
Correlated topic model, hierarchical topic model, …: Some topics resemble other topics.
Polylingual topic model: All versions of the same document use the same topic mixture, even if they’re in different languages. (Why useful?)
Relational topic model: Documents on the same topic are generated separately but tend to link to one another. (Why useful?)
Dynamic topic model: We also observe a year for each document. The k topics  used in 2011 have evolved slightly from their counterparts in 2010.

48. slide thanks to Dave Blei
Dynamic Topic Model
slide thanks to Dave Blei

49. slide thanks to Dave Blei
Dynamic Topic Model
slide thanks to Dave Blei

50. slide thanks to Dave Blei
Dynamic Topic Model
slide thanks to Dave Blei

51. slide thanks to Dave Blei
Dynamic Topic Model
slide thanks to Dave Blei

52. Remember: Finite-State Dirichlet Allocation (Cui & Eisner 2006)
“A different HMM for each document”   … z1 z2 z3 …   w1 w2 w3 K D

53. “Shared HMM for all documents” (or just have 1 document)
Bayesian HMM
“Shared HMM for all documents”
(or just have 1 document)   … z1 z2 z3   w1 w2 w3 K D
We have to estimate transition parameters  and emission parameters .

54. FIN