1. An Algorithm for Computing the Gene Tree Probability under the Multispecies Coalescent and its Application in the Inference of Population Tree
Yufeng Wu
Dept. of Computer Science & Engineering
University of Connecticut, USA
ISMB 2016, July 11, 2016

2. Coalescent Theory: Introduction
Population gene genealogy
Coalescent Theory: Introduction
Coalescence
Allele
Sampled allele
Generation
Wright-Fisher Model:
Non-overlapping generations
Constant population size
Random mating
Time
Coalescent: trace backward in time from sampled lineages.
Coalescence: two sample lineages find common ancestor.
Gene genealogy: determined by coalescent process
Stochastic: when coalescence occurs is a stochastic process

3. Coalescent in Multiple Populations
“Outline" tree is the species (population) tree: evolutionary history of populations T
“Embedded" tree is the gene tree: evolutionary history of individual alleles
Coalescent: tracing lineages backward in time to common ancestor in a population Multispecies coalescent: each extant/ancestral population runs a separate coalescent: determines gene genealogy with multiple populations
Coalescence
Gene tree/species tree discordance: gene trees for the a species tree with different topology due to stochastic coalescence.
Stochasticity: inherent in coalescent process (a gene tree has certain probability of being observed under the coalescent process).
Larger T: B and C more likely to coalesce within T
Small T: B and C may not coalesce with T (and may lead to different gene tree)
Multispecies coalescent: allow computation of probability of gene genealogies from multiple species (populations)

4. Gene Tree Probability
For a species tree, any gene tree topology can arise, but with probability.
For species tree Ts (with branch length) and a gene tree topology Tg:
Gene tree probability P(Tg|Ts): probability of observing a binary gene tree topology Tg for species tree Ts under coalescent theory.
The larger P(Tg|Ts) is, the more likely Tg will be observed.
Has multiple applications in population and evolutionary genomics.
e.g. Infer Ts from multiple gene genealogies Tg by maximum likelihood
For small gene tree and species tree, calculation by hand may be feasible (e.g. Hudson, 1983, Takahata and Nei, 1985, Rosenberg 2002): usually listing all possible gene trees and species trees with say five species.
For larger trees, an algorithm is needed. (Nowadays, gene trees and species trees can have hundreds of taxa/alleles)
Key: efficient computation of the gene tree probability.

5. An algorithm for Gene Tree Probability (Degnan and Salter, 2005)
History 1
History 2. Same gene tree
Coalescent history: each coalescent event occur at which species tree branch:
a and b coalesce within T
Then coalescence with c above T.
Note: a and b can coalesce above T: give different history T T a b c a b c
Degnan and Salter: P(Tg|Ts)=H P(Tg, H|Ts), H: coalescent histories of Tg
Why enumerate H? H specifies for each species tree branch, there are some u gene lineages coalescing into v lineages.
v lineages
Classic result in coalescent theory (Tavare 1984; Watterson 1984; Takahata and Nei 1985)
puv(T): probability of u (unlabeled) lineages coalesce to v lineages within time T is: T
u lineages
P(Tg, H|Ts): product of puv(T) over all species tree branches (independence assumed)

6. An algorithm for Gene Tree Probability (Degnan and Salter, 2005)
Recall: puv(T): the probability of u (not labeled) lineages coalesce to v lineages within time T
P(Tg, H|Ts) = p21(T1)
For a fixed coalescent history H:
* p22(T2)
* p31()
* C (combinatorial factor) 
Main challenge: need to consider all possible coalescent histories. T2
Assume coalescent events along different species tree branch are independent. Then, the gene tree probability is equal to the product of Puv(Ti) for each branch bi.
No known polynomial time algorithms for gene tree probability computation.
STELLS algorithm (Wu, Evolution, 2012): another algorithm for gene tree probability. Faster than Degnan and Salter’s but still slow for large trees (exponential time in general) T1

7. Population genetics: large number of gene alleles (small number of populations)
Phylogenetics study: Hundreds of species; 1 or several alleles per species
Population genetics study: 1000 Genomes Project: 26 populations; haplotypes from 2504 individuals
Question: can gene tree probability be computed efficiently for many gene alleles when the number of populations is small?
Application: inference of demographic history
This talk: Polynomial-time algorithm for computing gene tree probability computation when the number of populations is fixed to a constant
Basic idea: merge multiple coalescent history into a compact coalescent history

8. Compact Coalescent History (CCH)
Upper lineage count (ULC): number (not specific lineages as in coalescent history) of gene lineages at the top of population tree branch.
Compact Coalescent History (CCH)
ULC3=1
ULC3=1 h h
Coalescent history 2
Coalescent history 1 g d g c
ULC2=1
ULC2=1
ULC1=3
ULC1=3 f f A A B B e c e d a1 a2 a3 a4 b1 b2 b3 a1 a2 a3 a4 b1 b2 b3
CCH: (3, 1, 1)
Compact coalescent history (CCH): ordered list of ULC for each population tree branch
Two different coalescent histories Can lead to same CCH
Number of CCH is smaller than number of coalescent history
Gene tree probability: can be computed from CCH (details omitted)

9. Number of Compact Coalescent History is Polynomial Bounded for Constant Number of Populations
n: number of gene lineages
m: number of population tree branches When the number of populations is constant  m is constant
CCH: (c1,c2,c3,c4,1) c3 c4
c1 c2, c3, c4 ≤ n
|CCH| = m (constant) c2 c1
CCH: length m vector of integers. Each position, value range from 1 to n.
The number of CCH ≤ nm. Polynomial in n when m is constant.
Time (seconds)
Run-time (seconds) for computing the gene tree probability for 500 gene trees using new algorithm and original STELLS: two populations.
Allele number per population

10. Application: Inference of Population Trees Using Pairwise Distance (also see Wu, Bioinformatics, 2015)
Population haplotypes
Gene genealogies
Estimate population divergence distance of two populations
Population tree
Assume: infinite sites model and no intra-locus recombination
Maximum likelihood estimate (gene tree prob algorithm)
Neighbor joining
a1 AAAA
a2 CAGA
b1 CTGA
b2 AAAC A B
Haplotypes of two populations A and B b1 a2 a1 b2
AAAA 1 2 3 4
Mutation
D A B C A B C
Pairwise population distance A B b1 a2 b2 a1
1.1
0.9
1.0 A C B
Compare w/ TreeMix (Pickrell and Pritchard, 2012)
doesn’t consider linkage disequilibrium or LD).
Our approach is more accurate (but slower) than TreeMix
20 alleles for 8 populations each. Population tree height of 0.1
Average Robinson-Foulds error for our method: 0.11
TreeMix: 0.18
Partly supported by U.S. National Science Foundation grants IIS and IIS