The multispecies coalescent: implications for inferring species trees James Degnan 21 February 2008

Outline 1. Background --gene trees vs. species trees --coalescence and incomplete lineage sorting 2. Inferring species trees --Concatenation --Consensus Trees 3. Conclusions

Population Genetics and Phylogenetics Population genetics: traditionally used to analyze single populations. Phylogenetics: What is the best way to infer relationships between populations/species? Graphic by Mark A. Klinger, Carnegie Museum of Natural History, Pittsburgh

Desirable properties of species tree estimators 1. Statistical consistency (sample size = # of genes) 2. Efficiency 3. Robustness to violations in assumptions

Bridging the popgen/phylo divide “Incorporation of explicit models of lineage sorting will be needed for continued development of phylogenetic inference near the species level.” –Maddison and Knowles (2006). “Closer integration of population-genetic factors in phylogenetics, including further insights into gene-tree/species tree, and horizontal gene transfer.” --from Mike Steel’s website, My pick for five directions in phylogenetics that will grow in the next five years (2006).

The coalescent process Past Present

One population

Multiple populations/species Present Past

Gene tree in a species tree

Model species tree with gene tree A B C D The gene tree is a random variable. The gene tree distribution is parameterized by the species tree topology and internal branch lengths.

How can we compute probabilities of gene trees given species trees? -Under a coalescent model, probabilities for gene trees with three species were derived by Nei (1987): 1-(2/3)e-T -Probabilities for the gene tree to match the species tree topology for 4 and 5 species given by Pamilo and Nei (1988). -All 30 species tree/gene tree combinations for 4 species given by Rosenberg (2002). -General case solved by Degnan and Salter (2005) and implemented by program COAL. Also allows individuals sampled in species i.

This coalescent history: (1,3,3) Definition: a coalescent history is a list of the populations in which each coalescent event occurs. A B C D This coalescent history: (1,3,3) Other coalescent histories: (2,3,3), (3,3,3)

Gene tree probabilities

Gene tree probabilities combinatorial enumeration, complexity only known in special cases probability coalescences are consistent with g branch length u coalesce into v internal branches of S

Data from Ebersberger et al. 2007. Mol. Biol. Evol. 24:2266-2276. Theoretical distribution based on parameters from Rannala and Yang, 2003. Genetics 164:1645-1656. 1.2 4.2 t/N =

y x

Definition: a gene tree which is more probable than the gene tree matching the species tree is called an anomalous gene tree (Degnan and Rosenberg, 2006). Theorem 1. For the asymmetric species tree topology with four species and for any species tree topology with more than four species, there exist branch lengths such that at least one gene tree is anomalous (Degnan and Rosenberg, 2006).

Is species tree inference consistent in this setting? 1. Concatenation? 2. Consensus?

Species Tree inference—concatenation Species Trees are often estimated by concatenating several gene sequences and analyzing as one (data from Chen and Li, 2001). Gene 1 Human CTTGAATAATTTTTAC Chimp CTTCAATAATTTTTAC Gorilla TTTGAATAATTTTTAC Orang CTTGAATAATTTTTAT Gene 2 TAGAGTTTCCTTGTGGTG TAGAGTTTCCTTGTGGTA CAGAGTTTCCTTGTGGTC Gene 3 CGGTTT TGGTTT CRGTTT

Concatenation and gene tree discordance How does concatenation perform when sequences are generated from different topologies? CGGTTT TGGTTA TAGTTA CGATTA TGATTA TAATTT TGAATT TGCTAT CCCTAT Simulated gene trees Species tree: y = 1.0, x = 0.05 y x CGGTTT TGGTTA TAGTTA CGATTA TGATTA TAATTT TGAATT TGCTAT CCCTAT concatenated sequence

Trees inferred from concatenated sequences (Kubatko and Degnan, 2007) y = 1.0, x = 0.05 Number of genes

Is species tree inference consistent in this setting? 1. Concatenation? No. 2. Consensus?

Consensus (majority-rule)

Types of consensus trees Majority rule—consensus tree has all clades that were observed in > 50% of trees. Greedy—sort clades by their proportions. Accept the most frequently observed clades one at a time that are compatible with already accepted clades. Do this until you have a fully resolved tree. R*—for each set of 3 taxa, find the most commonly occurring triple e.g., (AB)C, (AC)B or (BC)A. Build the tree from the most commonly occurring triple. (AB)D, (CD)B are two rooted triples

Asymptotic consensus trees Consensus trees are usually statistics, functions of data like x-bar. Definition: an asymptotic consensus tree is the tree that is obtained by computing the consensus tree using topology probabilities from the multispecies coalescent model. Motivation: if there are a large number of independent loci, observed gene tree, clade, and rooted triple proportions should approximate their theoretical probabilities.

Simulated gene trees Greedy consensus tree

Greedy consensus tree

Greedy consensus tree Simulated gene trees Greedy consensus tree R* consensus tree

Majority-rule: unresolved zone

Too-greedy zone

Is species tree inference consistent in this setting? 1. Concatenation? No. 2. Consensus? Yes (R*), no for greedy and majority-rule.

Are consensus trees inconsistent estimators of species trees? Theorem 2. (i) Majority-rule asymptotic consensus trees (MACTs) do not have any clades not on the species tree. (ii) Majority-rule unresolved zones exist for any species tree topology with n ≥ 3 species. Theorem 3. Greedy asymptotic consensus trees (GACTs) can be misleading estimators of species trees for the 4-species asymmetric tree and for any species tree with n > 4 species. Theorem 4. R* asymptotic consensus trees (RACTs) always match the species tree.

What about finite samples? If you sample 10 loci, you could have: All 10 match the species tree 9 match the species tree, 1 disagrees 8 match the species tree, 2 disagree, etc. You can consider gene trees as categories and use multinomial probabilities for the probability of your sample

R* consensus, y = 0.4, x = 0.6

Conclusion Coalescent gene tree probabilities can be used to prove or disprove the statistical consistency of species tree estimators.

R* consensus, y = x = 0.1 Number of genes Probability