1. Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Arun ganesh (UC BERKELEY)
With QIUYI (RICHARD) ZHANG (UC BERKELEY Q GOOGLE)
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels

2. Setup Start with a “model” binary tree 𝑛 leaves = extant species
Image source: Bulbapedia
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

3. Setup 𝜎 𝑟 ~ {0, 1} 𝑘 Start with a “model” binary tree
𝜎 𝑟 ~ {0, 1} 𝑘
Start with a “model” binary tree
Sample a uniformly random bitstring (DNA) for root
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

4. Setup 011001010 Start with a “model” binary tree
Sample a uniformly random bitstring (DNA) for root
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

5. Setup 011001010 Start with a “model” binary tree
Sample a uniformly random bitstring (DNA) for root
DNA is inherited down the tree with mutations. On edge 𝑒, each bit:
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

6. Setup 011001010 Start with a “model” binary tree
Sample a uniformly random bitstring (DNA) for root
DNA is inherited down the tree with mutations. On edge 𝑒, each bit:
-Substitutes w.p. 𝑝 𝑠 (𝑒)
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

7. Setup 011001010 Start with a “model” binary tree
Sample a uniformly random bitstring (DNA) for root
DNA is inherited down the tree with mutations. On edge 𝑒, each bit:
-Substitutes w.p. 𝑝 𝑠 (𝑒)
-Inserts random bit w.p. 𝑝 𝑖 (𝑒)
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

8. Setup 011001010 Start with a “model” binary tree
Sample a uniformly random bitstring (DNA) for root
DNA is inherited down the tree with mutations. On edge 𝑒, each bit:
-Substitutes w.p. 𝑝 𝑠 (𝑒)
-Inserts random bit w.p. 𝑝 𝑖 (𝑒)
-Deletes w.p. 𝑝 𝑑 (𝑒) _
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

9. Setup 011001010 Start with a “model” binary tree
Sample a uniformly random bitstring (DNA) for root
DNA is inherited down the tree with mutations. On edge 𝑒, each bit:
-Substitutes w.p. 𝑝 𝑠 (𝑒)
-Inserts random bit w.p. 𝑝 𝑖 (𝑒)
-Deletes w.p. 𝑝 𝑑 (𝑒)
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

10. Setup Start with a “model” binary tree
Start with a “model” binary tree
Sample a uniformly random bitstring (DNA) for root
DNA is inherited down the tree with mutations. On edge 𝑒, each bit:
-Substitutes w.p. 𝑝 𝑠 (𝑒)
-Inserts random bit w.p. 𝑝 𝑖 (𝑒)
-Deletes w.p. 𝑝 𝑑 (𝑒)
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

11. Setup Start with a “model” binary tree
Sample a uniformly random bitstring (DNA) for root
DNA is inherited down the tree with mutations. On edge 𝑒, each bit:
-Substitutes w.p. 𝑝 𝑠 (𝑒)
-Inserts random bit w.p. 𝑝 𝑖 (𝑒)
-Deletes w.p. 𝑝 𝑑 (𝑒)
Algorithm is given leaf bitstrings
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

12. Setup Start with a “model” binary tree
Sample a uniformly random bitstring (DNA) for root
DNA is inherited down the tree with mutations. On edge 𝑒, each bit:
-Substitutes w.p. 𝑝 𝑠 (𝑒)
-Inserts random bit w.p. 𝑝 𝑖 (𝑒)
-Deletes w.p. 𝑝 𝑑 (𝑒)
Algorithm is given leaf bitstrings and must reconstruct the tree with high probability
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

13. Goal Reconstruct the tree with high probability,
using as short a sequence length 𝑘 as possible (as a function of 𝑛),
while tolerating as large mutation probabilities 𝑝 𝑠 , 𝑝 𝑖 , 𝑝 𝑑 as possible.
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

14. Motivation Many practical applications:
Reconstructing paths of migration
Linking mutations to disease
Determining origins of pathogens and likely paths of contamination
Informing policy on conservation of species
Image source: Tim Lohrentz
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

15. Prior work
One method: Try to align sequences, reducing to substitution-only case.
Image source: Wikipedia
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

16. Prior work
Theorem [DMR05]: Can reconstruct tree in substitution-only case using 𝑂( log 𝑛 ) bits.
With 𝑜( log 𝑛 ) length bitstrings, problem is impossible.
Requires 2 1−2 𝑝 𝑠 2 >1 (Kesten-Stigum threshold).
[BRZ95, Iof96, EKP00, BKM01, MSW04]: If 2 1−2 𝑝 𝑠 2 <1, need 𝑛 Ω 1 bits for reconstruction.
If we have good alignment methods, we’ve solved the problem to optimality!
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

17. Prior work
Unfortunately, multiple sequence alignment is NP-hard, and heuristics used in practice may induce problematic biases.
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

18. Prior work
GWEJ07, LG08, WSH08 give empirical evidence for biases in MSA.
DR10, ABH12 provide some guarantees for reconstruction with indels, but require polynomial sequence lengths or very small 𝑝 𝑖 , 𝑝 𝑑 .
What 𝑝 𝑖 , 𝑝 𝑑 can we handle with 𝑘= log 𝑂 1 𝑛 ?
𝑝 𝑖 , 𝑝 𝑑
DR10
GZ18
𝑂 1
ABH12
𝑂 1 log 2 𝑛
ESSW99
DMR05 𝑘
𝑛 𝑂 1
log 𝑂 1 𝑛
𝑂 log 2 𝑛
We show the answer is 𝑂(1).
𝑂 log 𝑛
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

19. Our contribution In particular, if:
2 1−2 𝑝 𝑠 𝑒 − 𝑝 𝑑 𝑒 𝑝 𝑖 𝑒 − 𝑝 𝑑 𝑒 −1 >1 for all edges (Kesten-Stigum threshold)
𝐷 𝑚𝑎𝑥 ≤ log 𝛼 𝑛
𝑝 𝑖 𝑒 − 𝑝 𝑑 𝑒 ≤ 𝛽 log log 𝑛 𝐷 𝑚𝑎𝑥
Then we can reconstruct the tree with 𝑘= log 𝜅 𝛼, 𝛽 𝑛 .
𝜅 𝛼,𝛽 is optimal up to a small multiplicative constant, and otherwise this result is optimal in every possible sense!
needed to avoid empty leaf strings
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

20. Roadmap Substitution only With indels
1. Estimating distances using bitwise correlation (ESSW99)
2. Using block signatures instead of bits (DR10)
3. Reconstructing bitstrings of internal nodes (Roc08)
4. Reconstructing signatures of internal nodes (GZ18)
𝑘= 𝑛 𝑂 1
5. Future Directions
𝑘= log 𝑂 1 𝑛
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

21. Roadmap Substitution only With indels
1. Estimating distances using bitwise correlation (ESSW99)
2. Using block signatures instead of bits (DR10)
3. Reconstructing bitstrings of internal nodes (Roc08)
4. Reconstructing signatures of internal nodes (GZ18)
𝑘= 𝑛 𝑂 1
5. Future Directions
𝑘= log 𝑂 1 𝑛
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
SETUP

22. ESTIMATING DISTANCES USING BITWISE CORRELATION
Distance estimation
Well known: distance estimates that concentrate well suffice to reconstruct the tree.
High-level algorithm:
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
ESTIMATING DISTANCES USING BITWISE CORRELATION

23. ESTIMATING DISTANCES USING BITWISE CORRELATION
Distance estimation
Well known: distance estimates that concentrate well suffice to reconstruct the tree.
High-level algorithm:
1. Use distances to identify siblings
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
ESTIMATING DISTANCES USING BITWISE CORRELATION

24. ESTIMATING DISTANCES USING BITWISE CORRELATION
Distance estimation
Well known: distance estimates that concentrate well suffice to reconstruct the tree.
High-level algorithm:
1. Use distances to identify siblings
2. Use distances to compute distances from parents
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
ESTIMATING DISTANCES USING BITWISE CORRELATION

25. ESTIMATING DISTANCES USING BITWISE CORRELATION
Distance estimation
Well known: distance estimates that concentrate well suffice to reconstruct the tree.
High-level algorithm:
1. Use distances to identify siblings
2. Use distances to compute distances from parents
3. Recurse on parents
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
ESTIMATING DISTANCES USING BITWISE CORRELATION

26. ESTIMATING DISTANCES USING BITWISE CORRELATION
To estimate distance, in substitution-only case can use bitwise correlation (linear rescaling of Hamming distance).
Think of bits as ±1 instead of 0-1. Let 𝜎 𝑎,𝑗 be 𝑗th bit of 𝑎’s bitstring. Bitwise correlation is 1 𝑘 𝑗=1 𝑘 𝜎 𝑎,𝑗 𝜎 𝑏,𝑗 .
Faraway nodes have dissimilar bitstrings
Siblings have very similar bitstrings
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
ESTIMATING DISTANCES USING BITWISE CORRELATION

27. ESTIMATING DISTANCES USING BITWISE CORRELATION
Claim: If we define edge lengths as
𝑑 𝑒 =− ln 1−2 𝑝 𝑠 𝑒
then E 1 𝑘 𝑗=1 𝑘 𝜎 𝑎,𝑗 𝜎 𝑏,𝑗 = 𝑒∈ 𝑃 𝑎,𝑏 1−2 𝑝 𝑠 𝑒 = 𝑒 −𝑑 𝑎,𝑏
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
ESTIMATING DISTANCES USING BITWISE CORRELATION

28. Concentration of bitwise correlation
How well does it concentrate?
Rough analysis:
Bitwise correlation has standard deviation ≈ 1 𝑘 .
For the correlation to concentrate at distance O log 𝑛 , need 𝑒 −𝑂 log 𝑛 > 1 𝑘 →𝑘= 𝑛 Ω 1 .
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
ESTIMATING DISTANCES USING BITWISE CORRELATION

29. Using block signatures
Roadmap
Substitution only
With indels
1. Estimating distances using bitwise correlation (ESSW99)
2. Using block signatures instead of bits (DR10)
3. Reconstructing bitstrings of internal nodes (Roc08)
4. Reconstructing signatures of internal nodes (GZ18)
𝑘= 𝑛 𝑂 1
5. Future Directions
𝑘= log 𝑂 1 𝑛
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Using block signatures

30. Using block signatures
Handling indels
Problem with bitwise correlation when indels are introduced: indels make bits move around a lot between bitstrings. 𝑗 𝑎
What bit in 𝑏’s bitstring does bit 𝑗 of 𝑎’s bitstring correspond to?
In the substitution only case:
With indels:
What if it doesn’t appear at all? 𝑏
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Using block signatures

31. Blockwise correlation
To handle shifts due to insertions and deletions, split bitstrings into blocks.
𝑘 3/4 𝑗 𝑎
Assume 𝑝 𝑖 = 𝑝 𝑑 everywhere
(not too hard to generalize).
If we split bitstrings into blocks of length, say, 𝑙= 𝑘 3/4 , most bits will stay within a block throughout the tree.
Any bit shifts by < 𝑘 positions in expectation, at most 𝑘 log 𝑛 with high probability on one edge.
< 𝑘 log 𝑛 𝑏
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Using block signatures

32. Blockwise correlation
Define signature of block 𝑖 in bitstring 𝑎, 𝑠 𝑎,𝑖 as sum of bits in block 𝑖, divided by 𝑙 .
Signatures 𝑠 𝑎,𝑖 are robust to shifts, so they behave like bits in substitution only case, i.e. 𝑠 𝑎,𝑖 𝑠 𝑏,𝑖 behaves like bitwise correlation.
𝑠 𝑎,𝑖 𝑠 𝑏,𝑖 = 1 𝑙 𝑗 𝑗 ′ 𝜎 𝑎,𝑗 𝜎 𝑏, 𝑗 ′
Fixing any series of indels, 𝜎 𝑎,𝑗 𝜎 𝑏, 𝑗 ′ is non-zero in expectation only if bits 𝑗 and 𝑗′ correspond to each other.
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Using block signatures

33. Blockwise correlation
Define 𝑑 𝑒 =− ln 1−2 𝑝 𝑠 𝑒 1− 𝑝 𝑑 𝑒
Lemma: E 𝑠 𝑎,𝑖 𝑠 𝑏,𝑖 = 1±𝑜 1 𝑒∈ 𝑃 𝑎,𝑏 1−2 𝑝 𝑠 𝑒 1− 𝑝 𝑑 𝑒 ≈ 𝑒 −𝑑(𝑎,𝑏)
Error term to account for the tiny fraction of bits that move in/out of blocks
Decay in number of corresponding bits between the blocks due to deletions
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Using block signatures

34. Challenges with concentration
𝑠 𝑎,𝑖 𝑠 𝑏,𝑖 includes lots of non-corresponding bitwise products that are expectation 0, but total variance may be high.
But each 𝑠 𝑎,𝑖 is 𝑂( log 𝑛 ) with high probability, and we have 𝑘 1/4 products – can boost 𝑘 to get concentration via Azuma’s if 𝑠 𝑎,𝑖 𝑠 𝑏,𝑖 are independent.
Unfortunately 𝑠 𝑎,𝑖 𝑠 𝑏,𝑖 and 𝑠 𝑎, 𝑖 ′ 𝑠 𝑏, 𝑖 ′ are not independent.
Knowing 𝑠 𝑎,𝑖 𝑠 𝑏,𝑖 gives you information about the indel process between 𝑎 and 𝑏, and thus about 𝑠 𝑎, 𝑖 ′ 𝑠 𝑏, 𝑖 ′ (since indels affect many blocks simultaneously).
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Using block signatures

35. Challenges with concentration
Our fix: Suppose we could define blocks based on actual alignment. Let 𝑠 𝑎,𝑖 ∗ be the signatures of these blocks.
Indels can only affect one “aligned” signature, so 𝑠 𝑎,𝑖 ∗ 𝑠 𝑏,𝑖 ∗ are independent and concentrate nicely.
We don’t know actual alignment, so can’t compute 𝑠 𝑎,𝑖 ∗ , but we can show 𝑠 𝑎,𝑖 𝑠 𝑏,𝑖 is very close to 𝑠 𝑎,𝑖 ∗ 𝑠 𝑏,𝑖 ∗ . 𝑎
Blocks our estimator uses
Aligned blocks 𝑏
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Using block signatures

36. Reconstructing bitstrings
Roadmap
Substitution only
With indels
1. Estimating distances using bitwise correlation (ESSW99)
2. Using block signatures instead of bits (DR10)
3. Reconstructing bitstrings of internal nodes (Roc08)
4. Reconstructing signatures of internal nodes (GZ18)
𝑘= 𝑛 𝑂 1
5. Future Directions
𝑘= log 𝑂 1 𝑛
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Reconstructing bitstrings

37. Challenges with 𝑘= log 𝑂(1) 𝑛
With 𝑘= log 𝑂 1 𝑛 , correlation only concentrates at distance 𝑂( log log 𝑛 ).
Can show far apart nodes have low correlation with high probability (no false positives), so can still reconstruct first Ω( log log 𝑛 ) levels just based off leaf distance estimates.
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Reconstructing bitstrings

38. Challenges with 𝑘= log 𝑂(1) 𝑛
𝑑(𝑎,𝑏) =O(log log 𝑛 ) 𝑎 𝑏
Use estimator
𝜎 𝑎,𝑚 = 1 2 ℎ 𝑗 𝑒 𝑑 𝑎, 𝑎 𝑗 𝜎 𝑎 𝑗 ,𝑚
Correct in expectation!
ℎ =Ω(log log 𝑛 )
But variance of correlation of 𝜎 𝑎 and 𝜎 𝑏 is large.
𝑎 1
𝑎 2 …
𝑎 2 ℎ
𝑏 1
𝑏 2 …
𝑏 2 ℎ
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Reconstructing bitstrings

39. Challenges with 𝑘= log 𝑂(1) 𝑛
To get better concentration, use median-of-means approach.
𝑑(𝑎,𝑏) =O(log log 𝑛 ) 𝑎 𝑏
For each descendant 𝑎 𝑗 , 𝑏 𝑗 at height ℎ below, compute 𝜎 𝑎 𝑗 , 𝜎 𝑏 𝑗 , and use correlation to get estimate of 𝑑 𝑎 𝑗 , 𝑏 𝑗 (and thus 𝑑(𝑎, 𝑏)).
ℎ =log log 𝑛
𝑎 1
𝑎 2 …
𝑎 2 ℎ
𝑏 1
𝑏 2 …
𝑏 2 ℎ
𝐴 1
𝐵 1
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Reconstructing bitstrings

40. Challenges with 𝑘= log 𝑂(1) 𝑛
We get log 𝑛 estimates of 𝑑(𝑎, 𝑏), one for each 𝑎 𝑗 , 𝑏 𝑗 pair.
𝑑(𝑎,𝑏) =O(log log 𝑛 ) 𝑎 𝑏
We can condition on having “good” bitstrings at all 𝑎 𝑗 , 𝑏 𝑗 .
ℎ =log log 𝑛
The log 𝑛 estimators of 𝑑(𝑎, 𝑏) are conditionally independent, so median concentrates well.
𝑎 1
𝑎 2 …
𝑎 2 ℎ
𝑏 1
𝑏 2 …
𝑏 2 ℎ
𝐴 1
𝐴 1
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Reconstructing bitstrings

41. Reconstructing signatures
Roadmap
Substitution only
With indels
1. Estimating distances using bitwise correlation (ESSW99)
2. Using block signatures instead of bits (DR10)
3. Reconstructing bitstrings of internal nodes (Roc08)
4. Reconstructing signatures of internal nodes (GZ18)
𝑘= 𝑛 𝑂 1
5. Future Directions
𝑘= log 𝑂 1 𝑛
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Reconstructing signatures

42. Reconstructing signatures
Signatures are robust to indels, so they behave similarly to bits in substitution-only case.
Suggests our algorithm: apply the reconstruction scheme to signatures.
Some technical challenges in the analysis we need to overcome.
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Reconstructing signatures

43. Challenges with reconstruction with indels
In reconstructing signatures, bits appearing in blocks of ancestors but not children or vice-versa may induce noise that is non-zero in expectation in the recursive estimator.
We show that since the noise also “decays”, it is tiny in expectation, so misalignment does not ruin the reconstructed signatures.
𝑎, ancestor which we condition on
signal
noise
decayed signal
noise
𝑎 𝑗 , descendant
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Reconstructing signatures

44. Challenges with reconstruction with indels
We again have to deal with 𝑠 𝑎 𝑗 ,𝑖 𝑠 𝑏 𝑗 ,𝑖 and 𝑠 𝑎 𝑗 , 𝑖 ′ 𝑠 𝑏 𝑗 , 𝑖 ′ not being independent when analyzing the variance of our recursive estimator for 𝑑 𝑎, 𝑏 .
We show that the covariance of the reconstructed blockwise correlations is small, i.e. 𝑠 𝑎,𝑖 𝑠 𝑏,𝑖 and 𝑠 𝑎, 𝑖 ′ 𝑠 𝑏, 𝑖 ′ are almost completely independent.
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Reconstructing signatures

45. Roadmap Substitution only With indels
1. Estimating distances using bitwise correlation (ESSW99)
2. Using block signatures instead of bits (DR10)
3. Reconstructing bitstrings of internal nodes (Roc08)
4. Reconstructing signatures of internal nodes (GZ18)
𝑘= 𝑛 𝑂 1
5. Future Directions
𝑘= log 𝑂 1 𝑛
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Future directions

46. Open questions/future directions
Our reconstruction guarantees are optimal up to some constants – what are the right constants?
We only use 𝑂 𝑘 bits of information per 𝑘-bit sequence, so there should be room for improvement.
Can we remove some of the strong assumptions in the model?
What if there isn’t sitewise independence of mutations?
What if the root bitstring isn’t chosen uniformly at random?
What if different parts of each sequence are generated using different trees?
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels
Future directions

47. Thank You! Questions?
Optimal Sequence Length Requirements for Phylogenetic Tree Reconstruction with Indels