Comparing Database Search Methods & Improving the Performance of PSI-BLAST Stephen Altschul

“Gold standards” for protein classification Traditional curated sequence databases with family and superfamily classifications: PIR SWISS-PROT Structure-based protein domain classification: SCOP

Measuring retrieval accuracy Sequence Search RelatedUnrelated Positive TP True Positive FP False Positive P = TP + FP Negative FN False Negative TN True Negative N = FN + TN R = TP + FN U = FP + TN Sensitivity: TP/RSpecificity: TP/P

Receiver Operating Characteristic curve False +True – False – True +

Random retrieval on a ROC plot

Line of fixed sensitivity

Line of fixed specificity

Line of fixed crossover ratio

ROC score: area under the ROC curve

Region of interest in ROC analysis

Truncated ROC, or ROC n curve 0 10 –3 Fraction unrelated accepted

ROC n score: area under the ROC n curve

Questions concerning ROC analysis What false-positive cutoff value should be used? When does it make sense to pool the results of database searches? When are the ROC scores for two different methods significantly different?

Marginal ratio of true to false positives

Definition of the ROC n score t: Total number of related sequences t i : Number of related sequences (true positives) returned before the ith false positive

“Random distribution” of ROC n scores Bootstrap resampling can be used to assign a statistical significance to differences in ROC n scores. Under reasonable assumptions, the distribution of bootstrapped ROC n scores is approximately normal. Resampling a small subset in a large database is equivalent to resampling the subset with independent Poisson distributions with mean 1.

Bootstrap resampling of false positives Retrieval Ranking of the Database The false records are the noise Only false records are resampled with replacement The true records are well characterized.

Mean and variance for the normal distribution of ROC n scores yielded by resampling only the false positives

Mean and variance for the normal distribution of the difference of two ROC n scores, yielded by resampling only the false positives

PSI-BLAST in a nutshell With a protein sequence as query, use BLAST to search a protein sequence database. Collapse significant local alignments (those with E- value less than or equal to a set threshold h) into a multiple alignment, using the residues of the query sequence as alignment-column placeholders. Abstract a position-specific score matrix from the multiple alignment. Search the database with the score matrix as query. Iterate a fixed number of times, or until convergence.

Protocol for evaluating PSI-BLAST For each query sequence, search a comprehensive protein sequence database (e.g. NCBI’s nr) through a fixed number of PSI-BLAST iterations, or until convergence. Use the resulting position-specific score matrix to search the “gold standard” database. Pool the search results for ROC analysis.

The effect of acceptance threshold h on PSI-BLAST accuracy

Some ideas for improving PSI-BLAST 1. New statistical parameters 2. Smith-Waterman alignment 3. Substitution matrix frequency ratios 4. Apply SEG to database sequences 5. Composition-based statistics 6. “Concentrated” accounting of gaps 7. “Dispersed” accounting of gaps 8. Exponentiate Henikoff weights 9. Reverse sequence normalization 10. Window for amino acid composition 11. Use pseudocounts with composition window 12. Vary gap costs 13. Generalized affine gap costs 14. Substitution score offset 15. Information-dependent pseudocount parameter 16. Database-sequence length- normalization 17. Restricted score rescaling 18. Adjust purging percentage 19. Adjust pseudocount parameter 20. Adjust acceptance threshold

The effect of composition-based statistics on PSI-BLAST accuracy

Composition-based statistics Statistics based on “standard” amino acid frequencies can differ by orders of magnitude from those based upon the peculiar composition of two proteins. Standard protein: 4.5 % N DNA pol III, β chain [M. genitalium]: 12.1 % N DNA pol III, β chain [C. jejuni]: 7.6 % N Depending upon the composition assumed, a search of nr with M. genitalium DNA pol III as query yields different E-values for C. jejuni DNA pol III, as well as for the highest-scoring false positive: “Standard” statistics: Composition-based statistics: At a threshold of , “standard” statistics yield 54 true positives, while at 0.1, composition-based statistics yield 55 true positives.

The effect of dispersed accounting of gaps on PSI-BLAST accuracy

The effect of restricted score rescaling and parameter tuning on PSI-BLAST accuracy

Accuracy of PSI-BLAST Program versionROC 100 score Original h = ± Composition-based statistics h = ± “Dispersed” gap accounting h = ± Restricted score rescaling b = 9 ; p = ± 0.003

PSI-BLAST accuracy as a function of the number of iterations

Literature ROC analysis Swets, J.A. (1988) Science 240: Gribskov, M. & Robinson, N.L. (1996) Comput. Chem. 20:25-33 PSI-BLAST Altschul, S.F. et al. (1997) Nucl. Acids Res. 25: Composition-based statistics Karplus, K. et al. (1998) Bioinformatics 14: Schäffer, A.A. et al. (1999) Bioinformatics 15: Mott, R. (2000) J. Mol. Biol. 300: Statistics of ROC n resampling Schäffer, A.A. et al. (2001) Nucl. Acids Res. 29: Spouge, J.L. & Czabarka, E. (2002) ISMB Poster 133A

Acknowledgements Analysis of ROC n score distribution John Spouge Eva Czabarka Improvements to PSI- BLAST Alejandro Schäffer L. Aravind Thomas Madden Sergei Shavirin John Spouge Yuri Wolf Eugene Koonin