SEQC Pipeline Comparison Using SEQC data to create performance metrics for RNA Quantification pipelines.

Overview Used only 25 million reads per replicate 4 Samples, with 4 technical replicates (labeldd A1,A2 through D3,D4) – UHR (Sample A), Brain (Sample B) – Two mixes of UHR & Brain (C & D), where C is 75% A and D is 25% A Detection is arbitratily defined as FPKM>.15 “Titration” results should therefore be in order… – A>C>D>B – A<C<D<B – Titration numbers are reported as #genes titrated/#genes detected For all analyses, reads were clipped & filtered using the fastq-mcf tool (default settings) prior to analysis Summary tool is available to be run on an SEQC FPKM matrix at

Pipelines Used tophat2 & tophat1.4 – with the –G parameter, so it uses the same GTF file as other methods – Tophat1 & 2 were both filtered for “FAIL” entries. – Tophat1 & 2 were roughly equal, tophat1 only is shown in some graphs because it was very slightly better on all metrics – Bias correction algorithm xprs –using BWA alignment & the express quantifier – eXpress isoforms were filtered using the “F” flag. When this filtering was turned off we generated a set labeled“xprsnf” – Expectation maximization algorithm rsem – version 1.2.0, default parameters – Expectation maximization algorithm rsembwa – version 1.2.0, using the same BWA alignments as xprs refq – using the same BWA alignments & counting genes & isoforms that are weighted towards unambiguous alignments (1/sq(ambig)). This is used as a “naive” quantifier.

Titration & Detection Results From the refq results, it’s clear that consistent, but possibly false, results can manifest as be “high detection and high titration”. Therefore titration and detection are not solely good indicators of a quality quantifier. Good CVS as can be faked (as in the refq results) by a flat isoform distribution. These measures must be coupled with a measure of false-positives (qPCR). BWA improves the repeatability of RSEM Tophat’s gene repeatability is good, but isoform repeatability is best in the EM estimators Tophat1 & naïve counting had the best gene-level “titration+detection” RSEM/Xprs has the best isoform-level titration and repeatability, with the exception of the known-false naïve reference counting titration detection titration detection

PCR – Slope & Correlation (one value per replicate-sample) rsembwa and eXpress with bwa had the highest correlations, the only tools that beat naïve quantification for median correlation. rsem -> rsembwa shows a large improvement in correlation, with some loss in slope tophat’s results, while good in titration, were poor when compared with PCR, better than naïve on slope, but not correlation rsembwa/xPress perform similarly (higher correlation, lower slope) when eXpress is unfiltered tophat/cufflinks’s performance was not rescued when unfiltered (shown as th1nf)

RSEM has stratification of very low FPKM values, because of rounding. Above a minimum mean value of 1, RSEM performed as well Tophat1 performed very well according to this metric. As a control, the naïve plots also performed very well, since they fail to distinguish isoforms well Pipeline Stability – Bland Altman

xPresstophat1rsemBWA When filtering matrices for “F” or “FALSE” out of xPress and tophat, we improved titration, but reduced detection compared to naïve, resulting in zeroes tophat had a lower correlation than the EM methods Simple, naïve counting had a comparable correlation, but a flatter Deming slope than other methods refq (naïve) rsem Isoform PCR - Log Log Plots (B1)

xPressNFth1nfrsemBWA These plots have “very low count” genes filtered out, ro prevent “dividing by near zero” in the fold-change comparisons. These outliers will be tracked separately. naïve counting results in better correlated log ratios than other methods, this seems to indicate that the qPCR used may not be very isoform specific Xpress, rsem and tophat/cufflinks performed similarly refq (naïve)rsem Isoform PCR - Log A-B Plots (Rep 1)

xPressNFth1nfrsemBWA Since C & D are mixtures, threfore closer, it was harder to match PCR on all methods The range of signal differences in PCR should be 50% of A/B. Xpress, naïve and rsembwa matched that expectation Again, naïve counting results in better correlated log ratios, and a poor slope Again, unfiltered xpress had the best slope, and bwa xpress and rsem had the best log ratio correlations refq (naïve)rsem Isoform PCR - Log C-D Plots (Rep 1) Undetected by other methods, possibly isoform ambiguity issue?

Metric Summary Although no one method is the clear “winner” by all metrics, the BWA/EM Methods have good PCR correlation, even in the more difficult c/d test, while maintaining low variability The inclusion of a “naïve” metric (underlines) is an important control, highlighting the flaws in any one measure of quality It’s important to measure quality using both isoform-level and gene-level metrics. ISOFORM METRICSGENE METRICS iso titration pct med detection count med detection pct median cv bland- altman stdev pcr correlation pcr a/b correlation pcr c/d correlation titration pct med detection count med detection pct median cv bland- altman stdev tophat th1nf rsem rsembwa xprs xprsnf refq Q Q *top 10% are shaded, top 25% are bordered