Preprocessing Data Rob Schmieder

Bad data analysis

Good data analysis New dataset Quality control & Preprocessing Similarity search Assembly

3 Tools for metagenomic data http://prinseq.sourceforge.net http://tagcleaner.sourceforge.net http://deconseq.sourceforge.net

http://edwards.sdsu.edu/prinseq Quality control and data preprocessing Rob Schmieder

Number and length of sequences Bad Reads should be approx. same length (same number of cycles) → Short reads are likely lower quality Good

Linearly degrading quality across the read Trim low quality ends

High quality throughout the sequence Good quality through the length of the sequence Sequence quality falls off quickly → Bad sequence data

Ion quality scores

Low quality sequence issues Most assemblers or aligners do not take into account quality scores Errors in reads complicate assembly, might cause misassembly, or make assembly impossible

What if quality scores are not available ? Alternative: Infer quality from the percent of Ns found in the sequence Removes regions with a high number of Ns Huse et al. found that presence of any ambiguous base calls was a sign for overall poor sequence quality Huse et al.: Accuracy and quality of massively parallel DNA pyrosequencing. Genome Biology (2007)

What if quality scores are not available ? Alternative: Infer quality from the percent of Ns found in the sequence Removes regions with a high number of Ns Huse et al. found that presence of any ambiguous base calls was a sign for overall poor sequence quality Huse et al.: Accuracy and quality of massively parallel DNA pyrosequencing. Genome Biology (2007)

Ambiguous bases If you can afford the loss, filter out all reads containing Ns Assemblers (e.g. Velvet) and aligners (SHAHA2, BWA, …) use 2-bit encoding system for nucleotides some replace Ns with random base, some with fixed base (e.g. SHAHA2 & Velvet = A) 2-bit example: 00 – A, 01 – C, 10 – G, 11 - T

Quality filtering Any region with homopolymer will tend to have a lower quality score Huseet al. found that sequences with an average score below 25 had more errors than those with higher averages Huse et al.: Accuracy and quality of massively parallel DNA pyrosequencing. Genome Biology (2007)

Sequence duplicates

Real or artificial duplicate ? Metagenomics = random sampling of genomic material Why do reads start at the same position? Why do these reads have the same errors? No specific pattern or location on sequencing plate 11-35% Gomez-Alvarez et al.: Systematic artifacts in metagenomes from complex microbial communities. ISME (2009) 16

One micro-reactor – Many beads Martine Yerle (Laboratory of Cellular Genetics, INRA, France)

Impacts of duplicates False variant (SNP) calling Require more computing resources Find similar database sequences for same query sequence Assembly process takes longer Increase in memory requirements Abundance or expression measures can be wrong

Impacts of duplicates False variant (SNP) calling Require more computing resources Find similar database sequences for same query sequence Assembly process takes longer Increase in memory requirements Abundance or expression measures can be wrong Reference ...ACCACACGTGTTGTGTACATGAACACAGTATATGAGCATACAGAT... GTGTTGTGTACATGAACACAGTATATGAGCATACAGAT... GTGTACATGAACACAGTATATGAGCATACAGAT... TGAACACAGTCTATGAGCATACAGAT... TGAACACAGTCTATGAGCATACAGAT... TGAACACAGTCTATGAGCATACAGAT... TGAACACAGTCTATGAGCATACAGAT... TGAACACAGTCTATGAGCATACAGAT...

Impacts of duplicates False variant (SNP) calling Require more computing resources Find similar database sequences for same query sequence Assembly process takes longer Increase in memory requirements Abundance or expression measures can be wrong

Detect and remove tag sequences http://edwards.sdsu.edu/tagcleaner

No tag MID tag WTA tags

Imperfect primer annealing

Fragment-to-fragment concatenations

Data upload Tag sequence definition

Tag sequence prediction

Parameter definition Download results

Identification and removal of sequence contamination http://edwards.sdsu.edu/deconseq

Contaminant identification Previous methods had critical limitations Dinucleotide relative abundance uses information content in sequences  can not identify single contaminant sequences Sequence similarity seems to be only reliable option to identify single contaminant sequences BLAST against human reference genome is slow and lacks corresponding regions (gaps, variants, …) Novel sequences in every new human genome sequenced* * Li et al.: Building the sequence map of the human pan-genome. Nature Biotechnology (2010)

Faster algorithms for Next-gen data

Principal component analysis (PCA) of dinucleotide relative abundance Microbial metagenomes Viral metagenomes

Contaminant identification Current methods have critical limitations Dinucleotide relative abundance uses information content in sequences  can not identify single contaminant sequences Sequence similarity seems to be only reliable option to identify single contaminant sequences BLAST against human reference genome is slow and lacks corresponding regions (gaps, variants, …) Novel sequences in every new human genome sequenced* * Li et al.: Building the sequence map of the human pan-genome. Nature Biotechnology (2010)

DeconSeq web interface Two types of reference databases Remove Retain

DeconSeq web interface (cont.)

DeconSeq Identity = How similar is the query sequence to the reference sequence How much of query sequence is similar to reference sequence Coverage =

DeconSeq Blue = More similar to “retain” Red = More similar to “remove”

Human DNA contamination identified in 145 out of 202 metagenomes

http://prinseq.sourceforge.net/manual.html