NGS data transmission, A point view from a user LU Gang Chinese Human Genome Center of Shanghai 2010/10/7

outline Data challenge in NGS era Experience in submission of Schistosoma japonicum project. Peer-to-peer network Conclusion

Challenge in NGS era Massive data generation Solexa GAIIX 100G Solexa HiSeq 2000 200~300G SOLiD4 100G SOLiD4 HQ 300G *More than 1500 NGS sequencer installed. Huge volume rawdata for genome project

Experience on Schistosoma japonicum(Sj) project Summary of Sj draft project Asian blood fluke Whole genome short gun, ~4 million capillary reads ~397M genome size Nature 460, 345-351 (16 July 2009) | doi:10.1038/nature08140; The Schistosoma japonicum genome reveals features of host–parasite interplay The Schistosoma japonicum Genome Sequencing and Functional Analysis Consortium

Data submitted No raw reads 95,265 contigs, 25,048 scaffolds 13,469 protein coding genes No raw reads

It’s a pleasant journey to make the submission to EBI on this project. However several bottlenecks exists. Site bandwidth limit from userside Bandwidth utilization Data Transfer protocol

P2P, New way to transfer data? Peer-to-peer (P2P) computing or networking is a distributed application architecture that partitions tasks or work loads between peers. Peers are equally privileged, equipotent participants in the application. They are said to form a peer-to-peer network of nodes. Peers make a portion of their resources, such as processing power, disk storage or network bandwidth, directly available to other network participants, without the need for central coordination by servers or stable hosts. Peers are both suppliers and consumers of resources, in contrast to the traditional Client/Server model where only servers supply, and clients consume.

If P2P, Bandwidth limit – different route A user maybe don’t have fast connection to international center like NCBI,EBI, but maybe he have a neighbor on network who has faster link to Europe. Bandwidth utilization – much higher According to personal experience, FTP transfer speed normally won’t be higher than 200kB, most time around 100kB, But for P2P download, if the file is hot, most time it is on full speed of the bandwidth Data Transfer protocol – advanced data compression algorithms can be applied to shorten transfering time.

We need Two-way BitTorrent Chop into pieces Directly send to destination or ask a neighbor next to you in network to send it for you Collect all data trunk at receiver side, unpack and assemble them into original files.

Network structure Similar to skype, KaZaA, Normal node (normal user) Super node (any node with good connection and computing resources Cluster (super node and normal nodes close to it make a cluster) network over internet. (cluster to cluster, make the whole network)

User experience will be A single transfer program, can packing and encrypt multi data files at a time, easy to use User can choose how much bandwidth it will used, flexible. The program runs background automatically as a services, comfortable.

Ranking board The more people participated in this network, the total data transfer speed will be faster. To encourage more people to use it, a ranking board can be set. People helps more in uploading data can get better priority in further data exchange. Uploading raise the rank level.

Conclusion Improve hard links speed P2P can help improve aggregated bandwidth Build a community network to share the data, include upload and download

Thanks