1. Next Generation Cyberinfrastructures for Next Generation Sequencing and Genome Science AAMC 2013 Information Technology in Academic Medicine Conference Vancouver, CA. June 5, 2013 William K. Barnett, Ph.D. Richard LeDuc, Ph.D. National Center for Genome Analysis Support

2. National Center for Genome Analysis Support: http://ncgas.orghttp://ncgas.org Summary The NGS Big Data Problem NCGAS as a National Model NCGAS Cyberinfrastructure The 100 Gigabit Demonstration Software Optimization Current State and Prospects

3. The Next Generation Sequencing Big Data Problem

4. Changing genomics data environment Sequencing is:  Becoming commoditized at large centers,  Multiplying at individual labs, and  Generating (MUCH) more data Analytical capability lacks:  Bioinformatics support  Computational support  Storage support

5. Sequencing is getting cheaper Source: http://www.genome.gov/images/content/cost_per_genome.jpghttp://www.genome.gov/images/content/cost_per_genome.jpg

6. 6 omicsmaps.com listed 931 Next Generation Sequencers in the US on April 29, 2013

7. NGS Data growth is outstripping Storage growth Source: Stein Genome Biology 2010 11:207 doi:10.1186/gb-2010-11-5-207

8. Source: Nancy Spinner Keynote at 2012 Internet2 meeting: http://events.internet2.edu/2012/fall- mm/agenda.cfm?go=session&id=10002624&event=1149http://events.internet2.edu/2012/fall- mm/agenda.cfm?go=session&id=10002624&event=1149 Genomics Data are Big

9. One Degree Imager: 1.4 Petabytes per year Large Hadron Collider: 15 Petabytes per year. The Open Science Grid moves 2 Petabytes per day The Square Kilometer Array:2,000 Petabytes per year in 2013 But other disciplines already handle Bigger

10. The National Center for Genome Analysis Support as a National Model

11. 1.Bioinformatics consulting for biologists 2.Large memory clusters for assembly 3.Optimized software for better efficiency Initially Funded by the National Science Foundation Collaboration across multiple institutions Open for business at: http://ncgas.orghttp://ncgas.org

12. Making it easier for Researchers Easy to use web interface Preinstalled bioinformatics tools Large memory clusters for assembly For research and education. Common Rare Computational Skills LOW HIGH

13. NCGAS Service Model Hardware Layer OS Layer Services Layer Applications Bioinformatics Network Layer Infrastructure as a Service Public Cloud Supercomputer Cntrs Professional Admins Parallel Optimized Envs. Galaxy and Tuned Apps Expert Consulting 100 Gbps Internet2 NEEDSNCGAS Platform as a Service Software as a Service Aligned with HIPAA for Clinical research

14. National Center for Genome Analysis Support: http://ncgas.orghttp://ncgas.org NCGAS Cyberinfrastructure

16. NSF-Funded or XSEDE Allocation Federally Funded NCGAS Galaxy Portal POD Galaxy Portal 5 PB D.C. 6 PB Storage 5.5 PB Storage 4 PB Storage TACC SDSC PSC Mason POD Sequencing Center NCBI 100 Gig Internet2 10 Gig NLR NCGAS Virtual Instrument IU

17. Commodity Internet (1Gbps but highly variable) Internet2 (100 Gbps) 0 100 Gpbs 50 National Lambda Rail to Sequencing Centers (10 Gbps) IU Data Capacitor (40 Gbps) Ultra SCSI 160 Disk (1.2 Gbps, 160 MBps) Research networks are now (much) faster than hard drives

18. Lots of SchleppingOne and Done

19. How does this work at scale? 1.Researchers use Galaxy to transfer files and run jobs Files transferred to Data Capacitor over Internet2 Data Capacitor mounted on several Supercomputers Data Capacitor mounts reference data from NCBI 2.Workflows execute across multiple systems 3.Results delivered through web interfaces and to visualization or other science tools

20. Rates For NSF Funded Researchers at IU or XSEDE Allocations: Consulting: $0 – currently 2.5 FTE bioinformaticians Data Transfer: $0 Data Storage: $0 – contemplating 25 TB allocation/project Computation: $0 Other Federally funded Researchers running on the POD Consulting: $60/hour for short consults Data Transfer: $20 per transfer Data Storage: $.10/GB/month Computation: $.09/core hour (128GB/node cluster)

21. The 100 Gigabit Testbed

22. NCGAS 100 Gbps Demo at SC 11 STEP 1: data pre- processing, STEP 2: Sequence alignment STEP 3: SNP detection IndianaSeattle

23. Software Optimization

24. Performance Improvement - Trinity

25. Current State and Prospects

26. What have we done so far? 1.Partnering on 37 research projects, managing 73 TB 2.IU infrastructure aligned with HIPAA 3.XSEDE Tier 2 Service Provider 4.Optimized Trinity with Broad Institute 5.LustreWAN node at U. Hawaii at Hilo Next: 1.LustreWAN node at NCBI (July 2013) 2.More Science 3.More LustreWAN nodes at sequencing sites

27. Biomedical Research Examples Identifying the causative mutation that generates ethanol sensitivity in a mutagenized zebrafish line. Johann Eberhart, PI. University of Texas at Austin. A genomewide association testing whether a particular allele at a single SNP a group of SNPs is more common in individuals with an intracranial aneurysm as compared with healthy controls. Tatiana Foroud, PI. Indiana University School of Medicine.

28. In Sum… NGS is creating data and analytical problems that cannot be solved locally NCGAS provides a national model that can support NGS NCGAS provides improved interfaces and tools to lower barriers to supercomputing

29. NCGAS Partners:

30. Questions? Bill Barnett (barnettw@iu.edu)barnettw@iu.edu Rich LeDuc (rleduc@iu.edu)rleduc@iu.edu

31. This material is based upon work supported by the National Science Foundation under Grant No. ABI-1062432, Craig Stewart, PI. William Barnett, Matthew Hahn, and Michael Lynch, co-PIs. This work was supported in part by the Lilly Endowment, Inc. and the Indiana University Pervasive Technology Institute Any opinions presented here are those of the presenter(s) and do not necessarily represent the opinions of the National Science Foundation or any other funding agencies Acknowledgements and Disclaimers

32. License Terms Please cite as: Barnett, W.K., and R.D. LeDuc, Next Generation Cyberinfrastructures for Next Generation Sequencing and Genome Science, presented at the AAMC 2013 Information Technology in Academic Medicine Conference, Vancouver, CA. http://hdl.handle.net/2022/16668http://hdl.handle.net/2022/16668 Items indicated with a © are under copyright and used here with permission. Such items may not be reused without permission from the holder of copyright except where license terms noted on a slide permit reuse. Except where otherwise noted, contents of this presentation are copyright 2011 by the Trustees of Indiana University. This document is released under the Creative Commons Attribution 3.0 Unported license (http://creativecommons.org/licenses/by/3.0/). This license includes the following terms: You are free to share – to copy, distribute and transmit the work and to remix – to adapt the work under the following conditions: attribution – you must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). For any reuse or distribution, you must make clear to others the license terms of this work.http://creativecommons.org/licenses/by/3.0/