1. Introduction to iPlant Dan Stanzione dan@iplantcollaborative.org The iPlant Collaborative September 16th, 2013

2. The iPlant Collaborative Cyberinfrastructure for the Plant Sciences

3. “BGI, based in China, is the world’s largest genomics research institute, with 167 DNA sequencers producing the equivalent of 2,000 human genomes a day. BGI churns out so much data that it often cannot transmit its results to clients or collaborators over the Internet or other communications lines because that would take weeks. Instead, it sends computer disks containing the data, via FedEx.” The Problem of Big Data in Biology

4. Human Genome: $2.7 Billion, 13 Years Human Genome: $900, 6 Hours 2012: Oxford Nanopore MiniION 2003: ABI 3730 Sequencer The Problem of Big Data in Biology A decade’s progress

5. The Problem of Big Data in Biology

7. Data-intensive biology will mean getting biologists comfortable with new technology…

8. 1973 Sharp, Sambrook, Sugden Gel Electrophoresis Chamber, $250 1958 Matt Meselson & Ultracentrifuge, $500,000 The Problem of Big Data in Biology hopefully comfortable enough to minimize the technology and focus on the biology.

9. What is iPlant? The iPlant Collaborative is a community-driven organization building cyberinfrastructure for the plant (and animal) sciences.

10. What is Cyberinfrastructure? Cyberinfrastructure is the coordinated aggregate of software, hardware and other technologies, as well as human expertise, required to support current and future discoveries in science and engineering. --Fran Berman Cyberinfrastructure consists of computing systems, data storage systems, instruments and data repositories, visualization environments, and people, linked together by software and networks to improve research productivity and enable breakthroughs not otherwise possible. --Craig Stewart At iPlant, we make computation,data storage, cloud services, and software tools easily available to informaticians and researchers, leveraging existing CI investments. The Stampede System at the Texas Advanced Computing Center

11. The iPlant Community Over 9,000 researchers now access iPlant services or data, in diverse areas from ecology to epigenomics In my collaboration with Nathan, the tasks they used to call me to do as the “bioinformatics expert”, the students now do on their own through the Discovery Environment -U. of Minnesota “What my users used to call me for, they now do on their own through Atmosphere. Now I can scale up my user community” -U. of Wisconsin “The resources available change your research landscape… the amounts and types of analyses that you do.” Haibao Tang, J. Craig Venter Institute

12. Scientific Achievements through iPlant’s Open Infrastructure In conjunction with the BIEN project, generating Range maps for thousands of species (can compute in a few hours) The 1KP project has stored tens of millions of sequence reads with iPlant; have a richer catalog of plant data with iPlant than NCBI; 2.6 m hours of BLAST to annotate In conjunction with USDA and researchers at Iowa State, pipelines in place which can re-sequence cattle/buffalo at 3 hours per animal. Speedups of thousands on certain GWAS & network comparison problems Integration of AgMIP agriculture models and data

13. Twig to Genome 13

14. Brief History of iPlant  Start of software development – Sep 2009  First prototypes delivered to public – Apr 2010  Discovery Environment release with user-driven tool integration – Jul 2011  Public launch of Atmosphere, “Powered by iPlant” – PAG 2012

15. Design Principles (this slide first shown in March, 2009) Ad hoc, reconfigurable analysis workflows Community-led authorship of new components Re-use of existing codebases Computational scalability Enable sharing and collaboration Presentation of well-designed APIs Attractive visualization of results Facile data integration 8

16. The iPlant Collaborative Cyberinfrastructure for the Plant Sciences The iPlant CI is designed as infrastructure. This means it is a platform upon which other projects can build. Use of the iPlant infrastructure can take one of several forms: Storage Computation Hosting Web Services Scalability

17. iPlant Today Today, iPlant provides a robust, maturing (and still growing) CI for thousands of users, leveraging other large scale NSF investments in hardware, software, and networking. While iPlant has many applications in many forms, the core CI Components are the: – Discovery Environment – Atmosphere “Cloud” Platform – Application Programmer Interfaces – Data Store – Foundation of Computation provided by NSF XSEDE iPlant can be used to run large scale, complex workflows using distributed, international resources. The rest of the speakers will show the novel ways iPlant is used to enable science!

18. iPlant Growth Over Time Grand Challenge team conferences, whitepapers and requirements gathering 778 registered users (collaborative tools) 93 users of DE 2200+ registered users (iPlant CI) 8800+ registered users >400 TB storage with growth rate of 1 TB/day 2012 2009201020112008 192 registered users (collaboration tools) Now ~12,000

19. For a challenge as broad as “plant science,” focus on specific applications/tools is a moving target, and never enough. Most important to build a *platform* that can support diverse and constantly evolving needs. “Cyberinfrastructure” is, in fact, infrastructure. The platform can lift all the apps, not select winners and losers. “The useful lifetime of our analysis tool chains is now 6 months” -Matthew Trunnel, Broad Institute The iPlant Collaborative Cyberinfrastructure for the Plant Sciences

20. We have designed iPlant to be consistent with the pillars of CIF21 High Performance Computing Data and Data Analysis Virtual Organization Learning and Workforce The iPlant Collaborative Cyberinfrastructure Philosophy

21. End Users Computational Users Teragrid XSEDE The iPlant Collaborative Cyberinfrastructure for the Plant Sciences

22. The iPlant Collaborative Ways to access iPlant Atmosphere: For virtual hosting of web apps, sites, databases. iPlant Data Storage: All data large and small The Discovery Environment: Integrated Web apps. DNASubway: Annotation and more Standalone Apps: TNRS, TreeViewer, PhytoBisque, etc The API: For programmers embedding iPlant CI capabilities Command line for experts (thru TeraGrid/XSEDE) We’ll cover each of these in more detail throughout the workshop.

23. The iPlant Collaborative Practical Benefits Powerful computational resources (Data analysis and storage) Experimental verifiability, reproducibility, provenance Interconnected resources / multiple levels of access Facilitation of collaboration Scalability/extensibility

24. About 500,000 Compute Cores available in a variety of platforms. Up to 1TB shared memory TACC Ranger PSC Blacklight TACC Corral EBI Web Services TACC Lonestar The iPlant Collaborative Scalable Computation for High Throughput Inquiry

25. Chris Pires, U. of Missouri – Assembly of Brassica Genomes on shared memory systems Haibo Tang, JCVI “ The resources available change your research landscape –the amounts and types of analyses that you do.” The iPlant Collaborative Scalable Computation for High Throughput Inquiry

26. A rich web client – Consistent interface to bioinformatics tools – Portal for users who won’t want to interact with lower level infrastructure An integrated, extensible system of applications and services – Additional intelligence above low level APIs – Provenance, Collaboration, etc. The iPlant Collaborative iPlant Discovery Environment

27. DE Growth Extensibility has meant an exponential curve as platform matures. Major DE Releases in last year: – January 2011: Extensible by programmers – July 2011: Extensible by Users – January 2012: Extensible workflow, too. 27 Today ~400 tools and counting!

28. API-compatible implementation of Amazon EC2/S3 interfaces Virtualize the execution environment for applications and services Up to 12 core / 48 GB instances Access to Cloud Storage + EBS Run servers, CloudBurst desktop use cases. Big data and the desktop are co- local again! >60 hosted applications in Atmosphere today, including users from USDA, Forest Service, database providers, etc. (30 more for postdocs and grad students for training classes) The iPlant Collaborative Project Atmosphere™: Custom Cloud Computing

29. Atmosphere – Exemplar Users Nathan Miller, Wisconsin “What my users used to call me for, they now do on their own through Atmosphere. Now I can scale up my user community” – Nathan Miller 29

30. Fast data transfers via parallel, non-TCP file transfer Move large (>2 GB) files with ease Multiple, consistent access modes iPlant API iPlant web apps Desktop mount (FUSE/DAV) Java applet (iDrop) Command line Fine-grained ACL permissions Sharing made simple Access and a storage allocation is automatic with your iPlant account The iPlant Collaborative Data Store

31. The iPlant API The API is the Application Programmer Interface. This is the way bioinformatics tools and data get integrated with iPlant. The API is out there now. Avoiding the cardinal sin of API support: – Release lots of versions, each incompatible with the last. – Our approach: Incremental releases; each release will add new areas of functionality, not change old syntax. 2011 Usage: 96,000 true hits, 1,000 large jobs, 37 apps and 3 XSEDE sites supported. 31

32. API Exemplar Users Carol Lushbough, Bioextract BioExtract, a DBI funded portal, is currently being re- written to take advantage of the API. Carol is running jobs on Lonestar through the API. 32 The API has also been adopted outside plants, by Apache Airavata, CyberGIS, and some XSEDE sites.

33. A number of other applications are “Powered by iPlant” but developed by our team on top of the infrastructure. In response to specific grand challenge team requests for things that needed their own web presence. TNRS, My-Plant, and more. The iPlant Collaborative

34. “Powered by iPlant” is the moniker for a variety of ways of using the iPlant infrastructure underneath another application that communicates with users; usually outside the iPlant project. Other major projects have adtoped the iPlant CI as their underlying infrastructure (some completely, some in limited ways – more on this later). The iPlant Collaborative

35. iPlant APIs Resources

36. iPlant Advanced Collaborative Support Based on the old TeraGrid AUS and XSEDE Extended Collaborative Support Provide a computing expert for an extended period of time to rebuild a popular tool for scalability, or other key functionality. Could be scaling, infovis, or just information architecture help.

37. Future of iPlant NSF invited iPlant to submit a renewal proposal in the summer of 2012. The proposal was submitted in September, and received a site visit (December 2012). Approved by the National Science Board at the May, 2013 meeting. A renewal may be formally announced, hypothetically, when a certain agency gets the press release out the door.

38. UA TACC CSHL The iPlant Collaborative A virtual organization

39. Staff: Greg Abram Sonali Aditya Roger Barthelson Brad Boyle Todd Bryan Gordon Burleigh John Cazes Mike Conway Karen Cranston Rion Doodey Andy Edmonds Dmitry Fedorov Michael Gatto Utkarsh Gaur Cornel Ghiban Michael Gonzales Hariolf Häfele Matthew Hanlon MetadataDataToolsWorkflowsViz Executive Team: Steve Goff Dan Stanzione Faculty Advisors & Collaborators: Ali Akoglu Greg Andrews Kobus Barnard Sue Brown Thomas Brutnell Michael Donoghue Casey Dunn Brian Enquist Damian Gessler Ruth Grene John Hartman Matthew Hudson Dan Kliebenstein Jim Leebens-Mack David Lowenthal Robert Martienssen Students: Peter Bailey Jeremy Beaulieu Devi Bhattacharya Storme Briscoe Ya-Di Chen John Donoghue Steven Gregory Yekatarina Khartianova Monica Lent Amgad Madkour B.S. Manjunath Nirav Merchant David Neale Brian O’Meara Sudha Ram David Salt Mark Schildhauer Doug Soltis Pam Soltis Edgar Spalding Alexis Stamatakis Ann Stapleton Lincoln Stein Val Tannen Todd Vision Doreen Ware Steve Welch Mark Westneat Andrew Lenards Zhenyuan Lu Eric Lyons Naim Matasci Sheldon McKay Robert McLay Angel Mercer Dave Micklos Nathan Miller Steve Mock Martha Narro Praveen Nuthulapati Shannon Oliver Shiran Pasternak William Peil Titus Purdin J.A. Raygoza Garay Dennis Roberts Jerry Schneider Anthony Heath Barbara Heath Matthew Helmke Natalie Henriques Uwe Hilgert Nicole Hopkins Eun-Sook Jeong Logan Johnson Chris Jordan B.D. Kim Kathleen Kennedy Mohammed Khalfan Seung-jin Kim Lars Koersterk Sangeeta Kuchimanchi Kristian Kvilekval Aruna Lakshmanan Sue Lauter Tina Lee Bruce Schumaker Sriramu Singaram Edwin Skidmore Brandon Smith Mary Margaret Sprinkle Sriram Srinivasan Josh Stein Lisa Stillwell Kris Urie Peter Van Buren Hans Vasquez-Gross Matthew Vaughn Fusheng Wei Jason Williams John Wregglesworth Weijia Xu Jill Yarmchuk Aniruddha Marathe Kurt Michaels Dhanesh Prasad Andrew Predoehl Jose Salcedo Shalini Sasidharan Gregory Striemer Jason Vandeventer Kuan Yang Postdocs: Barbara Banbury Jamie Estill Bindu Joseph Christos Noutsos Brad Ruhfel Stephen A. Smith Chunlao Tang Lin Wang Liya Wang Norman Wickett The iPlant Collaborative

40. iPlant Collaborative Data Mining with iPlant iPlant Collaborative Data Mining with iPlant Workshop Goals Demonstrate some of the ways iPlant CI can advance your science Familiarize you with iPlant tools and services Jumpstart your project’s involvement with iPlant through a direct consultation Helping you add your “voice” to the iPlant user community

41. Thanks! Thank you for taking the time to participate in this workshop; we hope to make it productive, and for those staying for tomorrow and the next day, increasingly hands-on. Questions?