CI Updates and Planning Discussion Dan Stanzione

CI Portfolio Technology Evaluation Projects Software Systems Prototyping/Design Activities Production Software Infrastructure and Services

First Release DE’s

First Release First IPTOL Release this March Some science, but largely the first release of many aspects of the new Architecture, e.g. First release of: The component model The authentication framework Provenance tracking Remote execution on Teragrid and other large systems Integration of existing bioinformatics tool as components

Architecture Evolution Architecture has evolved rapidly in direct response to working group discussions Move away from static workflows for G2P, focus instead on a dynamic, visual programming model (implications for release). Support a more “exploratory” mode Accelerate release of API, as this will be a bigger key than even I anticipated. Let me come back to this after describing a little more

Technology Eval Activities This is where we are “experimenting” with things that might work and are relevant. A couple of examples Exploring alternate implementations of QTL mapping algorithms Experimental Reproducability Also some more hardware oriented:

Experimental Systems We are experimenting with some newer technologies to plug gaps in the existing lineup for demonstrated needs (also leveraging some other funding) New model for shared memory (ScaleMP cluster to be deployed this quarter) Will support 100s of GB or RAM for *existing* codes. “Cloud Storage” models to reduce archive cost, increase capacity (HDFS system on commodity cluster to be deployed this quarter) Will also support Hadoop data processing

Prototyping/Design As some of you have seen, a number of our staff are engaged in prototyping as part of the design effort. Examples: Bernice, Greg, prototyping potential workflows with VisTrails Liya building prototype reference implementation with GLM

Systems and Services Put the infrastructure back in Cyberinfrastructure We aren’t just releasing application tools Provide access to large scale systems, storage, service hosting. Make resources available through the “web applications” as discussed yesterday

Existing Systems We have made resources available to iPlant users from a number of TeraGrid and local systems (and applied to TG for a larger allocation): Ranger (TG/Large Scale Supercomputer) Stampede (TACC/High Throughput) Eucalyptus/VM System (UA, “Cloud”) Longhorn (TG/Remote Visualization and GPU)

Storage Services We have also begun offering storage to a number of projects connected to the grand challenges in some way, as well as iPlant internal. IRODS interface Corral at TACC, a local storage array at UA

What the Architecture Looks Like Now The Discovery Environments are still a place to explore datasets, as always. Users will come to the web site, and will see: Toolkits from which they can select components and build their own workflows. Opportunities to use “gold standard” workflows, tested workflows of others. Resources will come from remote computing, increasing scale. Some may be simple: Select dataset, run massive GLM run, send me the data when it’s done More infrastructure than application

Near Term While finishing IPTOL release, continue developing component model and API Starting now, based on WG summary, begin building components described in NGS documents. Target a 2nd quarter release of the basic visual programming model with a “toolbox” of NextGen Seq tools Rapid release of tools thereafter, with occasional “infrastructure” updates

Near Term Start releasing “gold standard” or “approved” workflows once the tools are out (e.g. save and share “Ruth’s workflow”). Begin training in how others can integrate 3rd party tools in infrastructure this summer. Keep releasing tools/components as specified by working groups (will build schedule based on the WG final docs).

Next Steps WG progress looks fantastic… Priorities getting defined, iPlant people covering the group have a good idea of where to go next Some groups ahead of others, but all heading that way. Perhaps time to move to a new “mode”. Finalize docs, declare first version “done”. Move into either “beta test” mode as things come back (maybe after a brief meeting hiatus) Or, start on “version 2”, the next wishlist.

Discussion

Staffing 16 staff working on supporting working groups, prototyping, enhancing existing codes 8 staff at UA and 2 at TACC in production software, integration 3 staff in systems support 3 in project management, coordinating roles 9 faculty in CI advisory/Tech Eval Roles Roughly 10 students/postdocs supporting eval projects