Directions in eScience Interoperability and Science Clouds June Interoperability in Action – Standards Implementation in VENUS-C & the context of the SIENA Roadmap OGF35 at HPDC 2012 Delft Geoffrey Fox Director, Digital Science Center, Pervasive Technology Institute Associate Dean for Research and Graduate Studies, School of Informatics and Computing Indiana University Bloomington

Successes in eScience I Basic Supercomputer Architecture now being extended to Exascale – Grand Challenge activity produced consensus Basic OGF standards such as JSDL, BES, SAGA, GridFTP Software as a Service Use of Services Use of Workflow Use of Portals Say “use of” as details not agreed 2

More on Successes Appliances/Roles in Clouds (see Venus-C later) – Images defined explicitly (by construction) or implicitly by content Value added Platforms such as MPI, parallel domain specific Libraries, (Iterative) MapReduce, Queues, Tables and other NOSQL data models, Object Stores, HDFS/GFS style file systems PaaS delivered by tools/libraries/roles? Other good important general standards in security, OVF, accounting, networking 3

What Platforms to use in Clouds HDFS style file system to collocate data and computing Or Object Stores as basic scalable storage Queues to manage multiple tasks Tables to track job information MapReduce and Iterative MapReduce for parallelism Services for everything Portals as User Interface Appliances and Roles as customized images Software environments/tools like Google App Engine, memcached, Workflow to link multiple services (functions) 4

What to use in Grids and Supercomputers? Portals, Services and Workflow as in clouds MPI and GPU/multicore threaded parallelism Wonderful libraries supporting parallel linear algebra, particle evolution, partial differential equation solution Parallel I/O for high performance in an application Wide area File System (e.g. Lustre) supporting file sharing This is a rather different style of PaaS from clouds – we should unify? 5

Comments No agreement on problem to solve e.g. what is architecture for data intensive problems, role of clouds(!) Certainly no agreement on even style of workflow Services can be WSDL or REST Confusion as to architecture level being standardized – User or developer? – e.g. clouds may be built on federated infrastructure; that must be hidden from user 6

Some Standards Futures In general look for a few key SIMPLE concepts From past, SQL and MPI standardization very successful – suggesting that Cloud PaaS standards should be looked at – MapReduce – NOSQL data models Needs to be done at right time De facto standard “Hadoop” versus “real” standard What “roles” are important: Worker, Web, Grid, Worker + I/O, MPI, MapReduce, GPU – need a study? Roles v. Libraries v. Standard Interfaces GPU related standards: OpenACC extends OpenMP 7

Using Science Clouds in a Nutshell High Throughput Computing; pleasingly parallel; grid applications Multiple users (long tail of science) and usages (parameter searches) Internet of Things (Sensor nets) as in cloud support of smart phones (Iterative) MapReduce including “most” data analysis Exploiting elasticity and platforms (HDFS, Object Stores, Queues..) Use worker roles, services, portals (gateways) and workflow Good Strategies: – Build the application as a service; – Build on existing cloud deployments/roles such as Hadoop; – Use PaaS if possible; (This is not clearly eScience strategy – uses IaaS?) – Design for failure; (Not much work on what this means. Are there tools?) – Use as a Service (e.g. SQLaaS) where possible; (WHAT should be Provided) – Address Challenge of Moving Data (Need Production large scale Science Cloud) 8

Cosmic Comments Does Cloud + MPI Engine cover the future? – Will current High throughput computing and cloud concepts merge? Need Data analytics libraries for HPC and Clouds Does a “modest-size private science cloud” make sense – Too small to be elastic Should governments fund use of commercial clouds (or build their own) – Most science doesn’t have privacy issues motivating some private clouds Most interest in clouds from “new” applications such as life sciences Recent cloud infrastructure (Eucalyptus 3, OpenStack Essex) much improved More employment opportunities in clouds than HPC and Grids; so cloud related activities popular with students 9