All rights reserved. © CloudBroker GmbH Cloud Computing for Scientific and Technical Applications GridKa School 2012 August 28, 2012, Karlsruhe, Germany.

All rights reserved. © CloudBroker GmbH Cloud Computing for Scientific and Technical Applications GridKa School 2012 August 28, 2012, Karlsruhe, Germany Dr. Wibke Sudholt CloudBroker GmbH Technoparkstrasse 1, CH-8005 Zurich, Switzerland Phone: +41 44 633 79 34 Email: info@cloudbroker.com Web: http://www.cloudbroker.cominfo@cloudbroker.comhttp://www.cloudbroker.com 128.08.2012 GridKa School 2012

All rights reserved. © CloudBroker GmbH Overview Cloud computing basics Scientific computing in the cloud CloudBroker Platform Examples: –Protein modeling in the IBM Cloud for the Institute of Molecular Systems Biology (IMSB) at ETH Zurich –SCIentific gateway Based User Support (SCI- BUS) project –Further examples Demo 28.08.20122 GridKa School 2012

All rights reserved. © CloudBroker GmbH Google Trends 28.08.20124 GridKa School 2012 Source: http://www.google.com/trends/?q=cloud+computing,+grid+computing, 27.08.2012http://www.google.com/trends/?q=cloud+computing,+grid+computing grid computing cloud computing

All rights reserved. © CloudBroker GmbH The Way to the Cloud Internet Distributed computing Commodity clusters, server farms Grid computing, utility computing ASP = Application Service Provider Web services SOA = Service-Oriented Architecture Virtualization 28.08.20125 GridKa School 2012

All rights reserved. © CloudBroker GmbH Cloud Terms 28.08.2012 Utility Computing Computing on Demand Software as a Service Infrastructure as a Service Platform as a Service Multi-Tenancy Public Cloud Private Cloud Cloud Storage Pay-per-Use Elasticity 6 GridKa School 2012 Scalability Hybrid Cloud Cloud Bursting Self Service

All rights reserved. © CloudBroker GmbH Cloud Computing Definition Access to computer resources on demand without much initial investment in time or money (self service) Only pay for what you actually use in small steps (OpEx instead of CapEx) Nearly unlimited scalability (elasticity) =Change in business model =Interfaces set at the right place 28.08.20127 GridKa School 2012

All rights reserved. © CloudBroker GmbH Cloud Services 28.08.20128 GridKa School 2012 Web / office / business applications, … Salesforce, Google Apps,... SaaS = Software as a Service Development / deployment frameworks, distribution / messaging / monitoring systems, databases, … Microsoft Windows Azure, Hadoop / MapReduce,... PaaS = Platform as a Service Computing power, virtual machines, storage space, … Amazon EC2 and S3, IBM SmartCloud Enterprise,... IaaS = Infrastructure as a Service

All rights reserved. © CloudBroker GmbH Types of Clouds Private Cloud In-house or hosted Just for internal use Focus on self service and accountability Hybrid Cloud Mixture of public and private cloud(s) Focus on scalability and failover Public Cloud Offered by cloud providers Available to everybody Focus on on- demand and pay-per-use 28.08.20129 GridKa School 2012

All rights reserved. © CloudBroker GmbH Scientific Computing Scientific and technical applications (modeling, simulation, statistics, …) Mainly used in research and development (R&D) or forecast and analytics departments Usually complex algorithms and/or big data High throughput and/or high performance needs Often requiring high performance computing (HPC) resources (supercomputers, clusters, grids, clouds) Utilizing parallelization and/or distributed computing Mostly project-based or periodical usage patterns Increasing importance 28.08.201211 GridKa School 2012

All rights reserved. © CloudBroker GmbH Application Areas Particle physics Astrophysics Computational chemistry Molecular modeling Bioinformatics Drug design Material science … Engineering Insurance risk analysis Financial market calculations Weather and climate simulations Military research Image processing Rendering … 28.08.201212 GridKa School 2012

All rights reserved. © CloudBroker GmbH Problems of Traditional Scientific Computing HPC computer infrastructure, middleware tools and application software: –Require expert knowledge –Are expensive, time-consuming and complex to buy, set up, use and maintain –Are hard to integrate with existing systems and processes –Often operate at capacity limit ⇒ Hardly accessible or affordable for SMEs / small research groups, specialized application purposes or short-term projects 28.08.201213 GridKa School 2012

All rights reserved. © CloudBroker GmbH Advantages of Cloud Immediate access to infrastructure and/or applications on demand Availability of hardware and/or software resources not existing in-house Nearly unlimited scalability Possibility for spill-over / cloud bursting Temporary, non-binding utilization, no long-term contracts or licenses Self service by actual application users Pay-per-use with minimal initial investment Hardware, middleware and/or software maintained by cloud providers 28.08.201214 GridKa School 2012

All rights reserved. © CloudBroker GmbH Challenges of Cloud Infrastructure, middleware and applications remain complex to set up, use and maintain also in the cloud Dynamic features of the cloud as well as billing and accounting add to the complexity Performance limitations for some applications due to virtualization and available hardware (processors, memory, storage, network interconnects) Concerns because of security, outsourcing, internationality, service level agreements (SLAs), multi- tenancy, standardization and potential vendor lock-in Hardware and software vendors as well as application departments have to adapt to the pay-per-use and self service business model 28.08.201215 GridKa School 2012

All rights reserved. © CloudBroker GmbH Platform Solution Web-based application store for the deployment and execution of scientific and technical software in the cloud Uses IaaS from cloud providers Offers PaaS for software vendors and SaaS for end users Browser / programmatic / command-line access, cross- domain On demand, self service, pay-per-use, scalable Easy to use, speeds up time to market, no need for own HPC infrastructure or software licenses, OpEx instead of CapEx 28.08.201217 GridKa School 2012

All rights reserved. © CloudBroker GmbH Platform Architecture 28.08.201218 GridKa School 2012 CloudBroker Platform Amazon Cloud IBM Cloud … Cloud Chemistry Appli- cations Biology Appli- cations Health Appli- cations Web Browser UI … Appli- cations REST Web Service API CloudBroker Integration User Tools End Users, Software Vendors CLI Java Client Library Engineering Appli- cations

All rights reserved. © CloudBroker GmbH Platform Functionality 28.08.201219 Security Frame: Transport Layer Security, Access Rights Security IBM Cloud … Cloud Application Manager Application Manager Process Monitor Process Monitor Resource Manager Resource Manager Web Service API Web Browser UI Tools IBM Adapter IBM Adapter … Adapter Cloud Provider Access Manager Scalability and Fault Tolerance Handler User Manager User Manager Accounting Module Accounting Module Billing Module Billing Module Payment Module Payment Module Queuing System Queuing System Storage Manager Storage Manager Image Manager Image Manager Process Manager Process Manager Amazon Cloud Amazon Adapter Users GridKa School 2012

All rights reserved. © CloudBroker GmbH Job Submission Lifecycle 1.Financial or access authorization (user) 2.Software and resource selection and job creation (user) 3.Data file upload (user) to cloud storage (platform) 4.Job submission (user) 5.Compute instance startup or reuse (platform) 6.Data file upload from cloud storage to master node (platform) 7.Computations on worker nodes (platform, application) 8.Data file download from master node to cloud storage (platform) 9.Compute instance shutdown or reuse (platform) 10.Data file download (user) from cloud storage (platform) 11.Accounting and billing (platform / manager) 28.08.201220 GridKa School 2012

All rights reserved. © CloudBroker GmbH Application Requirements 28.08.201221 GridKa School 2012 ItemRequirements / possibilities TypeScientific and technical applications DomainAll LicenseOpen source, free, commercial CharacterCompute-intensive, batch-oriented InterfaceCommand line, non-interactive RuntimeHours, days, weeks Operating systemLinux ParallelizationSingle-threaded / single-core, multi-threaded / multi- core, embarrassingly parallel / high throughput, parallel across nodes (via MPI or other methods) Job typesSingle jobs, pipeline jobs, parameter sweeps (via API), workflows (via API)

All rights reserved. © CloudBroker GmbH Software Deployment Lifecycle 1.Clarification of licensing and pricing (vendor) 2.Generation of installation script (vendor) 3.Creation and configuration of software (vendor) 4.Adding of executables (vendor) 5.Adding of installation package (vendor), including installation script and software package, if needed 6.Deploying of software (vendor) 7.Checking of software and installation script (manager) 8.Generation of deployments (manager) 9.Running of special deployment jobs (platform) 10.Checking and activation of deployments (manager) 11.Activation of software (manager / vendor) 12.Publication of software (vendor) 13.Running of jobs (user) 14.Accounting and payment (platform / manager) 28.08.201222 GridKa School 2012

All rights reserved. © CloudBroker GmbH Current Applications 28.08.201223 GridKa School 2012 More applications continuously to be added Applications can also be added by users ApplicationDomainRemarks BLASTDNA and protein sequence alignmentAvailable in public platform AutoDockProtein-ligand dockingAvailable in public platform GromacsMolecular dynamics simulationsAvailable in public platform X! TandemMass spectrometry data matchingAvailable in public platform OpenFOAMComputational fluid dynamicsAvailable in public platform RosettaProtein modellingOnly with own license TransATComputational fluid dynamicsOnly with own license SiestaMaterial scienceOnly with own license T-CoffeeDNA and protein sequence alignmentAvailable upon request FoldXProtein modellingAvailable upon request RMathematics and statisticsAvailable upon request

All rights reserved. © CloudBroker GmbH Platform Clients and APIs CLI Command line interface Job submission Java Application programming interface Java client library REST Representational state transfer Web service interface 28.08.201224 GridKa School 2012

All rights reserved. © CloudBroker GmbH IMSB Project Background Collaboration with the Institute of Molecular Systems Biology (IMSB) at ETH Zurich and IBM Goal is to better understand the mechanisms of infectious diseases to fight antibiotics resistance For the example of the Streptococcus bacterium, predict the 3D structures of important proteins from their 1D sequence by means of the Rosetta software Find the important structural differences between less and more harmful bacteria strains Calculations expected to take several months on available cluster infrastructure Embarrassingly parallel, compute-intensive calculations, perfect fit for cloud computing Use the CloudBroker Platform to manage the calculations on top of IBM SmartCloud Enterprise compute and storage resources 28.08.201226 GridKa School 2012 Source: Dr. Lars Malmström, IMSB, ETH Zurich

All rights reserved. © CloudBroker GmbH IMSB Project Results 249 Streptococcus target proteins modeled using special Rosetta client for automation Up to 63 compute instances with 1008 virtual CPUs in parallel provided by the IBM SmartCloud Enterprise Number of instances in the cloud automatically adjusted to the workload by the CloudBroker Platform Optimized data transfer between ETH Zurich file server and compute and storage instances in the cloud About 36’000 single-threaded jobs created by the client, managed by the platform and computed in the cloud Almost 250’000 CPU hours utilized for the production calculations Ca. 2.3 Mio 3D protein structure models created Calculations finished within less than two weeks 28.08.201228 GridKa School 2012

All rights reserved. © CloudBroker GmbH SCI-BUS Project Overview Goal is to develop gateway technology for providing researchers seamless access to major computing, data and networking infrastructures and services Focus on scientific workflows Support for gateways from 11+ different communities: Proteomics, molecular science, seismology, business processes, medicine, rendering, astrophysics, SMEs, software testing, citizen web, heliophysics and others Supported by the EU FP7 Capacities Programme under contract no. RI-283481 15 partners from 10 different countries: 10 academic, 5 commercial 6 subcontractors Growing number of associated partners (signing of MoU) Running 10/2011-09/2014 Base technology is gUSE / WS- PGRADE from MTA SZTAKI on top of Liferay Role of CloudBroker is to provide access to commercial and open clouds and to enable commercial exploitation of gateways http://www.sci-bus.eu 28.08.201230 GridKa School 2012

All rights reserved. © CloudBroker GmbH SCI-BUS Project Results Already available: –Integration of CloudBroker Platform in gUSE / WS- PGRADE under DCI Bridge, security and job settings –Possibility to configure and run CloudBroker Platform jobs from within gUSE / WS-PGRADE workflows –Extended CloudBroker Platform REST and Java APIs –Development and testing platform for SCI-BUS In preparation: –Possibility to register own cloud resources in the CloudBroker Platform –Adapters to open cloud infrastructures (Eucalyptus, OpenStack, OpenNebula) –Commercial components in gUSE / WS-PGRADE for user mapping, billing and cost display –Extended CloudBroker Platform user guides 28.08.201232 GridKa School 2012

All rights reserved. © CloudBroker GmbH Web UI for TransAT TransAT = Transport phenomena Analysis Tool Computational multiphase fluid dynamics (CMFD) software from ASCOMP GmbH Collaboration with HSR: Bachelor thesis “Web UI-Front-End for Fluid Dynamics Cloud”, Microsoft Silverlight-based TransAT available in CloudBroker Platform with own license 28.08.201234 GridKa School 2012 Source: Anita Hollenstein and Patrice Mu ̈ ller, http://eprints.hsr.ch/193/, p. 59 http://eprints.hsr.ch/193/

All rights reserved. © CloudBroker GmbH Interfacing in KNIME KNIME = Konstanz Information Miner Open-source data integration, processing, analysis and exploration framework, supported by KNIME.com AG Computational chemistry workflow showcase: Calculation of dipole moments for a number of molecules CloudBroker Platform node available upon request 28.08.201235 GridKa School 2012

All rights reserved. © CloudBroker GmbH Public Platform Version https://platform.cloudbroker.com First preview version published in fall 2010, current version 1.0.4, version 1.1 with new SCI-BUS features upcoming Accessible for everybody after registration and user checking Currently available clouds: –Amazon EC2 and S3 (computing and storage) –IBM SmartCloud Enterprise (only computing, storage upcoming) Currently available applications: –BLAST, AutoDock, Gromacs, X! Tandem, OpenFOAM –Additional applications privately or upon request, with software checking before deployment Production version, monitored and maintained, with help desk under platform@cloudbroker.com platform@cloudbroker.com Standard pricing: Cloud resource costs + application software costs + CloudBroker Platform surcharges Also available as hosted and in-house versions Run jobs and add your own software! 28.08.201237 GridKa School 2012

All rights reserved. © CloudBroker GmbH Thanks! – Questions? Contact for more information: Dr. Wibke Sudholt CloudBroker GmbH Technoparkstrasse 1, CH-8005 Zurich, Switzerland Phone: +41 44 633 79 34 Email: wibke.sudholt@cloudbroker.com Web: http://www.cloudbroker.comwibke.sudholt@cloudbroker.comhttp://www.cloudbroker.com Try out the CloudBroker Platform yourself under https://platform.cloudbroker.com! https://platform.cloudbroker.com 28.08.201238 GridKa School 2012

All rights reserved. © CloudBroker GmbH Cloud Computing for Scientific and Technical Applications GridKa School 2012 August 28, 2012, Karlsruhe, Germany.

Similar presentations

Presentation on theme: "All rights reserved. © CloudBroker GmbH Cloud Computing for Scientific and Technical Applications GridKa School 2012 August 28, 2012, Karlsruhe, Germany."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

All rights reserved. © CloudBroker GmbH Cloud Computing for Scientific and Technical Applications GridKa School 2012 August 28, 2012, Karlsruhe, Germany.

Similar presentations

Presentation on theme: "All rights reserved. © CloudBroker GmbH Cloud Computing for Scientific and Technical Applications GridKa School 2012 August 28, 2012, Karlsruhe, Germany."— Presentation transcript:

Similar presentations

About project

Feedback