CCGrid 2014 Panel: Architect Cloud and HPC for Big Data Era Rajkumar Buyya Cloud Computing and Distributed Systems (CLOUDS) Lab Dept. of Computer Science and Software Engineering The University of Melbourne, Australia Major Sponsors/Supporters

2 Panel Questions 1. Is Big Data the same as data intensive computing? Do they bring in the same technical challenges from the computing point of view? 2. Can we “ignore” the Big Data problem? If not, what are the challenges which the Cloud and HPC community must address? 3. In your opinion, what are the solutions or possible solutions of these challenges? 4. In your opinion, what will be the programming model/environment or models/environments of next generation of Cloud and HPC? 5. Will Cloud and HPC go different way to provide services for different users and applications? 6. Will HPC and data-center become more and more the same or will become more and more apart?

3 About Me – Prof. Raj Buyya Future Fellow of the Australian Research Council University of Melbourne. Director of CLOUDS Melbourne Led development of some popular software – CloudSim, Aneka, InterCloud Brokers, GridSim, Workflow Engine– used in 40+ countries. Founder CEO of Manjrasoft Pty Ltd Editor-In-Chief of IEEE Transactions on Cloud Computing (TCC)

4 Lab: Melbourne Cloud Computing Initiative Market-Oriented Clouds SLA-based Resource Management Global Cloud Exchange Aneka –.NET-based Cloud Application Platform PaaS for Enterprise and Public Clouds InterCloud - Scaling Across Clouds (Meta Brokering) Federation of clouds for application scaling and reliability 3 rd Party Cloud Services (e.g., MetaCDN) Content Delivery Networks using different “vendors” Storage Clouds Workflow Engine for Cloud Computing Scheduling applications with multiple interlinked tasks and dependencies Green Clouds / Data Centers Energy Efficient and QoS Oriented Resource Allocation CloudSim: Toolkit for Simulation of Clouds Evaluation of resource management policies & algorithms IoT (Internet of Things) for Smart Cities Big Data Environment for Disaster Management Apps

5 1. Is Big Data the same as data intensive computing? Do they bring in the same technical challenges from the computing point of view? Data-intensive computing Associated with “massive” scientific data management and processing. Solving “Big science” problems. Conducing “Blue sky” research. Sources: Instruments LHC/Telescopes Big Data Associated with supporting enterprises and governments in gaining greater insights into people's behaviors and sentiments. “targeted” services or policies: RoI/satisfaction “Ethical challenges” Tracking people. Sources: CCTV

6 2.Can we “ignore” the Big Data problem? If not, what are the challenges which the Cloud and HPC community must address? No. If we “ignore” it, we will be “ignored”! We need to be relevant and be in business High-performance messaging Offer low-latency and high bandwidth substrate for BigData Large-scale data management Reliability, performance, and policy driven data replication and access management. Large-scale computations E.g., multi-site computing QoS-based Resource Management (T, $, accuracy, …) Heavy Vs. light-weight workload/operations Harness Big Data investment/paradigm for HPC/Clouds Unification of HPC/BigData  scientific and business paradigm

7 3. What are the solutions or possible solutions of these challenges? Cloud/Big Data Platforms Datacenters Clusters Desktop PCs Public Clouds (High Performance) Communication Infrastructure HPC Application Platform Application Development Unified Programming Models QoS (Time, Budget, Accuracy, Secrecy)

8 4. What will be the programming models/environments of next generation of Cloud and HPC? Aneka Container Core Services Aneka Container Core Services Aneka Container Core Services Aneka Container Core Services Aneka Container Core Services Aneka Container Core Services ……. Extensive Framework for implementation of Multiple Programming Models

9 5. Will Cloud and HPC go different way to provide services for different users and applications? HPC Tend to be dominated by scientific apps Massive scale computing Cloud Consumer apps Enterprise… BigData Good to have some “unified” programming and execution environment to benefit from “Clouds” economy of scale.

10 Aneka: Cloud Application Platform (CAP) for Resource-Intensive/Elastic Apps Multiple Infrastructures Multi-coreClusterGridCloud ThreadTask...MapReduce 2100 Aneka Multiple Applications 1. SDK 2. Runtime World-first platform supporting multiple Cloud programming models (Task, Thread, MapReduce) SDK (Software Development Kit) containing APIs for multiple programming models and tools Runtime Environment for managing application execution on Clouds Suitable for Development of Enterprise Cloud Applications Cloud enabling legacy applications Portability for Customer Apps: Enterprise ↔ Public Clouds.NET/Win ↔ Mono/Linux

11 Aneka: The Cloud Application Platform (CAP) for Resource-Intensive Apps (Available as Manjrasoft Product) Patent (USA) World-first platform supporting multiple Cloud programming models (Task, Thread, MapReduce) SDK (Software Development Kit) containing APIs for multiple programming models and tools Runtime Environment for managing application execution on Clouds Suitable for Development of Enterprise Cloud Applications Cloud enabling legacy applications Portability for Customer Apps: Enterprise ↔ Public Clouds.NET/Win ↔ Mono/Linux

12 Cloud-centric IoT Framework

13 6. Will HPC and data-center become more and more the same or more and more apart? Depends… Mid and low-end HPC systems will look more and more like Cloud Data Centers Extreme HPC (like those in Top 10 in Top500) Will be more and more apart from Data Centers Continues to be funded by Govt… Focuses on “few” “scientific” applications.

14 Thanks for your attention! Are there any Questions? Comments/Suggestions We welcome you to: Study/Research with Us | Do Business with us! | |