1. Luis Russi¹, Carlos R. Senna¹, Edmundo R. M. Madeira¹, Xuan Liu², Shuai Zhao², and Deep Medhi² Hadoop-in-a-Hybrid-Cloud GEC21 The 21st GENI Engineering Conference Oct 20-23, 2014-- Bloomington, IN, USA ¹Institute of Computing, State University of Campinas – Brazil ²University of Missouri–Kansas City – USA

2. Agenda Motivation and Objectives Proposed Architecture – Web Cloud Portal – Execution Engine – Execution Service Why using GENI Testbed GEC21 2014- 2

3. Motivation and Objectives Why – Hadoop installed in a private cloud may not have sufficient resources for all types of computational requirements – Need a seamless environment where Hadoop in a private cloud can access resources in other clouds Hybird Cloud An architecture to make the orchestration of Hadoop applications in hybrid clouds – Automatic preparation of a cross-domain cluster – Provisioning files – Making the results available to the user GEC21 2014- 3

4. Cont.. Execution of Hadoop applications in hybrid cloud is not easy! – Spends time – Needs technical knowledge – Continuous evaluation of cloud resources – On-demand preparation of public cloud resources – Hybrid cloud requires an appropriate model that combines performance with minimal cost GENI platforms allows us to test out the Hadoop in a hybrid cloud concept

5. The Proposed Architecture GEC21 2014- 5 HM – Hadoop Master Node HW – Hadoop Worker Nodes

6. Web Cloud Portal GEC21 2014- 6 User interface Management of files (application, data and submission) Simple XML-Based submission file – Number of Virtual Machines (VM) – Image identification (Hadoop Master and Workers) – Requirements of VMs (memory, disk, flavor, etc) Organizing the application workspace

7. Orchestration Engine GEC21 2014- 7 Prepares working place in the private cloud’s storage Creates an Execution Service Instance (ESI) already associated with this cloud storage area Releases the ESI to manage the application execution (asynchronously) Copies the resulting files from the cloud storage to the user’s working place Eliminates ESI Notifies WCP

8. Execution Engine GEC21 2014- 8 ES Instance interacts with the private cloud monitoring system to evaluate the computational resources conditions Checks for extra resources from the public cloud (if needed) Automatic Hadoop Cluster preparation (Master and Workers) Makes a copy of the resulting files from the HDFS to the cloud storage accessible by the Orchestration Engine Eliminates all involved VMs Notifies the Orchestration Engine about the end of processes Monitors all stages of processing

9. Great environment for testing the Hybrid Cloud High speed networks Provisinable environments for cloud computing Public cloud deployment Cluster installation automation API integration Why using GENI?

10. UNICAMP-UMKC Hybrid Testbed GEC21 2014- 10 Word Count Java software prototype

11. Initial Results GEC21 2014- 11 Deploy exoGENI virtual machines with Hadoop 2.5.1 Include the UMKC compute node at the UNICAMP cloud controller GRE Tunnel established between UMKC and UNICAMP

12. Future Work GEC21 2014- 12  ExoGENI virtual machines and cloud Hadoop cluster joint  Execute the Wordcount Hadoop application at the cluster (http://wiki.apache.org/hadoop/WordCount)  Integrate GENI API to the private cloud framework

13. Luis Russi¹, Carlos R. Senna¹, Edmundo R. M. Madeira¹, Xuan Liu², Shuai Zhao², and Deep Medhi² luisrussi@lrc.ic.unicamp.br¹, crsenna@ic.unicamp.br¹, edmundo@ic.unicamp.br¹, xuan.liu@mail.umkc.edu², shuai.zhao@mail.umkc.edu², dmedhi@umkc.edu² ¹Institute of Computing, State University of Campinas – Brazil ²University of Missouri–Kansas City – USA Thank you! Hadoop-in-a-Hybrid-Cloud