1. Introductory Tutorial: OpenStack, Chef, Hadoop, Hbase, Pig I590 Data Science Curriculum Big Data Open Source Software and Projects September 21 2014 Geoffrey Fox gcf@indiana.edu http://www.infomall.org School of Informatics and Computing Digital Science Center Indiana University Bloomington

2. This Tutorial 1)Create our Virtual Machine Instance Using OpenStack to create a VM on FutureGrid 2)Deploy Software with DevOps Tools Using Chef to install Apache software and dependencies 3)Using the Apache Big Data Stack Hands-on tutorials to demo Hadoop plus other packages

3. FutureSystems and CloudMesh The purpose of the first part of this tutorial is simply to create one or more virtual machines using OpenStack, where we will install Hadoop and related software You should already have seen the CloudMesh interface, and learned how it can be used to create VMs If you would like to continue using a VM created using CloudMesh, you can proceed to section 2. If you’d like to learn more about using OpenStack command line tools to create virtual machines, continue with this section of the tutorial. CloudMesh material; can be found here: – http://cloudmesh.github.io/introduction_to_cloud_computing/class/i590.html

4. OpenStack http://www.openstack.org/ OpenStack is a free, open-source cloud computing platform which is mainly used to provide infrastructure as a service (IaaS). Launched in 2010 as joint project between Rackspace Hosting and NASA, OpenStack has been overseen since 2012 by a non-profit organization, the OpenStack Foundation. Only in existence for four years, OpenStack’s growth has been phenomenal with hundreds of major companies involved in the project.

5. Creating Your VM – Prerequisites You have created an account on the FutureGrid portal – https://portal.futuregrid.org/ https://portal.futuregrid.org/ You have requested and have access to project FG-452 – https://portal.futuregrid.org/projects/452 https://portal.futuregrid.org/projects/452 You have set up your SSH keys to allow access to India

6. Creating Your VM – OpenStack Nova Load the Nova Client tools – $ module load novaclient Create a novarc file for credentials/environment variables – $ source ~/.futuregrid/openstack_havana/novarc Check your SSH key – $ nova keypair-list – should return something like this: – +-----------------+------------------------------------------------------------------+ – | Name | Fingerprint | – +-----------------+------------------------------------------------------------------+ – | -key | ab:a6:63:82:dd:08:d3:bc:c0:21:56:4c:e2:bb:22:ac | – +-----------------+------------------------------------------------------------------+ If the keypair-list does NOT display your key, add it – $ nova keypair-add $USER-key > ~/.ssh/$USER-key – $ chmod 600 ~/.ssh/$USER-key

7. Creating Your VM – OpenStack Nova See available flavors and images – $ nova flavor-list – $ nova image-list Request you new Virtual Machine – $ nova boot --flavor m1.medium --image "futuregrid/ubuntu-12.04“ -- key_name $USER-key $USER-001 – --flavor and --image choices may vary. See $ nova flavor-list and $ nova image- list commands Newly requested machine will not be available for a few minutes – $ nova list will display your VMs and their state You can access your VM using the private network address displayed by the nova list command See FutureGrid documentation to add an external IP address for access from other machines

8. Example Output from $ nova list Connect using the listed IP address – $ ssh -l ubuntu -i ~/.ssh/$ -key 10.35.23.18 Delete VM when no longer needed by name – $ nova delete $USER-001

9. Commands for Hands-On Session A text file containing the command for the following terminal session can be accessed at this link: – https://drive.google.com/open?id=0B5WPa- fii4KDUm5vUUVONFRvdkE&authuser=0 https://drive.google.com/open?id=0B5WPa- fii4KDUm5vUUVONFRvdkE&authuser=0

10. Terminal Session to Create VM