Hadoop Architecture Mr. Sriram Email: hadoopsrirama@gmail.com

Objectives Hadoop Cluster – A typical usage Hadoop 2.X Cluster Architecture Analyze Hadoop 2.X Cluster Architecture - Federation Analyze Hadoop 2.X Cluster Architecture - High Availability Hadoop 2.X Resource Management Run Hadoop in different Cluster modes Installation & Configuration of Hadoop Implement basic Hadoop commands on terminal Prepare Hadoop 2.X configuration files and analyze the parameters in it Implement different loading techniques

Hadoop Architecture Hadoop Cluster Typical Use Hadoop 2.X Cluster Architecture Hadoop 2.X Cluster Architecture – Federation Hadoop 2.X Cluster Architecture – High Availability Hadoop 2.X Resource Management Hadoop Cluster – Facebook Hadoop Cluster Modes

Hadoop Cluster – A Typical Use Case

Hadoop 1.0 Cluster

Hadoop 2.0 Cluster

Hadoop 2.X Cluster Master-Slave Architecture

Hadoop 2.X Cluster Architecture

Hadoop 2.X Cluster Architecture - Federation

Hadoop 2.X Cluster Architecture – High Availability

Hadoop 2.X Resource Management

Hadoop 2.X Resource Management..

Hadoop Cluster - Facebook

Hadoop Cluster Modes

Hadoop Installation & Configuration Hadoop FS shell Commands Terminal Commands Hadoop 2.X Configuration Files -> core-site.xml -> hdfs-site.xml -> mapred-site.xml -> yarm-site.xml Slaves & Masters Per-Process Run Time Environment Hadoop Daemons Hadoop Web UI Parts Hadoop Installation & Configuration

Hadoop Installation Pre-requisites Install JAVA(version 6 or later) Use java –version for checking for java installation Use which java to locate the java directory Hadoop runs on Unix and Windows Linux is the only supported production platform Windows and Mac is supported only as development platform Windows additionally requiresCygwinto run

Hadoop Installation We can install Hadoop in any of the following ways: 1) Automated method using Cloudera Manager 2) Manual methods described below: i. Install from a CDH5 tarball ii. Install from RPMs

Hadoop Installation from a Tarball Downloading Tarball Download stable Hadoop release from Cloudera Hadoop release page https://www.cloudera.com/content/support/en/downloads.html UnpackingTarball Unpack the Hadoop archive to /home/$USER/using $ tar xzfhadoop-x.y.z.tar.gz

Hadoop Installation from a Tarball Setting Environment Variables Edit conf/hadoop-env.sh in the Hadoop folder Specify the JAVA_HOME variable by adding export JAVA_HOME=/usr/java/ Edit.profile file using any text editor and set the HADOOP_HOME, JAVA_HOME & necessary CLASSPATHs

Hadoop Installation from RPM Download the CDH packages that matches your Red Hat or CentOS system from cloudera one click install page: archive.cloudera.com/cdh4/one-click-install/redhat Install the RPM: sudo yum--nogpgchecklocalinstallcloudera-cdh-4-0.x86_64.rpm Optionally Add a RepositoryKey Sudo rpm--importhttp://archive.cloudera.com/cdh4/redhat/5/x86_64/cdh/RPM Install Hadoop in pseudo-distributed mode Sudo yuminstallhadoop-0.20-conf-pseudo

Hadoop Installation from Cloudera Manager 1.Download and run the Cloudera Manager Installer Download cloudera-manager-installer.bin from the Cloudera Downloads page http://www.cloudera.com/content/support/en/downloads.html 2.After downloading cloudera-manager-installer.bin, change it to have executable permission chmod u+x cloude chmod u+x cloudera-manager-installer.bin 3.Run cloudera-manager-installer.bin sudo./cloudera-manager-installer.bin Read the Cloudera Manager Readme and then press Enter to choose Next

Hadoop Installation from Cloudera Manager 4. Start the Cloudera Manager Admin Console http://<Serverhost>:<port> http://myhost.example.com:7180/ Log into Cloudera Manager The default credentials are: Username: admin Password: admin Use Cloudera Manager for Automated CDH Installation and Configuration Find the cluster hosts you specify via hostname and ILP–address ranges Click Search Cloudera Manager identifies the hosts on your cluster to allow you to configure them for CDH Choose the CDH version to install

Hadoop FS Shell Commands

Terminal Commands

Terminal Commands – mkdir, touchz, ls, count Terminal Type admin terminal # | user terminal $ To make a directory $ hadoop fs -mkdir /user/cloudera/Monday To create an empty file $ hadoop fs -touchz /user/cloudera/Monday/one.txt To list number of files and directories present in HDFS location $ hadoop fs -ls /user/cloudera/Monday To count the number of files and directories available in HDFS location $ hadoop fs -count /user/cloudera/Monday

Terminal Commands - Copy To copy the file from LFS to HDFS $ hadoop fs -put /home/cloudera/Desktop/two.txt /user/cloudera/Monday (or) $ hadoop fs -copyFromLocal /home/cloudera/Desktop/three.txt /user/cloudera/Monday To copy file from HDFS to LFS $ hadoop fs -get /user/cloudera/Monday/two.txt /home/cloudera/Desktop/Tuesday (or ) $ hadoop fs -copyToLocal /user/cloudera/Monday/one.txt /home/cloudera/Desktop/Tuesday

Terminal Commands – cat, rm To print the contents of HDFS file: $ hadoop fs -cat /user/cloudera/Monday/two.txt or $ hadoop fs -text /user/cloudera/Monday/two.txt To remove the directory from HDFS location $ hadoop fs -rm -r /user/cloudera/Monday

Hadoop Configuration Files Each component in Hadoop is configured using an XML file These XML files are allocated in the conf subdirectory in Hadoop folder The three most important XML files are: Core-site.xml-Core properties Hdfs-site.xml-HDFS properties Mapred-site.xml-MapReduce properties To run Hadoop in a particular mode, you need to do two things: Set the appropriate properties in the configuration files Start the Hadoop daemons

Hadoop 2.x Configuration Files

Hadoop 2.x Configuration Files – Apache Hadoop

core-site.xml

hdfs-site.xml

mapred-site.xml

yarn-site.xml

Slaves & Masters

Per-Process Run Time Environment

All Properties

Running Hadoop Before Hadoop can be used,a brand-new HDFS installation needs to be formatted Commands To Format the Name Node: hadoop namenode -format To start the HDFS and MapReduce daemon $ sstart-dfs.sh, $ start-mapred.sh. Or use $ start-all.sh to start all daemons If you have placed configuration files outside the default conf directory, start the daemons with the— config option, start-xyz.sh—config path-to-config-directory To stop a daemon $stop-dfs.sh, $stop-mapred.sh Or use $stop-all.sh to stop all daemons.

Hadoop 2.x Daemons

Hadoop Daemons

Hadoop Web UI Parts

Hadoop Web UI URL’s

Hadoop Stack

Data Loading Techniques and Data Analysis

Data Loading using Flume

Data Loading using SQOOP

Further Reading

Further Reading..

Thank You !!!!!!!!!!!