인공지능연구실 이남기 ( beohemian@gmail.com ) 유비쿼터스 응용시스템: 실습 가이드 인공지능연구실 이남기 ( beohemian@gmail.com )

Environment Cloudera QuickStart VM with 5.4.2 Guide for Download http://ailab.ssu.ac.kr/rb/?c=8/29&cat=2015_2_%EC%9C%A0%EB%B9%84%EC %BF%BC%ED%84%B0%EC%8A%A4+%EC%9D%91%EC%9A%A9%EC%8B%9C% EC%8A%A4%ED%85%9C&uid=660

Contents Using HDFS Running MapReduce Job : WordCount How To Use How To Upload File How To View and Manipulate File Exercise Running MapReduce Job : WordCount Goal Remind MapReduce Code Review Run WordCount Program Importing Data With Sqoop Review MySQL

Using HDFS With Exercise

Using HDFS How to use HDFS How to Upload File How to View and Manipulate File

Using HDFS – How To Use (1) You see a help message describing all the commands associated with HDFS $ hadoop fs

Using HDFS – How To Use (2) You see the contents of directory in HDFS: $ hadoop fs –ls / $ hadoop fs –ls /user $ hadoop fs –ls /user/cloudera

Exercise How To Use

Using HDFS – How To Upload File (1) Unzip ‘shakespeare.tar.gz’: $ cd ~/training_materials/developer/data $ tar zxvf shakespeare.tar.gz

Using HDFS – How To Upload File (2) Insert ‘shakespeare’ directory into HDFS: $ hadoop fs -put shakespeare /user/cloudera/shakespeare

Exercise How To Upload

Using HDFS – How To View and Manipulate Files (1) Remove directory $ hadoop fs –ls shakespeare $ hadoop fs –rm shakespeare/glossary

Using HDFS – How To View and Manipulate Files (2) Print the last 50 lines of Herny IV $ hadoop fs –cat shakespeare/histories \ | tail –n 50

Using HDFS – How To View and Manipulate Files (3) Download file and manipulate $ hadoop fs –get shakespeare/poems \ ~/shakepoems.txt $ less ~/shakepoems.txt If you want to know other command: $ hadoop fs

Exercise How To View and Manipulate Files

Importing Data With Sqoop Review MySQL and Exercise

Importing Data With Sqoop Log on to MySQL: $ mysql --user=root \ --password=cloudera Select Database > use retail_db; Show Databases: > show databases;

Importing Data With Sqoop – Review MySQL (1) Log on to MySQL: $ mysql --user=root \ --password=cloudera Show Databases: > show databases; Select Databases: > use retail_db; Show Tables: > show tables;

Importing Data With Sqoop – Review MySQL (2) Review ‘customers’ table schema: > DESCRIBE customers;

Importing Data With Sqoop – Review MySQL (3) Review ‘customers’ table: > DESCRIBE customers; … > SELECT * FROM customers LIMIT 5;

Importing Data With Sqoop – How To Use (1) List the databases (schemas) in your database server: $ sqoop list-databases \ --connect jdbc:mysql://localhost \ --username root --password cloudera List the tables in the ‘retail_db’ database: $ sqoop list-tables \ --connect jdbc:mysql://localhost/movielens \ --username root --password cloudera

Importing Data With Sqoop – How To Use (2) Import the ‘customers’ table into HDFS $ sqoop import \ --connect jdbc:mysql://localhost/retail_db \ --table customers --fields-terminated-by '\t' \ --username root --password cloudera Verify that the command has worked $ hadoop fs –ls customers $ hadoop fs –tail movie/part-m-00000