Download presentation
Presentation is loading. Please wait.
Published byMorgan Singleton Modified over 9 years ago
1
Hola Hadoop
2
0. Clean-Up The Hard-disks Delete tmp/ folder from workspace/mdp-lab3 Delete unneeded downloads
3
0. Peligro! Please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please
4
… please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please please
5
Peligro! … please
6
… please be careful of what you are doing! Think twice before: rm mv cp kill emacs/vim/… configuration files
7
… please.
8
cluster.dcc.uchile.cl
9
1. Download tools http://aidanhogan.com/teaching/cc5212- 1/tools/ http://aidanhogan.com/teaching/cc5212- 1/tools/ Unzip them somewhere you can find them
10
2. Log-in PuTTy 1 2 3
11
3. Open DFS Browser http://cluster.dcc.uchile.cl:50070/
12
3. PuTTy: Upload data to HDFS hadoop fs -ls / hadoop fs -ls /uhadoop hadoop fs -mkdir /uhadoop/[username] – [username] = first letter first name, last name (e.g., “ahogan”) cd /data/hadoop/hadoop/data/ hadoop fs -copyFromLocal /data/hadoop/hadoop/data/es-abstracts.txt /uhadoop/[username]/es-abstracts.txt
13
Note on namespace If you need to disambiguate local/remote files HDFS file – hdfs://cm:9000/uhadoop/… Local file – file:///data/hadoop/...
14
4. Let’s Build Our First MapReduce Job Hint: Use Monday’s slides for “inspiration” – http://aidanhogan.com/teaching/cc5212-1/ http://aidanhogan.com/teaching/cc5212-1/ 1.Implement map(.,.,.,.) method 2.Implement reduce(.,.,.,.) method 3.Implement main(.) method
15
5. Eclipse: Build jar Right Click build.xml > dist (Might need to make a dist folder)
16
6. WinSCP: Copy.jar to Master Server Don’t save password! 1 2 3 4
17
6. WinSCP: Copy.jar to Master Server
18
Create dir: /data/2014/uhadoop/[username]/ Copy your mdp-lab4.jar into it
19
7. Putty: Run Job hadoop jar /data/2014/uhadoop/[username]/mdp- lab4.jar WordCount /uhadoop/[username]/es- abstracts.txt /uhadoop/[username]/wc/ All one command!
20
8. Look at output hadoop fs -ls /uhadoop/[username]/wc/ hadoop fs -cat /uhadoop/[username]/wc/part-00000 | more hadoop fs -cat /uhadoop/[username]/wc/part-00000 | grep - e "^de" | more All one command! Look for “de” … 4575144 occurrences in local run
21
9. Look at output through browser http://cluster.dcc.uchile.cl:50070/
Similar presentations
© 2024 SlidePlayer.com. Inc.
All rights reserved.