What is Serialization? Serialization is the process of turning structured objects into a byte stream for transmission over a network or for writing to persistent storage. Deserialization is the reverse process of turning a byte stream back into a series of structured objects.

Advantages of Serialization Compact - A compact format makes the best use of network bandwidth, which is the most scarce resource in a data center. Fast - Interprocess communication forms the backbone for a distributed system, so it is essential that there is as little performance overhead as possible for the serialization and deserialization process. Extensible - Protocols change over time to meet new requirements, so it should be straightforward to evolve the protocol in a controlled manner for clients and servers. Interoperable - For some systems, it is desirable to be able to support clients that are written in different languages to the server, so the format needs to be designed to make this possible.

What is Writable? Hadoop defines its own ‘box classes’ for strings, integers and so on – IntWritable for ints – LongWritable for longs – FloatWritable for floats – DoubleWritable for doubles – Text for strings – Etc. The Writable interface makes serialization quick and easy for Hadoop Any value’s type must implement the Writable interface

What is WritableComparable? A WritableComparable is a Writable which is also Comparable – Two WritableComparables can be compared against each other to determine their ‘order’ – Keys must be WritableComparables because they are passed to the Reducer in sorted order Note that despite their names, all Hadoop box classes implement both Writable and WritableComparable – For example, IntWritable is actually a WritableComparable

Instruction

HDFS상에서 주어진 경로상에 있는 컨텐츠 보는 명령어 hadoop fs HDFS상에서 주어진 경로상에 있는 컨텐츠 보는 명령어 hadoop fs -ls / hadoop fs -ls /user hadoop fs -ls /user/training HDFS에 파일 적재 로컬에 있는 Input파일을 HDFS상에 올린다. hadoop fs -put localpath/inputfile HDFS/ HDFS 디렉토리 생성 hadoop fs -mkdir weblog

Hadoop 구동 방법 Extract and upload the file in one step gunzip -c access_log.gz \ | hadoop fs -put - weblog/access_log tar zxvf shakespeare.tar.gz | hadoop fs -put shakespeare input Hadoop 구동 방법 Input 파일을 HDFS상에 올린다. Java 코드를 eclipse상에서 import한 후에 jar파일로 expor한다. 터미널 명령어로 컴파일 후 jar파일 만드는 법(주어진 vm에서는 안됨) javac -classpath `hadoop classpath` *.java jar cvf wc.jar *.class 터미널에서 Hadoop 명령어를 사용하여 실행 Hadoop jar [jar file] [Driver Class name] [hdfs-inputpath] [hdfs- outputpath] hadoop jar wc.jar WordCount shakespeare wordcounts

WordCount

Goal Input Output

데이터 업로드와 실행 Output $ cd ~/training_materials/developer/data $ tar zxvf shakespeare.tar.gz | hadoop fs -put shakespeare input $ hadoop jar WordCount.jar WordCount input/shakespeare/* /user/shakesOut Output

Inverted Index

Goal Input Output abominably hamlet@2787 abomination rapeoflucrece@876, rapeoflucrece@1124 abominations rapeoflucrece@2167, antonyandcleopatra@3028 abortive loveslabourslost@197, 2kinghenryvi@3108, kingrichardiii@1067, kingrichardiii@386 abortives kingjohn@2037

데이터 업로드와 실행 Output $ cd ~/training_materials/developer/data $ tar zxvf invertedIndexInput.tgz $ hadoop fs –put invertedIndexInput invertedIndexInput $ hadoop jar InvertedIndex.jar InvertedIndex invertedIndexInput output Output