Calculation of stock volatility using Hadoop and map-reduce Ravi Teja Nekkanti U00291556

Stock volatility: Volatility = swings(varies in short period of time) Measure of relative performance of a stock High volatility stocks = more swings(and vice versa)

Whether the values plummets or raises, the fluctuations depend on day to day basis

The Hadoop framework Hadoop is open source java framework Two parts : HDFS and Hadoop MapReduce HDFS splits files into small chucks Redundancy - replication Computing Cost Programming framework on top of hdfs Parallel computation

HADOOP DISTRIBUTED FILE SYSTEM 2 types of file system – local file system and HDFS Local file system – 4kb(block size) HDFS – 64 MB(block size) REPLICATIONS – 3

HDFS SERVICES OR DAEMONS HDFS PROVIDES FIVE SERVICES NAME NODE SECONDARY NAMENODE DATA NODE JOB TRACKER TASK TRACKER

MASTER SERVICES AND SLAVE SERVICES MASTER SERVICES OR MASTER DAEMONS ASSIGN TASKS FOR SLAVE SERVICES MASTER DAEMONS : NAME NODE, SECONDARY NAME NODE, JOB TRACKER SLAVE DAEMONS: DATA NODE, TASK TRACKER MASTER NODE SERVICES TALK TO EACH OTHER AND SLAVE SERVICES TALK TO EACH OTHER

DAEMONS MASTER DAEMONS SLAVE DAEMONS NAME NODE DATA NODE SECONDARY NN TASK TRACKER JOB TRACKER

NAME NODE TO SOTRE DATA IN HDFS, THE FIRST CONTACT IS NAMENODE. TELLS THE CLIENT WHERE THE DATA IS PRESENT STORES METADATA OF DATA TALKS TO SNN, DATA NODE, JOB TRACKER Receives heart beats from data nodes(for every 3 seconds)

Secondary name node STORES METADATA WHICH IS REPLICATED FROM NAMENODE NAME NODE CAN EDIT LOGS AND HAS FS IMAGE SECONDARY NAME NODE PULLS FS IMAGE FROM NAMENODE REASONABLE PULL TIME IS ONE HOUR

JOB TRACKER JOB TRACKER ASSIGNS JOBS TO TASK TRACKER JOB TRACKER ASKS FOR MATA DATA FROM NAME NODE JOB TRACKER SENDS JAR FILE TO TASK TRACKER WHERE DATA IS STORED

DATA NODE DATA NODE STORES THE DATA SENDS BLOCK REPORT FOR EVERY 3 SECONDS TO NAMENODE REPLICATION OF DATA TASK TRACKER TALKS TO DATA NODE TO PERFORM JOB ON DATA

TYPICAL LAYOUT OF HDFS NODE NN JT SNN NODE 1 NODE 2 NODE 3 NODE 4 DN TT DN TT DN TT DN TT DN TT NODE 6 NODE 7 NODE 8 NODE 9 NODE 10 DN TT DN TT DN TT DN TT DN TT

MapReduce design This framework allows to solve problems by parallelized. Takes jar file from client and performs calculations Takes data as input splits Gives jar to each task tracker Mapper – depends on number of input splits and it performs data execution Reducer – aggregates data from mapper

Basic design of MapReduce framework

Files responsible from Hadoop framework Core-site.xml – starts name node and secondary name node services Hdfs-site.xml – starts data node Mapred-site.xml – starts job tracker and task tracker services

Basic Hadoop commands List files : $HADOOP FS –LS / Make dir : $HADOOP FS –MKDIR /GOLIB Put data : $ HADOOP FS –PUT INPUT /GOLIB Perform job : $ HADOOP JAR JARFILE.JAR DRIVERCLASS INPUT OUTPUT

Mapper and reducer input and output Mapper reads input data line by line Mapper performs operations and gives output by key and value pairs Mapper generates intermediate data. Intermediate data is shuffled and sorted Reducer takes intermediate data and aggregates it as key and value pairs Number of output files depend on number of reducers

Map reduce flow chart Input file Input split Input split Input split Record reader Record reader Record reader Record reader Mapper Mapper Mapper Mapper Intermediate data reducer reducer reducer reducer Record writer Record writer Record writer Record writer output

Java programming Input is taken as Text Input Format Wrapper class Primitive data types Box classes Integer int IntWritable Float float FloatWritable Double doublt DoubleWritable String string Text Long long LongWritable

The program consists of 3 files – Mapper class, reducer class, main class 1000 stocks of nyse are taken and performed calculation on them. Output is given in the format of key and value pairs Stock volatility is calculated. As output is sorted, we can get the top 10 values

demo

Questions ???

Thank You