1. Lab #2 - Create a movies dataset
Lab #1 - VM setup
Lab #2 - Create a movies dataset
Use approximately 10 minutes for this lab, but get started early to consume slip time if people need help

2. Strata New York 2015 Building an Apache Hadoop Data Application
Ryan Blue and Tom White

3. Content for today’s tutorial
The Hadoop Ecosystem
Storage on Hadoop
Movie ratings app: Data ingest
Movie ratings app: Data analysis

4. The Hadoop Ecosystem
Time: 1 hour (including lab #1 and 10 minutes slip)

5. A Hadoop Stack

6. Processing frameworks
Code: MapReduce, Crunch, Spark, Flink
SQL: Hive, Impala, Phoenix, Trafodian, Drill, Presto
Tuples: Cascading, Pig
Streaming: Spark streaming (micro-batch), Flink streaming, Storm, Samza
I’m not an expert on Cascading or Streaming processing.
Mention Samza?

7. Coding frameworks Crunch
A layer around MR (or Spark) that simplifies writing pipelines
Spark
A completely new framework for processing pipelines
Takes advantage of memory, runs a DAG without extra map phases
Note that the last example is done in Crunch. It is similar to Spark, but can run on MR or Spark.

8. SQL on Hadoop Hive for batch processing Impala for low-latency queries
SparkSQL to use SQL with Spark applications
Phoenix and Trafodion for transactional queries on HBase
Run queries with Hive and Impala over the dataset loaded in the lab and compare the execution times

9. Ingest tools Relational: Sqoop, Sqoop2
Record channel: Kafka, Flume, NiFi
Files: NiFi
Numerous commercial options

10. Ingest tools Relational: Sqoop, Sqoop2 Record channel: Kafka, Flume
Files: NiFi
App
HDFS
Database
Flume
Flume
Hadoop

11. Relational DB to Hadoop
Sqoop
CLI to run MR-based import jobs
Sqoop2
Fixes configuration problems with Sqoop with credentials service
More flexible to run on non-MR frameworks
Under development
Run queries with Hive and Impala over the dataset loaded in the lab and compare the execution times

12. Ingest tools Relational: Sqoop, Sqoop2
Record channel: Kafka, Flume, NiFi
Files: NiFi
Flume
App
Flume
Channels
HDFS
Database
Hadoop

13. Record streams to Hadoop
Flume - source, channel, sink architecture
Well-established and integrated with other tools
No order guarantee, duplicates are possible
Kafka - pub-sub model for low latencies
Partitioned, provides ordering guarantees, easier to eliminate duplicates
More resilient to node failure with consumer groups
Flume is used in the example - because it is the established and more supported route

14. Files to Hadoop
NiFi
Web GUI for drag & drop configuration of a data flow
Enterprise features: back-pressure, monitoring, lineage, etc.
Read from local disk, HTTP, FTP, SFTP, HDFS, . . .
Originally for files, also handles record-based streams

15. NiFi

16. Data storage in Hadoop
Go through the solution to Lab #2

17. HDFS Blocks Blocks Increase parallelism Balance work Replicated
Configured by dfs.blocksize
Client-side setting
data1.avro
...
data2.avro

18. Splittable File Formats
Splittable: Able to process part of a file independently of others
Process blocks in parallel
Avro is splittable
Gzipped content is not splittable
CSV is effectively not splittable
Ask about splittability: Which of the following would be splittable?
XML
JSON

19. File formats Existing formats: XML, JSON, Protobuf, Thrift
Designed for Hadoop: SequenceFile, RCFile, ORC
Never use: Delimited text
Recommended: Avro or Parquet
Note that ORC is a well-designed format as well, but not on the recommended list because it is Hive-specific.

20. Avro Recommended row-oriented format
Broken into blocks with sync markers for splitting
Binary encoding with block-level compression
Avro schema
Required to read any binary-encoded data!
Written in the file header
Flexible object models
Time: 10 minutes for Avro overview and tools lab

21. Avro in-memory object models
generic
Object model that can be used with any schema
specific - compile schema to java object
Generates type-safe runtime objects
reflect - java object to schema
Uses existing classes and objects

22. Optional Lab #3 - Using avro-tools http://tiny.cloudera.com/StrataLab3
Inspect one of the Avro files in the dataset created for lab #2

23. Row- and column-oriented formats
Able to reduce I/O when projecting columns
Better encoding and compression A B C D a1 b1 c1 d1 a2 b2 c2 d2
...
Columns:
Rows:
Images © Twitter, Inc.

24. Parquet Recommended column-oriented format
Splittable by organizing into row groups
Efficient binary encoding, supports compression
Uses other object models
Record construction API rather than object model
parquet-avro - Use Avro schemas with generic or specific records
parquet-protobuf, parquet-thrift, parquet-hive, etc.
Time: 10 minutes for Parquet overview and tools lab

25. Parquet trade-offs
Rows are buffered into groups that target a final size
Row group size
Memory consumption grows with row group size
Larger groups get more I/O benefit and better encoding
Memory consumption grows for each open file

26. Optional Lab #4 Using parquet-tools
Copy the Avro dataset from lab #2 to a new Parquet dataset and inspect one of its files.
Things to note:
Parquet schema, not Avro schema
Row group info
Encoding and compression results
Page-level information

27. Partitioning Splittable file formats aren’t enough
Not processing data is better than processing in parallel
Organize data to avoid processing: Partitioning
Use HDFS paths for a coarse index: data/y=2015/m=03/d=14/

28. Partitioning day= 30 31 1 2 ... 3 month= November December January
February
year=
2014
2015
events

29. Partitioning Caution Partitioning in HDFS is the primary index to data
Should reflect the most common access pattern
Test partition strategies for multiple workloads
Should balance file size with workload
Lots of small files are bad for HDFS - partitioning should be more coarse
Larger files take longer to find data - partitioning should be more specific

30. Implementing partitioning
Build your own - not recommended
Hive and Impala managed
Partitions are treated as data columns
Insert statements must include partition calculations
Kite managed
Partition strategy configuration file
Compatible with Hive and Impala

31. Kite High-level data API for Hadoop Built around datasets, not files
Tasks like partitioning are done internally
Tools built around the data API
Command-line
Integration in Flume, Sqoop, NiFi, etc.

32. Lab #5 - Create a partitioned dataset

33. Movie ratings app: Data ingest pipeline
Time: 1 hour including Lab #6, 10 minutes slip

34. Movie ratings scenario
Your company runs a web application where users can rate movies
You want to use Hadoop to analyze ratings over time
Avoid scraping the production database for changes
Instead, you want to log every rating submitted
Ratings App
Database
Time: 20 minutes to setup the scenario and describe the Hadoop architecture
Also cover:
Where other tools might be used
Why the dataset is Avro and not Parquet
Why there is a division between Hadoop and the application
Security and impersonation with Flume

35. Movie ratings app Log ratings to Flume Otherwise unchanged HDFS Flume
dataset
Database
Hadoop

36. Why Flume? Database and application constraints
Can the application change?
Can the database handle exports?
Why not Kafka?
Kafka is a channel or queue
Listener and HDFS writer need to run somewhere
Flume
Ratings App
Flume
Flume
Time: 20 minutes to setup the scenario and describe the Hadoop architecture
Also cover:
Where other tools might be used
Why the dataset is Avro and not Parquet
Why there is a division between Hadoop and the application
Security and impersonation with Flume

37. Lab #6 - Create a Flume pipeline http://tiny.cloudera.com/StrataLab6
Time:
10 minutes for users to get through the application setup
5 minutes to run through the demo up front
5 minutes to go let everyone submit queries to the application and discuss
10 minutes to go over relevant code and configuration, discuss trade-offs, and note what could be changed

38. Movie ratings app: Analyzing ratings data

39. Movie ratings analysis
Now you have several months of data
You can query it in Hive and Impala for most cases
Some questions are difficult to formulate as SQL
Are there any movies that people either love or hate?
Time: 5 minutes to setup and describe task

40. Analyzing ratings Map Extract key, movie_id, and value, rating Reduce:
Reduce groups all of the ratings by movie_id
Count the number of ratings for each movie
If there are two peaks, output the movie_id and counts
Peak detection: difference between counts goes from negative to positive

41. Crunch background
Stack up functions until a group-by operation to make a map phase
Similarly, stack up functions after a group-by to make a reduce phase
Additional group-by operations set up more MR rounds automatically
PTable<Long, Double> table = collection
.by(new GetMovieID(), Avros.longs())
.mapValues(new GetRating(), Avros.ints())
.groupByKey()
.mapValues(new AverageRating(), Avros.doubles());
Time: 10 minutes to discuss crunch background and show application

42. Lab #7 - Analyze ratings with Crunch
Time:
10 minutes for students to run the lab
5 minutes to run the lab up front
10 minutes to discuss the results, add an output dataset

43. blue@cloudera.com tom@cloudera.com
Thank you