Apache Tez : Accelerating Hadoop Query Processing Page 1.

Slides:



Advertisements
Similar presentations
A Workflow Engine with Multi-Level Parallelism Supports Qifeng Huang and Yan Huang School of Computer Science Cardiff University
Advertisements

MAP REDUCE PROGRAMMING Dr G Sudha Sadasivam. Map - reduce sort/merge based distributed processing Best for batch- oriented processing Sort/merge is primitive.
MapReduce Online Created by: Rajesh Gadipuuri Modified by: Ying Lu.
MapReduce Online Veli Hasanov Fatih University.
Apache Tez : Accelerating Hadoop Data Processing
Parallel Computing MapReduce Examples Parallel Efficiency Assignment
© Hortonworks Inc Running Non-MapReduce Applications on Apache Hadoop Hitesh Shah & Siddharth Seth Hortonworks Inc. Page 1.
Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM.
Spark: Cluster Computing with Working Sets
Data-Intensive Computing with MapReduce/Pig Pramod Bhatotia MPI-SWS Distributed Systems – Winter Semester 2014.
APACHE GIRAPH ON YARN Chuan Lei and Mohammad Islam.
Clydesdale: Structured Data Processing on MapReduce Jackie.
Distributed Computations
Homework 2 In the docs folder of your Berkeley DB, have a careful look at documentation on how to configure BDB in main memory. In the docs folder of your.
Putting the Sting in Hive Page 1 Alan F.
CPS216: Advanced Database Systems (Data-intensive Computing Systems) How MapReduce Works (in Hadoop) Shivnath Babu.
Lecture 2 – MapReduce CPE 458 – Parallel Programming, Spring 2009 Except as otherwise noted, the content of this presentation is licensed under the Creative.
Take An Internal Look at Hadoop Hairong Kuang Grid Team, Yahoo! Inc
Hadoop: The Definitive Guide Chap. 8 MapReduce Features
HADOOP ADMIN: Session -2
Hadoop & Cheetah. Key words Cluster  data center – Lots of machines thousands Node  a server in a data center – Commodity device fails very easily Slot.
Advanced Topics: MapReduce ECE 454 Computer Systems Programming Topics: Reductions Implemented in Distributed Frameworks Distributed Key-Value Stores Hadoop.
MapReduce.
USING HADOOP & HBASE TO BUILD CONTENT RELEVANCE & PERSONALIZATION Tools to build your big data application Ameya Kanitkar.
Map Reduce for data-intensive computing (Some of the content is adapted from the original authors’ talk at OSDI 04)
HBase A column-centered database 1. Overview An Apache project Influenced by Google’s BigTable Built on Hadoop ▫A distributed file system ▫Supports Map-Reduce.
MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean and Sanjay Ghemawat.
Our Experience Running YARN at Scale Bobby Evans.
MapReduce: Hadoop Implementation. Outline MapReduce overview Applications of MapReduce Hadoop overview.
Introduction to Hadoop and HDFS
f ACT s  Data intensive applications with Petabytes of data  Web pages billion web pages x 20KB = 400+ terabytes  One computer can read
HAMS Technologies 1
Contents HADOOP INTRODUCTION AND CONCEPTUAL OVERVIEW TERMINOLOGY QUICK TOUR OF CLOUDERA MANAGER.
Restore : Reusing results of mapreduce jobs Jun Fan.
MapReduce M/R slides adapted from those of Jeff Dean’s.
Harp: Collective Communication on Hadoop Bingjing Zhang, Yang Ruan, Judy Qiu.
Tool Integration with Data and Computation Grid GWE - “Grid Wizard Enterprise”
Grid Computing at Yahoo! Sameer Paranjpye Mahadev Konar Yahoo!
Hung-chih Yang 1, Ali Dasdan 1 Ruey-Lung Hsiao 2, D. Stott Parker 2
Department of Computer Science MapReduce for the Cell B. E. Architecture Marc de Kruijf University of Wisconsin−Madison Advised by Professor Sankaralingam.
CS525: Big Data Analytics MapReduce Computing Paradigm & Apache Hadoop Open Source Fall 2013 Elke A. Rundensteiner 1.
© Hortonworks Inc Hadoop: Beyond MapReduce Steve Loughran, Big Data workshop, June 2013.
C-Store: MapReduce Jianlin Feng School of Software SUN YAT-SEN UNIVERSITY May. 22, 2009.
Tool Integration with Data and Computation Grid “Grid Wizard 2”
MapReduce Joins Shalish.V.J. A Refresher on Joins A join is an operation that combines records from two or more data sets based on a field or set of fields,
Experiments in Utility Computing: Hadoop and Condor Sameer Paranjpye Y! Web Search.
Next Generation of Apache Hadoop MapReduce Owen
Beyond Hadoop The leading open source system for processing big data continues to evolve, but new approaches with added features are on the rise. Ibrahim.
Learn. Hadoop Online training course is designed to enhance your knowledge and skills to become a successful Hadoop developer and In-depth knowledge of.
1 Student Date Time Wei Li Nov 30, 2015 Monday 9:00-9:25am Shubbhi Taneja Nov 30, 2015 Monday9:25-9:50am Rodrigo Sanandan Dec 2, 2015 Wednesday9:00-9:25am.
COMP7330/7336 Advanced Parallel and Distributed Computing MapReduce - Introduction Dr. Xiao Qin Auburn University
MSBIC Hadoop Series Implementing MapReduce Jobs Bryan Smith
TensorFlow– A system for large-scale machine learning
Hadoop.
Distributed Programming in “Big Data” Systems Pramod Bhatotia wp
Chapter 10 Data Analytics for IoT
Spark Presentation.
Apache Spark Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing Aditya Waghaye October 3, 2016 CS848 – University.
MapReduce Computing Paradigm Basics Fall 2013 Elke A. Rundensteiner
MapReduce Simplied Data Processing on Large Clusters
湖南大学-信息科学与工程学院-计算机与科学系
Introduction to Apache
Overview of big data tools
Charles Tappert Seidenberg School of CSIS, Pace University
Cloud Computing for Data Analysis Pig|Hive|Hbase|Zookeeper
Overview of Workflows: Why Use Them?
COS 518: Distributed Systems Lecture 11 Mike Freedman
MapReduce: Simplified Data Processing on Large Clusters
Map Reduce, Types, Formats and Features
Pig Hive HBase Zookeeper
Presentation transcript:

Apache Tez : Accelerating Hadoop Query Processing Page 1

Agenda Page 2 Overview of Tez –Goals –High-level architecture –Status –Roadmap Hive on Tez Pig on Tez

© Hortonworks Inc Tez – Introduction Page 3 Distributed execution framework targeted towards data-processing applications. Based on expressing a computation as a dataflow graph. Built on top of YARN – the resource management framework for Hadoop. Open source Apache incubator project and Apache licensed.

© Hortonworks Inc Tez – Design Themes Page 4 Empowering End Users Execution Performance

© Hortonworks Inc Tez – Empowering End Users Expressive dataflow definition API’s Flexible Input-Processor-Output runtime model Data type agnostic Simplifying deployment Page 5

© Hortonworks Inc Tez – Empowering End Users Expressive dataflow definition API’s –Enable definition of complex data flow pipelines using simple graph connection API’s. Tez expands the logical plan at runtime. –Targeted towards data processing applications like Hive/Pig but not limited to it. Hive/Pig query plans naturally map to Tez dataflow graphs with no translation impedance. Page 6 TaskA-1 TaskA-2 TaskB-1 TaskB-2 TaskC-1 TaskC-2 TaskD-1 TaskD-2 TaskE-1 TaskE-2

© Hortonworks Inc Aggregate Stage Partition Stage Preprocessor Stage Tez – Empowering End Users Expressive dataflow definition API’s Page 7 Sampler Task-1 Task-2 Task-1 Task-2 Task-1 Task-2 Samples Ranges Distributed Sort

© Hortonworks Inc Tez – Empowering End Users Flexible Input-Processor-Output runtime model –Construct physical runtime executors dynamically by connecting different inputs, processors and outputs. –End goal is to have a library of inputs, outputs and processors that can be programmatically composed to generate useful tasks. Page 8 IntermediateReduce ShuffleInput ReduceProcessor FileSortedOutput FinalReduce ShuffleInput ReduceProcessor HDFSOutput PairwiseJoin Input1 JoinProcessor FileSortedOutput Input2

© Hortonworks Inc Tez – Empowering End Users Data type agnostic –Tez is only concerned with the movement of data. Files and streams of bytes. –Does not impose any data format on the user application. MR application can use Key-Value pairs on top of Tez. Hive and Pig can use tuple oriented formats that are natural and native to them. Page 9 File Stream Key Value Tez Task Tuples User Code Bytes

© Hortonworks Inc Tez – Empowering End Users Simplifying deployment –Tez is a completely client side application. –No deployments to do. Simply upload to any accessible FileSystem and change local Tez configuration to point to that. –Enables running different versions concurrently. Easy to test new functionality while keeping stable versions for production. –Leverages YARN local resources. Page 10 Client Machine Client Machine Node Manager Node Manager TezTask Node Manager Node Manager TezTask TezClient HDFS Tez Lib 1 Tez Lib 2 Client Machine Client Machine TezClient

© Hortonworks Inc Tez – Empowering End Users Expressive dataflow definition API’s Flexible Input-Processor-Output runtime model Data type agnostic Simplifying usage With great power API’s come great responsibilities Tez is a framework on which end user applications can be built Page 11

© Hortonworks Inc Tez – Execution Performance Performance gains over Map Reduce Optimal resource management Plan reconfiguration at runtime Dynamic physical data flow decisions Page 12

© Hortonworks Inc Tez – Execution Performance Performance gains over Map Reduce –Eliminate replicated write barrier between successive computations. –Eliminate job launch overhead of workflow jobs. –Eliminate extra stage of map reads in every workflow job. –Eliminate queue and resource contention suffered by workflow jobs that are started after a predecessor job completes. Page 13 Pig/Hive - MR Pig/Hive - Tez

© Hortonworks Inc Tez – Execution Performance Optimal resource management –Reuse YARN containers to launch new tasks. –Reuse YARN containers to enable shared objects across tasks. Page 14 YARN Container TezTask Host TezTask1 TezTask2 Shared Objects YARN Container Tez Application Master Tez Application Master Start Task Task Done Start Task

© Hortonworks Inc Tez – Execution Performance Plan reconfiguration at runtime –Dynamic runtime concurrency control based on data size, user operator resources, available cluster resources and locality. –Advanced changes in dataflow graph structure. –Progressive graph construction in concert with user optimizer. Page 15 HDFS Blocks YARN Resources YARN Resources Stage 1 50 maps 100 partitions Stage 1 50 maps 100 partitions Stage reducers Stage reducers Stage 1 50 maps 100 partitions Stage 1 50 maps 100 partitions Stage reducers Stage reducers Only 10GB’s of data

© Hortonworks Inc Tez – Execution Performance Dynamic physical data flow decisions –Decide the type of physical byte movement and storage on the fly. –Store intermediate data on distributed store, local store or in- memory. –Transfer bytes via blocking files or streaming and the spectrum in between. Page 16 Producer (small size) Producer (small size) In-Memory Consumer Producer Local File Consumer At Runtime

© Hortonworks Inc Tez – Deep Dive – API dag.addVertex(map1); dag.addVertex(map2); dag.addVertex(reduce1); dag.addVertex(reduce2); dag.addVertex(join1); Edge edge1 = Edge(map1, reduce1, BIPARTITE, STABLE, OnFileSortedOutput.class, ShuffledMergedInput.class); Edge edge2 = Edge(map2, reduce2, BIPARTITE, STABLE, OnFileSortedOutput.class, ShuffledMergedInput.class); Edge edge3 = Edge(reduce1, join1, BIPARTITE, STABLE, OnFileSortedOutput.class, ShuffledMergedInput.class); Edge edge4 = Edge(reduce2, join1, BIPARTITE, STABLE, OnFileSortedOutput.class, ShuffledMergedInput.class); dag.addEdge(edge1); dag.addEdge(edge2); dag.addEdge(edge3); dag.addEdge(edge4); Page 17 reduce1 map2 reduce2 join1 map1 Stable Bipartite Stable Bipartite

© Hortonworks Inc Tez – Deep Dive – Execution Page 18 reduce1 map2 reduce2 join1 map1

© Hortonworks Inc Tez – Deep Dive – Scheduling Page 19 reduce1 map1 Start vertex Vertex Scheduler Start tasks DAG Scheduler DAG Scheduler Get Priority Start vertex Task Scheduler Task Scheduler Get container Vertex Scheduler Determines when tasks in a vertex can start DAG Scheduler Determines priority of task Task Scheduler Allocates containers from YARN and assigns them to tasks

© Hortonworks Inc Tez – Deep Dive – Task Execution Page 20 Task Attempt (real on machine) Task Attempt (real on machine) Task Attempt (logical in AM) Task Attempt (logical in AM) Env, cmd line, resources Task JVM Input Processor Output Input Processor Output Get Task Start container Input Processor Output Data Information Data Information Data Events Start task shell with user specified env, resources etc. Fetch and instantiate Input, Processor, Output objects Receive (incremental) input information and process the input Provide output information

© Hortonworks Inc Tez – Current status Apache Incubator Project –Rapid development. Over 330 jiras opened. Over 220 resolved. –Growing community. Focus on stability –Testing and quality are highest priority. –Working on Tez+YARN to fix basic performance overheads. –Code ready and deployed on multi-node environments. DAG of MR processing is working – Already functionally equivalent to Map Reduce. Existing Map Reduce jobs can be executed on Tez with few or no changes. – Working Hive prototype that can target Tez for execution of queries (HIVE-4660). –Work started on prototype of Pig that can target Tez. Page 21

© Hortonworks Inc Tez – Current status Page 22 Fact Table Dimension Table 1 Result Table 1 Dimension Table 2 Result Table 2 Dimension Table 3 Result Table 3 Join Typical pattern in a TPC-DS query Fact Table Dimension Table 1 Optimization for small data sets Both can now run as a single Tez job

© Hortonworks Inc Tez – MRR Performance Page 23 TPC-DS Query 12 with Hive on Tez

© Hortonworks Inc Tez – Roadmap Full DAG support –Multi-way input and output. –Other graph connection patterns. Performance optimizations –Container reuse –Cross task shared resources –Using HDFS data caching Runtime plan optimizations –Automatic input (map) parallelism –Automatic aggregation (reduce) parallelism Usability. –Stability and testability –Recovery and history Page 24

© Hortonworks Inc Tez – Community Early adopters and contributors welcome –Adopters to drive more scenarios. Contributors to make them happen. Stay tuned for Tez meetups with deep dives on Tez architecture and using Tez Useful links –Work tracking: –Code: –High level design document and API specification: – Developer list: User list: Issues list: Page 25

© Hortonworks Inc Tez – Takeaways Distributed execution framework that works on computations represented as dataflow graphs Naturally maps to execution plans produced by query optimizers Execution architecture designed to enable dynamic performance optimizations at runtime Open source Apache project – your use-cases and code are welcome It works and is already being used by Hive Page 26

© Hortonworks Inc Tez Thanks for your time and attention! Questions? Page 27