1. ©2014 LinkedIn Corporation. All Rights Reserved. Gobblin’ Big Data with Ease Lin Qiao Data Analytics Infra @ LinkedIn

2. ©2014 LinkedIn Corporation. All Rights Reserved. Overview Challenges What does Gobblin provide? How does Gobblin work? Retrospective and lookahead

3. ©2014 LinkedIn Corporation. All Rights Reserved. Overview Challenges What does Gobblin provide? How does Gobblin work? Retrospective and lookahead

4. ©2014 LinkedIn Corporation. All Rights Reserved. Perception

5. ©2014 LinkedIn Corporation. All Rights Reserved. Reality 5

6. ©2014 LinkedIn Corporation. All Rights Reserved. Challenges @ LinkedIn Large variety of data sources Multi-paradigm: streaming data, batch data Different types of data: facts, dimensions, logs, snapshots, increments, changelog Operational complexity of multiple pipelines Data quality Data availability and predictability Engineering cost

7. ©2014 LinkedIn Corporation. All Rights Reserved. Open source solutions sqoop p flume p morphline p RDBMS vendor- specific connectors p aegisthus logstash Camus

8. ©2014 LinkedIn Corporation. All Rights Reserved. Goals Unified and Structured Data Ingestion Flow – RDBMS -> Hadoop – Event Streams -> Hadoop Higher level abstractions – Facts, Dimensions – Snapshots, increments, changelog ELT oriented – Minimize transformation in the ingest pipeline

9. ©2014 LinkedIn Corporation. All Rights Reserved. Central Ingestion Pipeline

10. ©2014 LinkedIn Corporation. All Rights Reserved. Overview Challenges What does Gobblin provide? How does Gobblin work? Retrospective and lookahead

11. ©2014 LinkedIn Corporation. All Rights Reserved. Gobblin Usage @ LinkedIn Business Analytics – Source data for, sales analysis, product sentiment analysis, etc. Engineering – Source data for issue tracking, monitoring, product release, security compliance, A/B testing Consumer product – Source data for acquisition integration – Performance analysis for email campaign, ads campaign, etc.

12. ©2014 LinkedIn Corporation. All Rights Reserved. Key Features  Horizontally scalable and robust framework  Unified computation paradigm  Turn-key solution  Customize your own Ingestion

13. ©2014 LinkedIn Corporation. All Rights Reserved. Scalable and Robust Framework 13 Scalable Centralized State Management State is carried over between jobs automatically, so metadata can be used to track offsets, checkpoints, watermarks, etc. Jobs are partitioned into tasks that run concurrently Fault Tolerant Framework gracefully deals with machine and job failures Query Assurance Baked in quality checking throughout the flow

14. ©2014 LinkedIn Corporation. All Rights Reserved. Unified computation paradigm Common execution flow Common execution flow between batch ingestion and streaming ingestion pipelines Shared infra components Shared job state management, job metrics store, metadata management.

15. ©2014 LinkedIn Corporation. All Rights Reserved. Turn Key Solution Built-in Exchange Protocols Existing adapters can easily be re-used for sources with common protocols (e.g. JDBC, REST, SFTP, SOAP, etc.) Built-in Source Integration Fully integrated with commonly used sources including MySQL, SQLServer, Oracle, SalesForce, HDFS, filer, internal dropbox) Built-in Data Ingestion Semantics Covers full dump and incremental ingestion for fact and dimension datasets. Policy driven flow execution & tuning Flow owners just need to specify pre-defined policy for handling job failure, degree of parallelism, what data to publish, etc.

16. ©2014 LinkedIn Corporation. All Rights Reserved. Customize Your Own Ingestion Pipeline Extendable Operators Configurable Operator Flow Operators for doing extraction, conversion, quality checking, data persistence, etc., can be implemented or extended against common API. Configuration allows for multiple plugin points to add in customized logic and code

17. ©2014 LinkedIn Corporation. All Rights Reserved. Overview Challenges What does Gobblin provide? How does Gobblin work? Lookahead

18. ©2014 LinkedIn Corporation. All Rights Reserved. Under the Hood

19. ©2014 LinkedIn Corporation. All Rights Reserved. Computation Model Gobblin standalone – single process, multi-threading – Testing, small data, sampling Gobblin on Map/Reduce – Large datasets, horizontally scalable Gobblin on Yarn – Better resource utilization – More scheduling flexibilities

20. ©2014 LinkedIn Corporation. All Rights Reserved. Scalable Ingestion Flow 20 Source Work Unit Work Unit Work Unit Data Publisher ExtractorConverter Quality Checker Writer ExtractorConverter Quality Checker Writer ExtractorConverter Quality Checker Writer Task

21. ©2014 LinkedIn Corporation. All Rights Reserved. Sources Determines how to partition work -Partitioning algorithm can leverage source sharding -Group partitions intelligently for performance Creates work-units to be scheduled Source Work Unit Publisher Extractor Converter Quality Checker Writer

22. ©2014 LinkedIn Corporation. All Rights Reserved. Job Management Job execution states – Watermark – Task state, job state, quality checker output, error code Job synchronization Job failure handling: policy driven 22 State Store Job run 1 Job run 3 Job run 2

23. ©2014 LinkedIn Corporation. All Rights Reserved. Gobblin Operator Flow Extract Schema Extract Record Convert Record Check Record Data Quality Write Record Convert Schema Check Task Data Quality Commit Task Data 23

24. ©2014 LinkedIn Corporation. All Rights Reserved. Extractors Source Work Unit Publisher Extractor Converter Quality Checker Writer Specifies how to get the schema and pull data from the source Return ResultSet iterator Track high watermark Track extraction metrics

25. ©2014 LinkedIn Corporation. All Rights Reserved. Converters Allow for schema and data transformation – Filtering – projection – type conversion – Structural change Composable: can specify a list of converters to be applied in the given order Source Work Unit Publisher Extractor Converter Quality Checker Writer

26. ©2014 LinkedIn Corporation. All Rights Reserved. Quality Checkers Ensure quality of any data produced by Gobblin Can be run on a per record, per task, or per job basis Can specify a list of quality checkers to be applied – Schema compatibility – Audit check – Sensitive fields – Unique key Policy driven – FAIL – if the check fails then so does the job – OPTIONAL – if the checks fails the job continues – ERR_FILE – the offending row is written to an error file 26 Source Work Unit Publisher Extractor Converter Quality Checker Writer

27. ©2014 LinkedIn Corporation. All Rights Reserved. Writers Writing data in Avro format onto HDFS – One writer per task Flexibility – Configurable compression codec (Deflate, Snappy) – Configurable buffer size Plan to support other data format (Parquet, ORC) Source Work Unit Publisher Extractor Converter Quality Checker Writer

28. ©2014 LinkedIn Corporation. All Rights Reserved. Publishers Determines job success based on Policy. -COMMIT_ON_FULL_SUCCESS  COMMIT_ON_PARTIAL_SUCCESS Commits data to final directories based on job success. Task 1 Task 2 Task 3 File 1 File 2 File 3 Tmp Dir File 1 File 2 File 3 Final Dir File 1 File 2 File 3 Source Work Unit Publisher Extractor Converter Quality Checker Writer

29. ©2014 LinkedIn Corporation. All Rights Reserved. Gobblin Compaction Dimensions: – Initial full dump followed by incremental extracts in Gobblin – Maintain a consistent snapshot by doing regularly scheduled compaction Facts: – Merge small files 29 Ingestion HDFS Compaction

30. ©2014 LinkedIn Corporation. All Rights Reserved. Overview Challenges What does Gobblin provide? How does Gobblin work? Retrospective and lookahead

31. ©2014 LinkedIn Corporation. All Rights Reserved. Gobblin in Production > 350 datasets ~ 60 TB per day Salesforce Responsys RightNow Timeforce Slideshare Newsle A/B testing LinkedIn JIRA Data retention 31 Production Instances Data Volume

32. ©2014 LinkedIn Corporation. All Rights Reserved. Lesson Learned Data quality has a lot more work to do Small data problem is not small Performance optimization opportunities Operational traits

33. ©2014 LinkedIn Corporation. All Rights Reserved. Gobblin Roadmap Gobblin on Yarn Streaming Sources Gobblin Workbench with ingestion DSL Data Profiling for richer quality checking Open source in Q4’14 33

34. ©2014 LinkedIn Corporation. All Rights Reserved.