1. Big Data and HADOOP

2. BIG DATA AND HADOOP

3. BIG DATA AND HADOOP Paresh Motiwala, PMP ® 781 254 4096
DBA Manager at Nuance Communications
Chapter Leader: PASS DBA VC, NESQL, Boston_BI, PASSDBA VC, PASS PD
Co-organizer: Providence SQL Saturday, Global Azure Bootcamp

5. BIG DATA AND HADOOP Who should attend DBAs CIO Marketing peeps
Developers
Big Data Enthusiasts
Who should not attend

6. BIG DATA AND HADOOP
Let’s grab a byte
Brontobyte

7. BIG DATA AND HADOOP

8. BIG DATA AND HADOOP Misc info on Big Data Sources- Bread Crumbs
Definition
Privacy concerns
Data Lake
Storing- Hadoop
Processing – MapReduce
Presentation
Data Science and Scientists
Few Hadoop stacks
Summary

9. BIG DATA AND HADOOP So why should I care about this?
Data is the new Electricity (Satya Nadella, Spring 2016)
Companies Generate data, Distribute, Meter, and Use it
Where is data stored?
Current: SQL Server, Oracle, Teradata, DB2, Netezza,
Open Source Databases; Casandra, MySQL, MongoDB
Unstructured: Hadoop, Spark, Data Lakes
What type of data is stored?
Traditional: Rows and Columns
Big Data Explosion: Images, streaming data, internet-connected devices (IoT),
Machine data
BIG DATA AND HADOOP
Source: Microsoft

10. Big Data: driving transformative changes
Traditional
Big Data
Relational data
with highly modeled schema
All data
with schema agility
Data
characteristics
Costs
Specialized HW
Commodity HW
Culture
Operational reporting
Focus on rear-view analysis
Experimentation leading to intelligent action With machine learning, graph, a/b testing
Source: Microsoft

11. Big Data: Decision making
Today’s
Big Data
Rearview Mirror
Forward Looking
Effect
< 10%
Any and All
Data Used
Batch, Incomplete and Disjointed
Real-time, Correlated, Governed
Quality
Purpose
Business Monitoring
Business Optimization
Source: The Big Data by Schmarzo

12. BIG DATA AND HADOOP Sources – Bread Crumbs Cell Phones Social Media
Credit Cards
GPSs
IoT
Wearables

13. BIG DATA AND HADOOP

14. BIG DATA AND HADOOP
Value

15. BIG DATA AND HADOOP Desired Properties: Robustness- Fault Tolerance
Low Latency
Scalability
Generalization
Extensibility
Ad hoc Queries
Minimal Maintenance
Debuggability

16. WAS CREATED IN PAST 2 YEARS
BIG DATA AND HADOOP
Flow
Collection
Pre-processing
Hygiene
Intervention
Visualization
Analysis
OVER 90% OF TODAY’S DATA
WAS CREATED IN PAST 2 YEARS

17. BIG DATA AND HADOOP 5 Rs of Data Quality Relevancy Recency Range
Robustness
Reliability

18. BIG DATA AND HADOOP Privacy of Data If I collect the data, is it mine?
Ownership Vs Rights
Share Answers not Data
Let them know
Why you are collecting
What you are collecting

19. BIG DATA AND HADOOP FIPP- Fair Information Privacy Principles
Individual Control
Transparency
Respect for Context
Security
Access and Accuracy
Focused Collection
FERPA- Family Education Rights and Privacy Act

20. BIG DATA AND HADOOP What is a data lake? ---Courtesy : James serra
the Parallel Data Warehouse Appliance
11/11/2018
BIG DATA AND HADOOP What is a data lake? ---Courtesy : James serra
A storage repository, usually Hadoop, that holds a vast amount of raw data in its native format until it is needed.
A place to store unlimited amounts of data in any format inexpensively, especially for archive purposes
Allows collection of data that you may or may not use later: “just in case”
A way to describe any large data pool in which the schema and data requirements are not defined until the data is queried: “just in time” or “schema on read”
Complements EDW and can be seen as a data source for the EDW – capturing all data but only passing relevant data to the EDW
Frees up expensive EDW resources (storage and processing), especially for data refinement
Allows for data exploration to be performed without waiting for the EDW team to model and load the data (quick user access)
Some processing in better done with Hadoop tools than ETL tools like SSIS
Easily scalable
Also called bit bucket, staging area, landing zone or enterprise data hub (Cloudera)
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. BIG DATA AND HADOOP The “data lake” Uses A Bottoms-Up Approach
Ingest all data regardless of requirements
Store all data
in native format without schema definition
Do analysis
Using analytic engines like Hadoop
Devices
Social
Batch queries
Devices
LOB apps
Video
Interactive queries
Social
LOB applications
Real-time analytics
Sensors
Web
Sensors
Video
Relational
Machine Learning
Web
Clickstream
Data warehouse
Relational
Clickstream
Data Lake quickly turns into a data swamp if you don’t invest in data quality
Courtesy : James Serra

22. BIG DATA AND HADOOP
Doug Cutting
and
Mike Cafarella
In 2005

23. BIG DATA AND HADOOP
Benefits of hadoop

24. BIG DATA AND HADOOP

25. BIG DATA AND HADOOP
Data Lake
Big Data

26. BIG DATA AND HADOOP

27. BIG DATA AND HADOOP MapReduce Map –Sends Queries
Reduce – Collects Results
Job Tracker
Task Tracker
YARN

28. BIG DATA AND HADOOP

29. Base Architecture : Big Data Advanced Analytics Pipeline
11/11/2018 1:56 PM
Data Sources
Ingest
Prepare
(normalize, clean, etc.)
Analyze
(stat analysis, ML, etc.)
Publish
(for programmatic consumption, BI/visualization)
Consume
(Alerts, Operational Stats, Insights)
OnPrem Data
Azure Services
Near Realtime Data Analytics Pipeline using Azure Steam Analytics
Machine Learning
(Anomaly Detection)
Data Stream
Telemetry
Event Hub
Stream Analytics
(real-time analytics)
Live / real-time data stats, Anomalies and aggregates
PowerBI dashboard
Data
in Motion
Data
at Rest
Interactive Analytics and Predictive Pipeline using Azure Data Factory
Realtime Readings and Operational Data
HDI Custom ETL
Aggregate /Partition
Machine Learning
Local DB Sensor Readings
Local DB Logs
Customer MIS
dashboard of predictions / alerts
(Replaced by Azure SQL)
Legacy
Azure Storage Blob
Azure SQL
(Predictions)
Historic Laser Data (1 time drop)
Fault and Maintenance Data (1 time drop)
Scheduled hourly transfer using Azure Data Factory
Big Data Analytics Pipeline using Azure Data Lake
Sensor Readings
Device Health
dashboard of operational stats
Azure Data Lake Storage
Azure Data Lake Analytics
(Big Data Processing)
Azure SQL
Operational Logs
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

30. Vision for Big Data and Data Warehousing
Bing SMB Advertisers – Search Ads
11/11/2018
BIG DATA AND HADOOP
Data Warehouse
“Big Data”
Microsoft Azure
Microsoft Azure
Vision for Big Data and Data Warehousing
Cloud
VMs
HADOOP
Data Lake
Devices
Relational
Sensors
Video
LOB applications
Web
Social
Clickstream
VMs
SQL DW
Azure Data Factory +
Federated Query
Microsoft SQL Server
On-Premises
APS
SQL Server
HDP
APS

31. BIG DATA AND HADOOP
Presentation R
Python
Power BI
Power BI Desktop

32. BIG DATA AND HADOOP
Data Science and Scientist

33. BIG DATA AND HADOOP

34. BIG DATA AND HADOOP

35. Big Data101

36. BIG DATA AND HADOOP Summary: Misc info on Big Data Sources Definition
Privacy concerns
Data Lake
Storing- Hadoop
Processing – MapReduce
Presentation
Data Science and Scientists
Few Hadoop stacks

37. BIG DATA AND HADOOP- Conclusion
SQL Server is the best Relational Database
The world is much bigger than any one relational database
What is your company’s data strategy?
What is your company’s cloud strategy?
Learn adjacent technologies that will make you valuable.
Power BI?
Hadoop?
NoSQL?

38. Someday Big Data will just become data Thank You
BIG DATA AND HADOOP
Someday Big Data will just become data Thank You

39. Paresh Motiwala, PMP ® pareshmotiwala@gmail.com
@pareshmotiwala

40. BIG DATA AND HADOOP http://www.datasciencecentral.com/
BIBLIOGRAPHY –
0mOCwxJ6B_OxTlpevxJNAa7GfCLd3l
components-and-architecture-big-data-and- hadoop-training/114
MIT Big Data Analytics Course
Data Lake presentation by James Serra
Future of Data…..(or something like that) by George Walters
Big Data Analytics with Microsoft HADOOP in 24 Hours