1. Big Data 101 Seriously, it is just 101

2. 781 254 4096 pareshmotiwala@gmail.com @pareshmotiwala
Paresh Motiwala, PMP ®
@pareshmotiwala

3. BIG Data 101 Who should attend DBAs CIO Marketing peeps Developers
Big Data Enthusiasts
Who should not attend

4. Big data 101
Let’s grab a byte
Brontobyte

5. BIG Data 101

6. Big data 101 Misc info on Big Data Sources Definition Privacy concerns
Data Lake
Storing- Hadoop
Processing – MapReduce
Presentation
Data Science and Scientists
Few Hadoop stacks
Summary

7. So why should I care about this?
Data is the new Electricity (Satya Nadella, Spring 2016)
Companies Generate data, Distribute, Meter, and Use it
Where is data stored?
Current: SQL Server, Oracle, Teradata, DB2, Netezza,
Open Source Databases; Casandra, MySQL, MongoDB
Unstructured: Hadoop, Spark, Data Lakes
What type of data is stored?
Traditional: Rows and Columns
Big Data Explosion: Images, streaming data, internet-connected devices (IoT),
Machine data
Source: Microsoft

8. Big Data is driving transformative changes
Traditional
Big Data
Relational data
with highly modeled schema
All data
with schema agility
Data
characteristics
Costs
Specialized HW
Commodity HW
Culture
Operational reporting
Focus on rear-view analysis
Experimentation leading to intelligent action With machine learning, graph, a/b testing
Source: Microsoft

9. Big data 101 Sources Cell Phones Social Media Credit Cards GPSs IoT
Wearables

10. Big data 101

11. Big Data 101
Value

12. Big data 101 Desired Properties: Robustness- Fault Tolerance
Low Latency
Scalability
Generalization
Extensibility
Ad hoc Queries
Minimal Maintenance
Debuggability

13. WAS CREATED IN PAST 2 YEARS
Big data 101
Flow
Collection
Pre-processing
Hygiene
Intervention
Visualization
Analysis
OVER 90% OF TODAY’S DATA
WAS CREATED IN PAST 2 YEARS

14. Big data 101 5 Rs of Data Quality Relevancy Recency Range Robustness
Reliability

15. Big data 101 Privacy of Data If I collect the data, is it mine?
Ownership Vs Rights
Share Answers not Data
OpAl (
Enigma (more resilient and secure data systems using secure multiparty computation and secret sharing over blockchain)
Let them know
Why you are collecting
What you are collecting

16. Big data 101 FIPP- Fair Information Privacy Principles
Individual Control
Transparency
Respect for Context
Security
Access and Accuracy
Focused Collection
FERPA- Family Education Rights and Privacy Act

17. Big data 101 What is a data lake? ---Courtesy : James serra
the Parallel Data Warehouse Appliance
4/11/2018
Big data 101 What is a data lake? ---Courtesy : James serra
A storage repository, usually Hadoop, that holds a vast amount of raw data in its native format until it is needed.
A place to store unlimited amounts of data in any format inexpensively, especially for archive purposes
Allows collection of data that you may or may not use later: “just in case”
A way to describe any large data pool in which the schema and data requirements are not defined until the data is queried: “just in time” or “schema on read”
Complements EDW and can be seen as a data source for the EDW – capturing all data but only passing relevant data to the EDW
Frees up expensive EDW resources (storage and processing), especially for data refinement
Allows for data exploration to be performed without waiting for the EDW team to model and load the data (quick user access)
Some processing in better done with Hadoop tools than ETL tools like SSIS
Easily scalable
Also called bit bucket, staging area, landing zone or enterprise data hub (Cloudera)
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. Big data 101 The “data lake” Uses A Bottoms-Up Approach
Ingest all data regardless of requirements
Store all data
in native format without schema definition
Do analysis
Using analytic engines like Hadoop
Devices
Social
Batch queries
Devices
LOB apps
Video
Interactive queries
Social
LOB applications
Real-time analytics
Sensors
Web
Sensors
Video
Relational
Machine Learning
Web
Clickstream
Data warehouse
Relational
Clickstream
Data Lake quickly turns into a data swamp if you don’t invest in data quality
Courtesy : James Serra

19. Big data 101
Doug Cutting
and
Mike Cafarella
In 2005

20. Big data 101
Benefits of hadoop

21. Big data 101

22. BIG DATA 101

23. Big data 101 MapReduce Map –Sends Queries Reduce – Collects Results
Job Tracker
Task Tracker
YARN

24. Big data 101

25. Base Architecture : Big Data Advanced Analytics Pipeline
4/11/2018 8:53 AM
Data Sources
Ingest
Prepare
(normalize, clean, etc.)
Analyze
(stat analysis, ML, etc.)
Publish
(for programmatic consumption, BI/visualization)
Consume
(Alerts, Operational Stats, Insights)
OnPrem Data
Azure Services
Near Realtime Data Analytics Pipeline using Azure Steam Analytics
Machine Learning
(Anomaly Detection)
Data Stream
Telemetry
Event Hub
Stream Analytics
(real-time analytics)
Live / real-time data stats, Anomalies and aggregates
PowerBI dashboard
Data
in Motion
Data
at Rest
Interactive Analytics and Predictive Pipeline using Azure Data Factory
Realtime Readings and Operational Data
HDI Custom ETL
Aggregate /Partition
Machine Learning
Local DB Sensor Readings
Local DB Logs
Customer MIS
dashboard of predictions / alerts
(Replaced by Azure SQL)
Legacy
Azure Storage Blob
Azure SQL
(Predictions)
Historic Laser Data (1 time drop)
Fault and Maintenance Data (1 time drop)
Scheduled hourly transfer using Azure Data Factory
Big Data Analytics Pipeline using Azure Data Lake
Sensor Readings
Device Health
dashboard of operational stats
Azure Data Lake Storage
Azure Data Lake Analytics
(Big Data Processing)
Azure SQL
Operational Logs
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

26. Vision for Big Data and Data Warehousing
Bing SMB Advertisers – Search Ads
4/11/2018
Vision for Big Data and Data Warehousing
Data Warehouse
“Big Data”
Microsoft Azure
Microsoft Azure
Cloud
VMs
HDInsight
Data Lake
Devices
Relational
Sensors
Video
LOB applications
Web
Social
Clickstream
VMs
SQL DW
Comprehensive
Connected
Choice
Azure Data Factory +
Federated Query
Microsoft SQL Server
Your data
Your workload
Your business
Your way
On-Premises
APS
SQL Server
HDP
APS

27. Big Data 101
Presentation R
Python
Power BI
Power BI Desktop

28. Big data 101
Data Science and Scientist

29. BIG DATA 101

30. Big data 101

31. Big data 101

32. Big data 101 Summary: Misc info on Big Data Sources Definition
Privacy concerns
Data Lake
Storing- Hadoop
Processing – MapReduce
Presentation
Data Science and Scientists
Few Hadoop stacks

33. Big data 101 - Conclusion SQL Server is the best Relational Database
The world is much bigger than any one relational database
What is your company’s data strategy?
What is your company’s cloud strategy?
Learn adjacent technologies that will make you valuable.
Power BI?
Hadoop?
NoSQL?

34. Someday Big Data will just become data Thank You

35. Paresh Motiwala, PMP ® pareshmotiwala@gmail.com
@pareshmotiwala

36. Big data 101 Paresh Motiwala, PMP ® http://www.datasciencecentral.com/
BIBLIOGRAPHY –
0mOCwxJ6B_OxTlpevxJNAa7GfCLd3l
architecture-big-data-and-hadoop-training/114
MIT Big Data Analytics Course
Data Lake presentation by James Serra
Future of Data…..(or something like that) by George Walters
Big Data Analytics with Microsoft HDInsight in 24 Hours
Paresh Motiwala, PMP ®