1. Big Data for the SQL Eye Cindy Gross @SQLCindy

2. Look, it’s SQL! SELECT score, fun FROM toDo WHERE type = 'they pay me for this?'; @SQLCindy

4. And yet it’s more! CREATE EXTERNAL TABLE IF NOT EXISTS toDo (fun STRING, rank INT COMMENT 'rank the greatness', type STRING) COMMENT 'two tables walk into a bar....' ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE LOCATION '/data/demo/'; @SQLCindy

6. A mix of old and new -- read some data SELECT 'you cannot make me ', score, fun, type FROM toDo WHERE score <= 0 ORDER BY score; SELECT 'when can we ', score, fun, type FROM toDo WHERE score > 0 DISTRIBUTE BY score SORT BY score; @SQLCindy

8. Visual job graph @SQLCindy

9. That’s Hive folks! Hive on Hadoop on HDInsight on Azure Big Data in the cloud! @SQLCindy

10. Back to You Have you used Big Data? Azure? What questions do you have? What do you want to know by the end of this talk? What makes your projects go right or wrong? Will you use Big Data? @SQLCindy

11. Big Data Uses @SQLCindy

12. Part 3: Single Slide A leading game development studio that creates, develops, produces, and publishes a number of popular video games needed to analyze large amounts of in-game data that were unstructured. They chose Azure HDInsight, Data Factory, SQL Server on-premises, Power View, Power Query to do in- game analytics and understanding what gamers do during game-play and what campaigns they can run to influence in-game purchases. Finally, twitter sentiment is collected to correlate with sales.

13. Game Development Company Gaming A predominantly mobile-based game development company. While they are a mid-sized organization, they have partnered with media giants on various gaming projects Part 1: What They Did | In-game Analytics Challenge As a game development studio, they wanted to do in-game analytics to understand their players more and what they do in the games Solution Azure HDInsight (MapReduce and Storm), Service Bus, SQL Server for reporting Collects telemetry and logging data to gain in-game analytics: How many players using the game How many players invited their friends How far along did players get into the tutorial How many attempts did they make on one level/stage In-game Analytics Media tonic

14. BK1 Game Development Company Part 2: How They Did It | In-game Analytics How They Did It Collect data from games in Azure Blobs Game sends telemetry/logging data as JSON files Contains every action of user in the game Data is pushed to Azure Service Bus as real-time Tens of Gigabytes of data captured daily HDInsight picks up real-time data and processes From Service Bus, HDInsight processes using Apache Storm and MapReduce Constantly running experiments to determine insight A/B testing In-game metrics and analytics Spin up 32-node cluster nightly for four hours Output sent to SQL Server for BI Transfer data to SQL Server for BI In-game Analytics Service Bus SQL Server On-premises

15. Part 3: Single Slide A game development studio that wanted to do in-game analytics to understand their players more and what they do in their games. They chose Azure HDInsight including Storm in HDInsight so they can do near real-time in-game analytics of their users. Now, they can understand how many players are playing, how many are referring the game, how difficult a game level is, etc.

16. Typical Big Data Use Cases Smart meter monitoring Equipment monitoring Advertising analysis Life sciences research Fraud detection Healthcare outcomes Weather forecasting Natural resource exploration Social network analysis Churn analysis Traffic flow optimization Legal discovery Telemetry IT infrastructure optimization

17. It depends It’s mostly true

18. Hadoop Shines When…. Data exploration, analytics and reporting, new data-driven actionable insights Rapid iterating Unknown unknowns Flexible scaling Data driven actions for early competitive advantage or first to market Low number of direct, concurrent users Low cost data archival

19. Hadoop Anti-Patterns…. Replace system whose pain points don’t align with Hadoop’s strengths OLTP needs adequately met by an existing system Known data with a static schema Many end users Interactive response time requirements (becoming less true) Your first Hadoop project + mission critical system

20. You tell me… What is Big Data? What is Hadoop? What specific scenario would you use it for? @SQLCindy

21. Hortonworks HDP @SQLCindy

22. Business Scenarios Recommendations, customer churn, forecasting, etc. Perceptual Intelligence Face, vision Speech, text Personal Digital Assistant Cortana Dashboards and Visualizations Power BI Machine Learning and Analytics Azure Machine Learning Azure Stream Analytics Cortana Analytics Suite DATA Business apps Custom apps Sensors and devices INTELLIGENCEACTION People Automated Systems Big Data Stores Azure SQL Data Warehouse Information Management Azure Data Factory Azure Data Catalog Azure Event Hub Azure Data Lake Store Azure HDInsight (Hadoop) Azure Data Lake Analytics

23. Azure Data Lake - The Pieces @SQLCindy

24. Azure Data Lake store Azure Data Lake managed clusters Hadoop Cluster Azure Data Lake: store & managed clusters HDFS/WebHDFS API

25. HDInsight “managed Hadoop clusters” Input File Output File n1n2n3 n4 Hive/Pig/etc. job ADLA Account lots of containers YARN Layer U-SQL/Hive/Pig job ADLA “analytics service” HDInsight Cluster Storage (Blob or ADLS) Analytics: Two Form Factors 45

31. Azure Data Lake - The Action

35. Azure has so much more Go straight to the business code Scale storage and compute separately Open Source Linux Managed and unmanaged services Hybrid On-demand and 24x7 options SQL Server @SQLCindy

36. It’s a Polyglot Stream your data into a lake Pick the best compute for each task @SQLCindy

37. And it’s Fun! @SQLCindy

38. Big Data for the SQL Eye Cindy Gross @SQLCindy

39. And back to you @SQLCindy

40. What is Big Data? It Is Scale Out Enables elasticity Encourages exploration Faster data ingestion Lower TCO Empowers self-service BI and analytics Rapid time to insight It Is NOT A well-defined thing About volume, size A replacement for everything The answer to every problem

41. What is Hadoop? Conceptual View It Is A type of Big Data Just another data source A loose collection of open source code Distributed by many Handles loosely structured data Write once, read many It Is Not Actually a thing! The only way to do Big Data

42. Business Scenarios Recommendations, customer churn, forecasting, etc. Perceptual Intelligence Face, vision Speech, text Personal Digital Assistant Cortana Dashboards and Visualizations Power BI Machine Learning and Analytics Azure Machine Learning Azure Stream Analytics Cortana Analytics Suite DATA Business apps Custom apps Sensors and devices INTELLIGENCEACTION People Automated Systems Big Data Stores Azure SQL Data Warehouse Information Management Azure Data Factory Azure Data Catalog Azure Event Hub Azure Data Lake Store Azure HDInsight (Hadoop) Azure Data Lake Analytics

43. Microsoft Azure Data Lake YARN U-SQL Analytics Service HDInsight 1 1 1 1 1 1 1 1 1 1 1 1 Store HDFS