Motivation

Customer Trends Reporting  Insights, predictions, actions Static data  Dynamic intelligence Operational efficiency  Competitive advantage Technical Trends Data Storage Scarcity  Data Storage Abundance Operational Data  All data Highly Modeled Schema  Schema agility & exploratory analysis Relational Algebra  ML, Image Processing, Graph, Streaming Reporting  Insights, predictions, actions

Customer Stories

JustGiving is a global online social platform for giving that lets you raise money for a cause you care about through your network of friends. Their goal is to become "Facebook of Giving“. They chose Azure HDInsight, a Data Lake service to give a way to recommend to fundraisers about a potential fundraising goal, recommended causes that they might be interested in and other people in the social graph to add to their fundraising initiative.

One of the leaders in the development and management of renewable energy, infrastructure, water and services needed to understand data coming from their wind turbines/wind farms in an Internet of Things (IoT) scenario. They chose Azure Data Lake to work with SQL Server and Excel reports to generate analytics around power/consumption of each windmill turbine. They provide querying capabilities to their customers to understand consumption.

A government organization that handles finances, taxes, budget, income, and national debt for their country. Their tax department allows clients to uploads their digital documents (pay stubs, expenditure slips) and now have billions of documents uploaded. They chose Azure HDInsight, SQL Server, to run queries and to process the electronic invoices to gain insights. HDInsight is able to handle a peak of 150+ million invoices uploaded / day. They can now do fraud detection by understanding what people are doing and flagging/detecting anomalies.

The Tools

A hyper-scale repository for big data analytics workloads A Hadoop Distributed File System for the cloud No fixed limits on file size No fixed limits on account size Unstructured and structured data in their native format Massive throughput to increase analytic performance High durability, availability, and reliability Azure Active Directory access control LOB ApplicationsSocial Devices ClickstreamSensorsVideo Web Relational HDInsight ADL Analytics Machine Learning Spark R ADL Store Big Data Stores SQL Data Warehouse Data Lake Store

Elastic data warehouse as a service with enterprise-class features Petabyte scale with massively parallel processing Independent scaling of compute and storage—in seconds Transact-SQL queries across relational and non-relational data Full enterprise-class SQL Server experience Works seamlessly with Power BI, Machine Learning, HDInsight, and Data Factory Power BI App Service SQL Database SQL Data Warehouse Machine Learning Hadoop Intelligent App Big Data Stores SQL Data Warehouse Data Lake Store

Big data analytics made easy Analyze data of any kind and size Develop faster, debug and optimize smarter Interactively explore patterns in your data No learning curve—use U-SQL, Spark, Hive, HBase and Storm Managed and supported with an enterprise-grade SLA Dynamically scales to match your business priorities Enterprise-grade security with Azure Active Directory Built on YARN, designed for the cloud Data Lake Analytics SQL DWSQL DB Storage BlobsData Lake StoreSQL DB in a VM Machine Learning and Analytics HDInsight (Hadoop and Spark) Stream Analytics Data Lake Analytics Machine Learning

Comprehensive set of managed Apache big data projects Scale to petabytes on demand Process unstructured and semi-structured data Develop in Java,.NET, and more Skip buying and maintaining hardware Deploy in Windows or Linux Spin up an Apache Hadoop cluster in minutes Visualize your Hadoop data in Excel Easily integrate on-premises Hadoop clusters Core Engine Batch Map Reduce Script Pig SQL Hive NoSQL HBase Streaming Storm In-Memory Spark Machine Learning and Analytics HDInsight (Hadoop and Spark) Stream Analytics Data Lake Analytics Machine Learning

Machine Learning and Analytics HDInsight (Hadoop and Spark) Stream Analytics Data Lake Analytics Machine Learning Real-time stream processing in the cloud Perform real-time analytics for your Internet of Things solutions Stream millions of events per second Get mission-critical reliability and performance with predictable results Create real-time dashboards and alerts over data from devices and applications Correlate across multiple streams of data Use familiar SQL-based language for rapid development Event Hubs Blob Storage Stream Analytics SQL Database Event Hubs Power BI Blob Storage Table Storage

Compose and orchestrate data services at scale PREPARE TRANSFORM & ANALYZE PUBLISH SQL DATA CONSUMPTION INGEST SQL <> SQL DATA SOURCES { } SQL Create, schedule, orchestrate, and manage data pipelines Visualize data lineage Connect to on-premises and cloud data sources Monitor data pipeline health Automate cloud resource management Move relational data for Hadoop processing Transform with Hive, Pig, or custom code Information Management Event Hubs Data Catalog Data Factory

Easily build, deploy, and share predictive analytics solutions Simple, scalable, cutting edge. A fully managed cloud service that enables you to easily build, deploy, and share predictive analytics solutions. Deploy in minutes. Azure Machine Learning means business. You can deploy your model into production as a web service that can be called from any device, anywhere and that can use any data source. Publish, share, monetize. Share your solution with the world in the Gallery or on the Azure Marketplace. Machine Learning and Analytics HDInsight (Hadoop and Spark) Stream Analytics Data Lake Analytics Machine Learning

Build applications that understand people Faces, images, emotion recognition and video intelligence Spoken language processing, speaker recognition, custom speech recognition Natural language processing, sentiment and topics analysis, spelling errors Complex tasks processing, knowledge exploration, intelligent recommendations Bing engine capabilities for Web, Autosuggest, Image, Video and News Intelligence Cortana Bot Framework Cognitive Services

Putting it All Together

JustGiving, Non-Profit How They Did It | Recommendation engine How They Did It Collect data in Azure Blobs HDInsight processes data for insights Generates a real-time recommendation SQL Server On-premises Agent Azure Blobs Azure HDInsight Activity Feeds Give Graph Azure Tables Web API Website + Event store Service Bus Serves results Azure Cache

Server Leading Computer Manufacturer / Retailer How They Did It | Analyzes Clickstream and Provide Real-time Recommendations Online How They Did It Collect clickstream data In tab separated text files Adding 22 new files per hour ~5-18 MB/file Currently 1TB and growing Spin up Hadoop Use Hive scripts because of SQL-like syntax Extracts click behavior like buys, additions to carts, reviews etc. and assigns scores Jobs run hourly Currently 8-nodes with plans to 16 Clickstream, Recommendation BK1 HDInsight Cluster AzureML Event Hub NRT AzureML Azure SQL DE Blog Storage NoSQL Storage IaaS VM MB ase Targeted Persisted Storage Blog Storage Blog Storage Visitor Information Service Omn Iture Product Catalog Website.com

BK1 Industrial automation company partnering with multinational oil company How They Did It | IoT internet-connected sensors to generate analytics for proactive maintenance How They Did It Collect data from internet-collected sensors Tens of thousands data points per second Interpolate time-series prior to analysis Stored raw sensor data in Blobs every 5 minutes Use Hadoop to execute scripts and Data Factory to orchestrate Hive and Pig scripts orchestrated by Data Factory Data resulting from scripts loaded in SQL Database Queries detect site anomalies to indicate maintenance/tuning Produced dashboards with role-based reporting Azure Machine Learning, SSRS, Power BI for O365 Provide users with customizable interface View current and historical data (day-to-day operations, asset performance over time, etc.) Leveraged Azure Mobile Notification Hub for real-time notifications, alarms, or important events Use Azure ML to predict Understand which pumps, run at what speeds, maximized water supply while minimizing energy use IoT, Analytics

Now What

Explore Cortana Intelligence today 1. Learn more Keep up on the latest cloud-based analytics news on our blog ▶blog 3. Find or become a partner2. Get trained Participate in Data Science and Machine Learning Essentials course through edX ▶ course Watch the Cortana Intelligence Workshop ▶ Workshop Become Become a Cortana Intelligence Partner ▶ Connect with a partner Connect with a partner who can help tailor Cortana Intelligence to your needs ▶ Visit the Cortana Intelligence page ▶page