Presentation is loading. Please wait.

Presentation is loading. Please wait.

Agenda for the Day 01 – Introduction to APS & Demo

Similar presentations


Presentation on theme: "Agenda for the Day 01 – Introduction to APS & Demo"— Presentation transcript:

1 Agenda for the Day 01 – Introduction to APS & Demo
02 – APS Hardware, Software & Architecture 03 – Distribution Theory & Design ----- Meal Break 04 – APS Data Loading 05 - HDI Region Software, Tools & Polybase 06 – BI Integration & End-to-End Demo

2 Microsoft Analytics Platform System 01 – Introduction To APS
Dan Kogan | Microsoft ​Sr. Product Marketing Manager Jesse Fountain| Microsoft ​WW TSP Lead Sanjay Soni | Microsoft ​Sr. Product Marketing Manager September 20, 2018

3 AGENDA APS Introduction APS High-Level Architecture
Key Features / Differentiators APS Integrated Demonstration

4 The traditional data warehouse
Microsoft Analytics Platform System 9/20/2018 The traditional data warehouse BI and analytics Dashboards Reporting Data warehouse ETL Data sources OLTP ERP CRM LOB © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5 The traditional data warehouse
Microsoft Analytics Platform System 9/20/2018 The traditional data warehouse BI and analytics Dashboards Reporting Real-time data 2 Data warehouse ETL Increasing data volumes 1 New data sources and types 3 Cloud-born data 4 Data sources OLTP ERP CRM LOB Non-relational data Devices Web Sensors Social © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6 Microsoft Analytics Platform System
9/20/2018 Introducing the Microsoft Analytics Platform System The turnkey modern data warehouse appliance Enterprise-ready Big Data Next-generation performance at scale Engineered for optimal value Relational and non-relational data in a single appliance Enterprise-ready Hadoop Integrated querying across Hadoop and PDW using T-SQL Direct integration with Microsoft BI tools such as Microsoft Excel Near real-time performance with In-Memory Columnstore Ability to scale out to accommodate growing data Removal of data warehouse bottlenecks with MPP SQL Server Concurrency that fuels rapid adoption Industry’s lowest data warehouse appliance price per terabyte Value through a single appliance solution Value with flexible hardware options using commodity hardware © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

7 Financial Services In a US$1.2 million win, a Canadian bank is transforming operations and staying competitive by choosing Microsoft APS and self-service BI tools for all employees Challenge To stay competitive, a recently rebranded Canadian bank wanted a better analytics and business intelligence (BI) solution Faced rapid data growth from 5 TB to 25 TB over the next three years Wanted to give 1,000 employees more self-service BI features and access to data anytime, anywhere from mobile devices Needed better insight into customer behavior, but the existing Major RDBMS data warehouse was costly and difficult to maintain. Lacked the analytics capabilities and scalability needed to support business growth and transformation Hoped to control costs—maintaining BI tools from multiple vendors had become cumbersome and expensive Solution Chose the Microsoft Analytics Platform System and is standardizing its BI environment with technologies such as Microsoft SQL Server Analysis Services and SQL Server Reporting Services Provides real-time, self-service BI access to all employees, driving better business decisions companywide Improves insight to identify new revenue opportunities as well as the reasons existing customers leave the bank Gains more scalable, flexible solution with potential for cloud and big-data integration

8 Why an Appliance?

9 Microsoft Analytics Platform System
9/20/2018 Blazing-fast performance MPP and In-Memory Columnstore for next-generation performance 9 Up to 100x faster queries Updateable clustered columnstore vs. table with customary indexing Up to 15x more compression Columnstore index representation C1 C2 C3 C4 C5 C6 Parallel query execution Store data in columnar format for massive compression Load data into or out of memory for next-generation performance with up to 60% improvement in data loading speed Updateable and clustered for real-time trickle loading Query Results © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

10 Microsoft Analytics Platform System
9/20/2018 Software and hardware re-designed together Built-in software features lead to efficient hardware Windows Server 2012 built-in software drives efficient hardware Virtualization for streamlining hardware footprint High-end storage features built into software Storage Spaces Windows virtualized storage Storage pool Reduced costs Built-in best practices Physical storage © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11 Microsoft Analytics Platform System
9/20/2018 Scaling out your data to petabytes Scale-out technologies in the Analytics Platform System Scale out Multiple nodes with dedicated CPU, memory, and storage Ability to incrementally add hardware for near-linear scale to multiple petabytes Ability to handle query complexity and concurrency at scale No “forklift” of prior warehouse to increase capacity Ability to scale out HDInsight and PDW PDW / HDInsight PDW / HDInsight PDW / HDInsight PDW / HDInsight PDW / HDInsight PDW / HDInsight PDW 0 terabytes 6 petabytes 11 © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12 Microsoft Analytics Platform System
9/20/2018 Connecting islands of data with PolyBase Bringing Hadoop point solutions and the data warehouse together for users and IT Select… Result set Provides a single T-SQL query model for PDW and Hadoop with rich features of T- SQL, including joins without ETL Uses the power of MPP to enhance query execution performance Supports Windows Azure HDInsight to enable new hybrid cloud scenarios Provides the ability to query non-Microsoft Hadoop distributions, such as Hortonworks and Cloudera SQL Server Parallel Data Warehouse Microsoft Azure HDInsight PolyBase Hortonworks for Windows and Linux Cloudera Microsoft HDInsight © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

13 Recent POC Results Public Transport Company (vs. Major RDBMS)
200 user concurrency test, 7 years worth of data ETL Batch window test Major RDBMS APS 2 hours 12 minutes 4 hours 3 minutes Advertising Company (vs. Hadoop) 120 node Hadoop cluster vs. 10 node APS Hadoop queries executed through Hive Hadoop APS 7 minutes 1 minute 5 minutes 1 second Internal Testing (vs. SQL Server Standard) SQL Server on SMP (32-core, 256GB, 152 HDDs) SQL Server on SMP (80-core, 1TB, 192 HDDs) APS-HP Single rack (128-core, 2TB, 288 HDDs) 48 min 34 min < 1 min

14 Retail A global retailer of consumer electronics that reported US$42 billion in annual sales in 2013 is improving operational efficiency companywide by using APS to power real-time reporting for 80,000 stores. Challenge A leading global retailer of consumer electronics wanted to help its thousands of outlets operate more efficiently, but faced a daily struggle to deliver timely performance reports to 80,000 stores Needed better access to operational data, including point-of-sale purchases, orders, and transport updates for approximately 125,000 retail employees and 1,200 corporate managers Wanted analytics and BI tools that would be easy to use for employees in multiple roles working in diverse locations, including store floors worldwide Expanding its existing MPP Data Warehouse architecture to meet the SLAs would be too costly Solution The retailer replaced its MPP Data Warehouse and BI Toolset with the Microsoft Analytics Platform System in an end-to-end solution that includes Microsoft BI tools and integrates with an existing Hadoop cluster Improves operational performance with real-time, on-demand insight into multiple business processes and data sources, easily handling 40,000 reports per hour Gains unprecedented insight into at least five years of performance information Scales easily and affordably to support new workloads and additional functionality Cuts annual power costs by millions of dollars by consolidating with APS and using built-in technology like clustered columnstore index

15 APS High-Level Architecture

16 APS Logical Architecture
Control Server A.K.A. ‘The Brains’’ 1. Optimizer creates parallel query plan 2. Each compute server runs a portion of the query in parallel 3. Data is combined and returned to user User Query Optimizer Metadata Statistics Data Movement Services Compute Server Compute Server A.K.A. ‘The Brawn’ Compute Server DMS Balanced Storage DMS Balanced Storage DMS Balanced Storage

17 Automatic MapReduce pushdown
Microsoft Analytics Platform System 9/20/2018 Automatic MapReduce pushdown Source systems Hadoop / Data Lake (Cloudera, Hortonworks, HDInsight) Analytics / Ad-hoc / Visualization SQL Server Data Marts Reporting Services Analysis Services Microsoft HDInsight SQL Server Parallel Data Warehouse PolyBase MapReduce T-SQL APS Day / Hour / Minute Refresh © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18 APS High-Availability No Single Point-Of-Failure
Infiniband 2 Ethernet 2 Control Host Compute Host 1 Compute 1 VM Compute 1 VM FAB AD VMM CTL Infiniband 1 Infiniband 1 Ethernet 1 Ethernet 1 Compute Host 2 Failover Host Compute 2 VM

19 the Analytics Platform System Appliance
9/20/2018 Concurrency that fuels rapid adoption Great performance with mixed workloads Analytics Platform System ETL/ELT with SSIS, DQS, MDS ERP CRM LOB APPS SQL Server SMP Intra-Day CRTAS Link Table PDW HDInsight PolyBase ETL/ELT with DWLoader Near real-time Reporting and cubes Columnstore Real-Time ROLAP / MOLAP DirectQuery Hadoop / Big Data Polybase SNAC BI Tools Ad hoc queries Fast ad hoc

20 Key Features & Differentiators

21 Microsoft Analytics Platform System
9/20/2018 How PolyBase works Direct and parallelized HDFS access Enhancing the Data Movement Service (DMS) of APS to allow direct communication between HDFS data nodes and PDW compute nodes Non-relational data Social apps Sensor and RFID Mobile apps Web apps Hadoop Relational data Traditional schema-based data warehouse applications PDW Regular T-SQL Results External table External data source External file format Enhanced PDW query engine HDFS bridge © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22 Microsoft Analytics Platform System
9/20/2018 Big Data insights for anyone New insights with familiar tools through native Microsoft BI integration Takes advantage of high adoption of Excel, Power View, PowerPivot, and SQL Server Analysis Services Minimizes IT intervention for discovering data with tools such as Microsoft Excel Everyone else using Microsoft BI tools Offers Hadoop tools like MapReduce, Hive, and Pig for data scientists Enables DBA and power users to join relational and Hadoop data with T-SQL Power users Data scientist © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

23 End-to-end Solution Azure Market Place Alerts, Notifications
Big Data Sources (Raw, Unstructured) SQL Server StreamInsight Data & Compute Intensive App Summarize & Load via Polybase SQL Server FTDW Data Marts Business Insights Sensors Hortonworks or Cloudera Hadoop SQL Server Reporting Services Devices HDInsight on Windows Azure HDInsight on Windows Server SQL Server Anlalytical Platform Server Interactive Reports Integrate/Enrich Bots SQL Server Analysis Server Azure Market Place ETL with SSIS, DQS, MDS Performance Scorecards Crawlers ERP CRM LOB APPS Source Systems

24 + ~ 3-4 months ~ 3 weeks APS Reduces TIMELINE SQL Server SQL Server
Infrastructure 5-8 weeks Installation 2-4 weeks System Design 2-3 weeks D/W Requirements Data Warehouse H/W Design Hadoop Requirements Hadoop H/W Design Infrastructure Acquisition DW H/W Installation Hadoop H/W Installation DW S/W Installation Hadoop S/W Installation Networking Database Design Physical Data Design Index Design Configuration Settings Performance Design 1 week SQL Server ~ 3 weeks D/W Requirements Hadoop Requirements Analytics Platform System

25 Microsoft’s big picture approach
DATA TYPES DATA SOURCES Apps, Biz process, ERP, CRM INFRASTRUCTURE DM SQL Server ETL TOOLS Known, known Excel Power Pivot Power View Data Explorer USERS Business Structured MM/DD/YYYY Known, known Known, known Known, known Analytical Platform Server Polybase Semi-structured web logs, RFID, “the internet of things” Known, unknown Machines and other devices Pig, R, map reduce, algorithms, machine learning Known, unknown “Data Scientists” “Quants” HDInsight Known, unknown Known, unknown Un-structured text, video, audio Unknown, unknown Collaboration and social , blogs, documents Indexing engine algorithms Unknown, unknown Everyone Unknown, unknown Unknown, unknown

26 Transportation and Logistics
With APS, a major city’s transit agency is improving the travel experience for millions with near-real-time insight into data collected from billions of trips across multiple types of public transportation. Challenge To improve customer experience, a government agency needed a better analytics solution to gain insight into billions of annual journeys across multiple types of public transit Agency responsible for a transport network needed to improve travel for millions Wanted to collect data from a recently implemented contactless card system, but lacked a sufficiently scalable platform Needed to look at data from multiple trips, including 1 billion subway journeys across 270 stations, and 2 billion bus journeys on 8,500 buses Subway journeys expected to increase 30% by 2020, and bus journeys are growing at the fastest rate since 1946 Solution Implemented the Microsoft Analytics Platform System Provides more granular analytics and data on congestion and travel disruption to help improve customer experience Gains 10x faster performance while querying 42x more data, and makes information accessible to 100x more users Automates refunds—identifies incorrect billing and immediately refunds the money to the customer accounts, avoiding the inconvenience and cost of customers’ contacting a representative to manually correct the error Collects more big data—integrates sensor information from a rental bicycle system called Barclays Cycle Hire. Integrates social media such as Twitter to better understand customers’ reactions. Other possible sources of data include WiFi, CCTV, and law-enforcement agencies.

27 Demo

28 Microsoft Cybercrime Center APS/BI Demo
9/20/2018 Microsoft Cybercrime Center APS/BI Demo Sanjay Soni Sr. Technical Product Marketing Manager (for Power BI + End to end BA demos) Microsoft © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

29 The Microsoft Cybercrime Center

30 The impact of cybercrime
Online Child Exploitation The NCMEC has reviewed more than 90 million images and videos of child pornography.*** Financial Fraud 53% of the world’s securities exchanges were targeted in 2012 Cybercrime costs consumers $113 billion a year* 1 in 5 small and medium enterprises are targeted by cyber criminals** Every second, 12 people are victims of cybercrime – nearly 400 million every year* 50% of online adults have been victims in the past year * 2013 Norton Report ** National Cyber Security Alliance *** National Center for Missing and Exploited Children

31 A new era in the fight against cybercrime
Microsoft Digital Crimes Unit Microsoft Consulting Services Trustworthy Computing Government Affairs Public Sector Cybercrime Center Security, Reliability, Privacy Solutions, Initiatives, Innovations Policy, Legislation Risk Assessment, Cybersecurity Services Industry Partners Criminal Law Enforcement Government Proactive Disruption We work with our customers and partners to proactively target online criminals MALICIOUS SOFTWARE CRIMES IP CRIMES CHILD EXPLOITATION Viruses Trojans Worms Botnets Counterfeiting End-User Piracy Child Abuse Images Sex Trafficking

32 Botnet Analytics – Architecture
Certs ISPs Others Partners / Subscribers Internet with billions of devices Internet with billions of devices Azure Machine Learning [Sinkhole] SQL Server Azure HDInsight Microsoft Analytics Platform System (APS) Visualizations & insights Excel & Power BI Processing 200M+ transactions per day and growing, the Microsoft Cybercrime Center APS & BI are powered by:

33 Use of Analytics Platform System/ Parallel Data Warehouse
Cybercrime analytics team uses PDW Microsoft’s data scientists support cybercrime operations by using PDW to run advanced algorithms We can aggregate 500 million rows of data a day into tables that bring even more speed to our ability to respond to rapidly evolving threats. 566,773,255 calls a day into the sinkhole from over 25 million distinct IP address representing many more than that number of computers, because some IP addresses are corporate consolidation points. Massive performance improvements: Our data is now responding at the speed of thought. Cybercrime analytics team uses PDW Microsoft’s data scientists support cybercrime operations by using PDW to run advanced algorithms We can aggregate 500 million rows of data a day into tables that bring even more speed to our ability to respond to rapidly evolving threats. 566,773,255 calls a day into the sinkhole from over 25 million distinct IP address representing many more than that number of computers, because some IP addresses are corporate consolidation points. Massive performance improvements: Our data is now responding at the speed of thought. Query Before PDW After using PDW 1 2 days, 8 minutes <3 minutes 2 4 hours 2 seconds

34 Microsoft Analytics Platform System
9/20/2018 © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION. © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.


Download ppt "Agenda for the Day 01 – Introduction to APS & Demo"

Similar presentations


Ads by Google