3. What’s New in SQL Server 2014 since SQL Server 2012 3 Mission Critical PerformanceFaster Insights from Any DataPlatform for Hybrid Cloud PERFORMANCE & SCALE In-Memory OLTP Enhanced In-Memory ColumnStore for DW Buffer Pool Extension to SSDs Enhanced Query Processing Resource Governor adds IO governance SysPrep at cluster level Predictable performance with tiering of compute, network, and storage with Windows Server 2012 R2 HIGH AVAILABILITY Enhanced AlwaysOn, with 8 secondaries and Replica Wizard Delayed Durability Clustered Shared Volume support, VHDX support (Windows Server 2012 R2) Manage on-premises and cloud apps (System Center 2012 R2) SECURITY Redefined engineering security processes CC certification at High Assurance Level New permissions enhance separation of duties Backup encryption support PROGRAMMABILITY Query optimization enhancements EASY ACCESS TO DATA, BIG & SMALL Power Query Windows Azure HDInsight Service & Microsoft HDInsight Server (preview) Parallel Data Warehouse with Polybase POWERFUL INSIGHTS WITH FAMILIAR TOOLS Excel in Office and Power BI in Office 365— Power Pivot and Power View Power Map Preview for Excel Mobile BI Apps for iPad & Windows 8 devices HYBRID CLOUD SOLUTIONS Backup to Windows Azure Cloud Disaster Recovery Extend on-premises apps to the cloud EASY ON-RAMP TO THE CLOUD New Windows Azure Deployment Wizard SQL Server in a Windows Azure Virtual Machine Windows Azure Data Availability Windows Azure SQL Database service Mission Critical PerformanceFaster Insights from Any DataPlatform for Hybrid Cloud PERFORMANCE & SCALE In-Memory OLTP Enhanced In-Memory ColumnStore for DW Buffer Pool Extension to SSDs Enhanced Query Processing Resource Governor adds IO governance SysPrep at cluster level Predictable performance with tiering of compute, network, and storage with Windows Server 2012 R2 HIGH AVAILABILITY Enhanced AlwaysOn, with 8 secondaries and Replica Wizard Delayed Durability Clustered Shared Volume support, VHDX support (Windows Server 2012 R2) Manage on-premises and cloud apps (System Center 2012 R2) SECURITY Enhanced separation of duty CC certification at High Assurance Level for 2014 Backup encryption support PROGRAMMABILITY Query optimization enhancements EASY ACCESS TO DATA, BIG & SMALL Power Query Windows Azure HDInsight Service Analytics Platform System (PDW V2) POWERFUL INSIGHTS WITH FAMILIAR TOOLS Power BI in Office 365 Power Map for Excel Mobile interfaces for Power BI HYBRID CLOUD SOLUTIONS Simplified backup to Windows Azure Support for backup of previous versions of SQL Server to Windows Azure Cloud back-up encryption support Simplified cloud Disaster Recovery with AlwaysOn replicas in Windows Azure VMs EASY ON-RAMP TO THE CLOUD New Windows Azure Deployment UI for SQL Server Larger SQL Server VMs and memory sizes now available in Windows Azure

4. What’s New in SQL Server 2014 since SQL Server 2008 R2 4 Mission Critical PerformanceFaster Insights from Any DataPlatform for Hybrid Cloud PERFORMANCE & SCALE In-Memory OLTP Enhanced In-Memory ColumnStore for DW Support for 640 logical proc. & 4 TB memory Support to 15,000 partitions Resource Governor IO governance Buffer Pool Extension to SSDs Query optimization enhancements SysPrep at cluster level Predictable performance with tiering of compute, network, and storage with Windows Server 2012 R2 HIGH AVAILABILITY SQL Server AlwaysOn Delayed Durability Recovery Advisor Windows Server Core Live Migration Online Operations enhancements Clustered Shared Volume support, VHDX support (Windows Server 2012 R2) Manage on-premises and cloud apps (System Center 2012 R2) SECURITY User-Defined Server Roles Default Schema for Groups SQL Server Audit SQL Server Fine-grained Auditing Enhanced separation of duty CC certification at High Assurance Level Backup encryption support PROGRAMMABILITY SQL Server Data Tools Local DB runtime (Express) Data-tier Application Component project template Data-Tier Application Framework (DAC Fx) Query optimization enhancements Interoperability support (ADO.NET, ODBC, JDBC, PDO, ADO APIs and.NET C/C++, Java, Linux, and PHP platforms) T-SQL enhancements Enhanced support for ANSI SQL standards Transact-SQL Static Code Analysis tools Transact-SQL code snippets Intellisense Unstructured & Complex Data Support FileTable built on FILESTREAM Remote Blob Storage with SharePoint 2010 Statistical Semantic Search Spatial features, including Full Globe & arcs Large user-defined data types MANAGEABILITY Distributed Replay Contained Database Authentication System Center Management Pack for SQL Server 2012 Windows PowerShell 2.0 support Multi-server Management with SQL Server Utility Control Point Data-Tier Application Component ACCESS ANY DATA Power Query Windows Azure HDInsight Service Analytics Platform System (PDW V2) Mash up data from different sources, such as Oracle & Hadoop INSIGHTS WITH FAMILIAR TOOLS Power BI in Office 365 Power Map for Excel Mobile interfaces for Power BI COMPLETE BI SOLUTION SQL Server BI Edition HA for StreamInsight, complex event processing BI Semantic Model SQL Server Data Tools support for BI Change Data Capture for Oracle Analysis Services Import PowerPivot models into Analysis Services Enhancements on productivity, performance Reporting Services Power View Configurable reporting alerts Reporting as SharePoint Shared Service Report Builder 3.0 Data Quality Services Build organizational knowledge base Connect to 3rd party data cleansing providers Master Data Services Master Data Hub Master Data Services Add-in for Microsoft Excel Integration Services Graphical tools in SSIS Extensible object model SSIS as a Server Broader data integration with more sources; DB vendors, cloud, Hadoop Pipeline improvements HYBRID CLOUD SOLUTIONS Simplified backup to Windows Azure Support for backup of previous versions of SQL Server to Windows Azure Cloud back-up encryption support Simplified cloud Disaster Recovery with AlwaysOn replicas in Windows Azure VMs EASY ON-RAMP TO THE CLOUD New Windows Azure Deployment UI for SQL Server Larger SQL Server VMs and memory sizes now available in Windows Azure DAC enhancements: Import/export with Windows Azure SQL Database COMPLETE AND CONSISTENT FROM ON-PREM TO CLOUD SQL Server Data Tools License Mobility (with SA) Resource Governor enhancements Snapshot backups to Windows Azure via SQL Server Management Studio

5. What’s New in SQL Server 2014 since SQL Server 2008 5 Mission Critical PerformanceFaster Insights from Any DataPlatform for Hybrid Cloud PERFORMANCE & SCALE In-Memory OLTP Enhanced In-Memory ColumnStore for DW Support for 640 logical proc. & 4 TB memory Support to 15,000 partitions Resource Governor IO governance Buffer Pool Extension to SSDs Query optimization enhancements SysPrep at cluster level Predictable performance with tiering of compute, network, and storage with Windows Server 2012 R2 Data Compression with USC-2 Unicode support Backup Compression HIGH AVAILABILITY SQL Server AlwaysOn Delayed Durability Recovery Advisor Windows Server Core Live Migration Online Operations enhancements Clustered Shared Volume support, VHDX support (Windows Server 2012 R2) Manage on-premises and cloud apps (System Center 2012 R2) SECURITY User-Defined Server Roles Default Schema for Groups SQL Server Audit SQL Server Fine-grained Auditing Enhanced separation of duty CC certification at High Assurance Level Backup encryption support PROGRAMMABILITY SQL Server Data Tools Local DB runtime (Express) Data-tier Application Component project template Data-Tier Application Framework (DAC Fx) Query optimization enhancements Interoperability support (ADO.NET, ODBC, JDBC, PDO, ADO APIs and.NET C/C++, Java, Linux, and PHP platforms) T-SQL enhancements Enhanced support for ANSI SQL standards Transact-SQL Static Code Analysis tools Transact-SQL code snippets Intellisense Unstructured & Complex Data Support FileTable built on FILESTREAM Remote Blob Storage with SharePoint 2010 Statistical Semantic Search Spatial features, including Full Globe & arcs Large user-defined data types MANAGEABILITY Distributed Replay Contained Database Authentication System Center Management Pack for SQL Server 2012 Windows PowerShell 2.0 support Multi-server Management with SQL Server Utility Control Point Data-Tier Application Component ACCESS ANY DATA Power Query Windows Azure HDInsight Service Analytics Platform System (PDW V2) Mash up data from different sources, such as Oracle & Hadoop INSIGHTS WITH FAMILIAR TOOLS Power BI in Office 365 Power Map for Excel Mobile interfaces for Power BI COMPLETE BI SOLUTION SQL Server BI Edition HA for StreamInsight, complex event processing BI Semantic Model SQL Server Data Tools support for BI Change Data Capture for Oracle Analysis Services Import PowerPivot models into Analysis Services Enhancements on productivity, performance Reporting Services Power View Configurable reporting alerts Reporting as SharePoint Shared Service Report Builder 3.0 Data Quality Services Build organizational knowledge base Connect to 3rd party data cleansing providers Master Data Services Master Data Hub Master Data Services Add-in for Microsoft Excel Integration Services Graphical tools in SSIS Extensible object model SSIS as a Server Broader data integration with more sources; DB vendors, cloud, Hadoop Pipeline improvements HYBRID CLOUD SOLUTIONS Backup to Windows Azure Cloud Disaster Recovery Extend on-premises apps to the cloud EASY ON-RAMP TO THE CLOUD New Windows Azure Deployment UI for SQL Server Larger SQL Server VMs and memory sizes now available in Windows Azure DAC enhancements: Import/export with Windows Azure SQL Database COMPLETE AND CONSISTENT FROM ON-PREM TO CLOUD SQL Server Data Tools License Mobility (with SA) Resource Governor enhancements Snapshot backups to Windows Azure via SQL Server Management Studio

7. What’s New in SQL Server 2014 since SQL Server 2005 7 Mission Critical PerformanceFaster Insights from Any DataPlatform for Hybrid Cloud PERFORMANCE & SCALE In-Memory OLTP Enhanced In-Memory ColumnStore for DW Support for 640 logical proc. & 4 TB memory Support to 15,000 partitions Resource Governor IO governance Buffer Pool Extension to SSDs Query optimization enhancements SysPrep at cluster level Predictable performance with tiering of compute, network, and storage with Windows Server 2012 R2 Data Compression with USC-2 Unicode support Backup Compression HIGH AVAILABILITY SQL Server AlwaysOn Database Mirroring Failover Clustering Database Snapshots Delayed Durability Recovery Advisor Windows Server Core Live Migration Online Operations Clustered Shared Volume support, VHDX support (Windows Server 2012 R2) Manage on-premises and cloud apps (System Center 2012 R2) SECURITY User-Defined Server Roles Default Schema for Groups SQL Server Audit Transparent Data Encryption Extensible Key Management Standards-based Encryption SQL Server Fine-grained Auditing Enhanced separation of duty CC certification at High Assurance Level Backup encryption support PROGRAMMABILITY SQL Server Data Tools Local DB runtime (Express) Data-tier Application Component project template Data-Tier Application Framework (DAC Fx) Query optimization enhancements Interoperability support (ADO.NET, ODBC, JDBC, PDO, ADO APIs and.NET C/C++, Java, Linux, and PHP platforms) T-SQL enhancements Enhanced support for ANSI SQL standards Transact-SQL Static Code Analysis tools Transact-SQL code snippets Intellisense Data Support FILESTREAM data type FileTable built on FILESTREAM Remote Blob Storage with SharePoint 2010 Spatial data support Full Text Search for unstructured files Statistical Semantic Search Large user-defined data types Programmability Support Support for LINQ and ADO.NET Entity Framework CLR Integration and ADO.NET Object Services MANAGEABILITY Distributed Replay Contained Database Authentication System Center Management Pack for SQL Server 2012 Windows PowerShell 2.0 support Multi-server Management with SQL Server Utility Control Point Data-Tier Application Component Multi-server Management with SQL Server Utility Control Point Data-Tier Application Component Policy-Based Management SQL Server Performance Data Collector Query enhancements SMTP mail for secure DB email w/o Outlook ACCESS ANY DATA Power Query Windows Azure HDInsight Service Analytics Platform System (PDW V2) Mash up data from different sources, such as Oracle & Hadoop INSIGHTS WITH FAMILIAR TOOLS Power BI in Office 365 Power Map for Excel Mobile interfaces for Power BI COMPLETE BI SOLUTION SQL Server BI Edition StreamInsight BI Semantic Model SQL Server Data Tools BI Development Studio Microsoft Visual Studio- based report dev tools Change Data Capture for Oracle Analysis Services Import PowerPivot models into Analysis Services Enhancements on productivity, performance Cube design tools, block computations, and write- back to MOLAP Reporting Services Power View Configurable reporting alerts Reporting as SharePoint Shared Service Report Builder 3.0 Report Designer Report Manager Data Quality Services Build organizational knowledge base Connect to 3rd party data cleansing providers Master Data Services Master Data Hub Master Data Services Add-in for Microsoft Excel Integration Services Graphical tools in SSIS Extensible object model SSIS as a Server Broader data integration with more sources; DB vendors, cloud, Hadoop Pipeline improvements Persistent lookups High-performance connectors Data profiling tool HYBRID CLOUD SOLUTIONS Backup to Windows Azure Cloud Disaster Recovery Extend on-premises apps to the cloud EASY ON-RAMP TO THE CLOUD New Windows Azure Deployment UI for SQL Server Larger SQL Server VMs and memory sizes now available in Windows Azure DAC enhancements: Import/export with Windows Azure SQL Database COMPLETE AND CONSISTENT FROM ON-PREM TO CLOUD SQL Server Data Tools License Mobility (with SA) Resource Governor enhancements Snapshot backups to Windows Azure via SQL Server Management Studio

10. Particle collider produces 1 petabyte of data a second…. THAT IS BIG DATA.

13. Cloud

14. Volume Variety Velocity Relational Data

16. Hadoop cluster

19. Name Node Data Node Hadoop cluster Data Node Hadoop Cluster Master Worker Nodes

20. Data Node 5Data Node 4Data Node3Data Node 2 Data Node 1 Block 1 Block 3 Block 2 Block 1 Block 3 Block 2 Block 3 Block 2 Block 1 Default Replication factor = 3 Default Block Size = 64MB

21. 21 NameNode Standby Namenode DataNode (heartbeat, balancing, replication, etc.) nodes write to local disk namespace

22. BlobContainerAccount http://.blob.core.windows.net/ / Pages/ Blocks contoso PIC01.JPG Block/Page PIC02.JPG images VID1.AVIvideos

24. MapReduce Layer HDFS Layer hadoop-namenode hadoop- datanode1 hadoop- datanode2 hadoop- datanode3 hadoop- datanode4 24 JobTracker TaskTracker DataNode NameNode MapReduce Layer HDFS Layer

25. 25 … … … … Output Reducer Reducer Reducer Sort and group by key DataNode Mapper Mapper Mapper Mapper

26. © Hortonworks Inc. 2013 Windows OS CloudVMAppliance Page 26 PLATFORM SERVICES HADOOP CORE HORTONWORKS DATA PLATFORM (HDP) Distributed Storage & Processing Enterprise Readiness DATA SERVICES OPERATIONAL SERVICES Store, Process and Access Data HCATALOG HIVE PIG SQOOP Oozie Mahout Python HBase Ambari

28. Highly fault tolerant Distributed batch processing at the largest possible scale today Relatively easy to write distributed computations over very large amounts of data MR framework removes burden of dealing with failures from programmer. Lots of Tools in Ecosystem Open source/no licensing costs Schema on read vs write (just put it there) Schema embedded in application code A lack of shared schema makes sharing data between applications difficult Not good at DBMS power like cost optimizer, indexes, deep statistics, optimized HW, adanved buffer pools, consistency. No declarative query language. Large clusters require a lot of management. High datacenter costs for large clusters

30. Background Research done by Gray System Lab lead by Technical Fellow David DeWitt High-level Goals Seamless Integration with Hadoop via regular T-SQL Enhancing PDW query engine to process data coming from HDFS Fully parallelized query processing, bi-directional import and export between PDW and HDFS Integration with various Hadoop implementations Hadoop on Windows Server, Hortonworks, and Cloudera

31. Parallel data transfers Control Node Compute Node Name Node Data Node PDW Appliance Hadoop cluster Compute Node Data Node Hadoop Cluster

32. Direct parallel data access between PDW Compute Nodes and Hadoop Data Nodes Introducing “structure” on the “unstructured” data Hadoop cluster Hadoop HDFS DB SQL in, results out Hadoop HDFS DB SQL in, results stored in HDFS

33. Cost-based decision on how much data needs to be pushed to PDW  Stats on PDW Control Node SQL operations on HDFS data pushed into Hadoop as MapReduce jobs Hadoop cluster HDFS PDWHadoop Results7 2 Map job 5 HDFS blocks DB 3 4 6 SQL1 MapReduce

38. Polybase

40. A New Approach to Customer Connection Store first, come up with the questions later Easily track gameplay quality, user behavior, etc by time Short timeline, small team, not Hadoop experts Results in common tools like Excel Start at scale of 100s of TB

41. Architecture – Use Cloud Building Blocks Blob Storage or In Memory (Landing Zone) Blob Storage (Persistent Storage) HDInsight Clusters (Hive, Pig, etc) REST Sqoop Self-Service Analytics Reporting / DW Curator Optimized for write throughput -Many small blobs -Raw/binary format -Data kept until curated -Azure Blob Storage if persisted -Azure Queues & Workers for in memory Optimized for query efficiency -Optimized size (combine blobs) -Cleansed/masked -Partitioned -Well-defined, semi-structured data Use Case Specific & General Processing -Data governance requirements (PII scrub) -Aggregate for efficient storage -Publish to real-time consumers and long term storage (Hadoop) Other

43. Windows Azure HDInsight Service “on demand, dedicated virtual clusters”  Portal  SDK & CLI  100% Apache Hadoop (HDP)  Seamless integration with Azure platform

44. http://azure.microsoft.com/en-us/pricing/member-offers/msdn-benefits/ $150/mo credits with MSDN http://azure.microsoft.com/en-us/pricing/member-offers/msdn-benefits/ http://azure.microsoft.com/en-us/documentation/articles/hdinsight-get- started/ http://azure.microsoft.com/en-us/documentation/articles/hdinsight-get- started/ https://cwiki.apache.org/confluence/display/Hive/LanguageManual www.pluralsight.com