HDInsight makes Hadoop Easy Agenda Collect and Load Big Data HDInsight Clusters Cluster Customizations Visual Studio Tooling Tools

Collect and load big data Prerequisites Blob storage concepts Data types and sources Performance and scalability Administration Reliability Security Data processing Pre-processing data Serialization and compression Choosing tools and technologies Tools

Collect and load big data 9/18/2018 Collect and load big data Interactive PowerShell Hadoop command line AzCopy Visual Studio Cloudberry 3rd party application Azure blob HDFS HDInsight 10 01 Interactive Relational Data Streaming data Automated Server log files Relational Data Azure Data Factory Apache Sqoop SQL Server Integration Services PolyBase in APS Streaming data Apache Storm on HDInsight Azure Stream Analytics Reactive extensions (RX) Custom or 3rd party application Server log files Apache Flume SQL Server Integration Services Custom solution using the Azure SDK Automated Azure Data Factory PowerShell with task scheduler SQL Server Integration Services Custom solution using the Azure SDK © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9/18/2018 Blob storage concepts http://<account>.blob.core.windows.net/<container>/<blobname> Store large amounts of unstructured text or binary data with the fastest read performance Account Contoso Container Images Video Blob PIC01.JPG VID1.AVI PIC02.JPG Page/blocks Block/Page Access a highly scalable, durable, and available file system Expose blobs publically over HTTP Securely lock down permissions © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Data types and sources devices and sensors geo-location data 9/18/2018 zzzz web clickstreams devices and sensors geo-location data social media server logs Azure blob storage Data types and sources Extract data in a form that can be easily consumed Stage data before submitting it to a big data cluster Submit data accurately to cluster storage, including data conversions Choose the right tool for your data If using HBase, use an appropriate technique to upload it © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Collect and load big data Prerequisites Blob storage concepts Data types and sources Performance and scalability Administration Reliability Security Data processing Pre-processing data Serialization and compression Choosing tools and technologies Tools

9/18/2018 Reliability Upload tool should handle transient connectivity and transmission failures Monitor upload to detect failures early Record each stage in a process that raised an error Scale out with multiple upload instances Validate the data before you upload © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Security Protect data at rest and in motion with secure authentication 9/18/2018 Security Protect data at rest and in motion with secure authentication Leverage local security policies and features Employ a robust auditing and monitoring process Remove non-essential sensitive data Encrypt essential sensitive data HDInsight clusters © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Collect and load big data 9/18/2018 Collect and load big data Prerequisites Blob storage concepts Data types and sources Performance and scalability Administration Reliability Security Data processing Pre-processing data Serialization and compression Choosing tools and technologies Tools © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Serialization and compression 9/18/2018 Serialization and compression Tools for Avro serialization and compression SDK is available from NuGet Tools provided by the codec supplier, eg. GZip and BZip2 for compression Use the classes in the NET Framework to perform GZip and DEFLATE compression on your source files Create a query job that is configured to write output in compressed form using one of the built-in codecs Azure SDK Azure SDK HDInsight compression libraries Format Codec Extension Splittable DEFLATE org.apache.hadoop.io.compress.DefaultCodec .deflate No GZip org.apache.hadoop.io.compress.GzipCodec .gz BZip2 org.apache.hadoop.io.compress.BZip2Codec (this codec is not enabled by default in configuration) .bz2 Yes © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Interactive data ingestion UI-based tools Cloudberry Explorer, Storage Explorer The hadoop dfs CopyFromLocal [source] [destination] command Codec supplier tools GZip and BZip2. Command line tool AzCopy to upload large files PowerShell commands Take advantage of the Azure PowerShell cmdlets Tools

Handling streaming data Apache Storm on HDInsight Open-source framework that runs on a Hadoop cluster to capture streaming data Azure Stream Analytics Stream Processing Service on Azure offering ease of use to consume and process event streams. Custom event or stream capture solution Feeds data into the cluster data store in real time or batches Microsoft StreamInsight Complex event processing (CEP) engine with a framework API for building apps that consume and process event streams Tools

Loading relational data Sqoop Extract required data from a table, view, or query in the source database and save the results as a file in your cluster storage. Interfaces that support connectivity to big data clusters Microsoft Analytics Platform System (APS) contains PolyBase, to expose a SQL-based interface for accessing data stored in Hadoop and HDInsight Tools

HDInsight Clusters Tools 9/18/2018 © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

HDInsight cluster architecture HTTP traffic ODBC/JDBC WebHCatalog Oozie Ambari Azure VNet Secure gateway AuthN HTTP Proxy Azure Storage Highly available Head nodes Worker nodes x

HDInsight service entry points 9/18/2018 HDInsight service entry points HDInsight cluster Remote desktop Oozie Command line REST SDK ODBC Hive Pig M/R Query console PowerShell Excel Visual Studio plugin © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Cluster Customizations 9/18/2018 Cluster Customizations Tools © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Cluster customization options 9/18/2018 Cluster customization options Hive/Oozie Metastore Storage accounts ScriptAction Via Azure portal HDInsight cluster provisioning states Ready for deployment Accepted Cluster storage provisioned AzureVM configuration Config values JAR file placement in cluster Via scripting / SDK Customize cluster? Running Cluster operational Configuring HDInsight No Timed Out Error Yes RDP to cluster, update config files (non-durable) Ad hoc Cluster customization (custom script running © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Visual Studio Tooling Tools 9/18/2018 © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Visual Studio tooling Ships with the Azure SDK Supported for VS 2012, 2013 and 2015 Enables Hive query authoring, submission and debugging Navigate Linked Resources Create Hive Tables Run Hive Queries View Hive Jobs Hive Script Local Validation IntelliSense Support for Hive (Preview) Table creation and schema management are also supported

Get started today! For more information visit: http://azure.microsoft.com/en-us/services/hdinsight/

© 2014 Microsoft Corporation. All rights reserved © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.