Enterprise security for big data solutions on Azure HDInsight 9/21/2018 7:06 PM Enterprise security for big data solutions on Azure HDInsight Saurin Shah Sr. Program Manager @saurinms © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9/21/2018 7:06 PM Azure HDInsight © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Microsoft Tech Summit FY17 9/21/2018 7:06 PM Fully-managed Hadoop and Spark for the cloud. 99.9% SLA 100% Open Source Hortonworks data platform Clusters up and running in minutes Familiar BI tools, interactive open source notebooks Multiple IDE tooling support, including remote debugging 63% lower TCO than deploy your own Hadoop on-premises* Scale clusters on demand Secure Hadoop and Spark via Active Directory and Ranger Best in class monitoring and predictive operations via OMS Native Integration with leading ISVs Azure HDInsight Open source analytics service for the Enterprise *IDC study “The Business Value and TCO Advantage of Apache Hadoop in the Cloud with Microsoft Azure HDInsight” © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo Azure HDInsight cluster creation 9/21/2018 7:06 PM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Big Data security – rings of defense Microsoft Ignite 2016 9/21/2018 7:06 PM Big Data security – rings of defense Perimeter level security Virtual network Network security (i.e. firewalls) Gateway Service Tunneling Authentication Kerberos Active directory Authorization Hive policies File and folder level ACLS Data security Encryption @ rest HTTPS/TLS In-transit © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Customer Scenarios Perimeter Security 9/21/2018 7:06 PM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

I want my data to be private .. always 9/21/2018 7:06 PM I want my data to be private .. always © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Virtual Networks & Network Security Groups 9/21/2018 7:06 PM Virtual Networks & Network Security Groups Malicious user Azure blob storage HDInsight cluster in a VNET With NSG rules Azure data lake store © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Service Tunneling Azure blob storage HDInsight cluster in a VNET From VNET to Storage Azure blob storage HDInsight cluster in a VNET Azure data lake store

9/21/2018 7:06 PM Demo © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Customer Scenarios Authentication 9/21/2018 7:06 PM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

I want only authenticated users to see data 9/21/2018 7:06 PM I want only authenticated users to see data © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Multi-user authentication Gateway layer to validate user HDInsight cluster Username/password Azure blob storage Azure data lake store unauthenticated user

Multi-user authentication in Standard Clusters

Multi-user authentication in Premium Clusters Azure Active Directory With Domain Services enabled Sync users Fetch Kerberos tickets Azure blob storage HDInsight cluster Azure data lake store

Demo Multi-user authentication 9/21/2018 7:06 PM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Customer Scenarios Authorization – Role based access control 9/21/2018 7:06 PM Customer Scenarios Authorization – Role based access control © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9/21/2018 7:06 PM I want only privileged users to access sensitive data and perform privileged operations © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Authorization in Ambari portal

Authorization using Apache Ranger

Authorization using Apache Ranger Azure Active Directory With Domain Services enabled Ranger database Sync users Fetch Kerberos tickets Fetch authorization policies Username password HDInsight cluster

Auditing using Apache Ranger

Authorization using File and Folder level ACLs Azure Storage No built-in File & Folder ACLs Apache Ranger plug-in available Azure Data Lake Store Built-in File & Folder ACLs Seamless integration with built-in support* * Available only for users that do not have Multi-factor authentication setup.

Demo Authorization 9/21/2018 7:06 PM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Customer Scenarios Encryption of data 9/21/2018 7:06 PM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

I want my data to be encrypted .. Always (at rest and in-transit) 9/21/2018 7:06 PM I want my data to be encrypted .. Always (at rest and in-transit) © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Server-Side encryption Azure Storage Transparent Server-Side encryption using Microsoft managed keys Transparent Server-Side encryption using customer keys coming soon Azure Data Lake Store Transparent Server-Side encryption using Microsoft managed keys As well as customer managed keys.

In Transit encryption https https (TLS 1.2) Transparent Server Side

9/21/2018 7:06 PM To summarize … © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Authorization & Auditing 9/21/2018 7:06 PM Apache Ranger RBAC for Admin POSIX ACLs for Data Plane Server-Side encryption at rest HTTPS/TLS In-transit Azure Active Directory Kerberos authentication Virtual Networks Network Security Groups Service Tunneling Perimeter Security Multi-user Authentication Authorization & Auditing Data Encryption © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9/21/2018 7:06 PM “HDInsight as Big Data Platform has enabled our data engineers and scientists to focus on developing data and analytics products rather than managing infrastructure and troubleshooting day-day issues related to very large clusters. The heavy lifting of installing & managing clusters, providing robust security with Apache Ranger, data at rest encryption, monitoring and scaling up/down is taken care by HDInsight.  This platform is used for variety of use cases like real time streaming, machine learning, visualization, ETL. Overall a very positive experience with HDInsight engineering, product and support teams.” -- Navaljit Bhasin, Big Data Engineering Director, Honeywell © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Get started with Azure HDInsight Azure HDInsight overview https://azure.microsoft.com/en-us/services/hdinsight/   Azure HDInsight documentation https://docs.microsoft.com/en-us/azure/hdinsight/ Azure HDInsight Training https://aka.ms/hdinsighttraining  hdifeedback@microsoft.com

9/21/2018 7:06 PM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Externalize Ranger database 9/21/2018 7:06 PM Externalize Ranger database "admin-properties": { "audit_db_name": "[parameters('rangerDbName')]", "audit_db_user": "[parameters('rangerDbUser')]", "audit_db_password": "[parameters('rangerDbPassword')]", "db_name": "[parameters('rangerDbName')]", "db_user": "[parameters('rangerDbUser')]", "db_password": "[parameters('rangerDbPassword')]", "db_host": "[parameters('rangerDbServerName')]", "db_root_user": "", "db_root_password": "" }, "ranger-admin-site": { "ranger.jpa.jdbc.url": "[concat('jdbc:sqlserver://', parameters('rangerDbServerName'), ';databaseName=', parameters('rangerDbName'))]" "ranger-env": { "ranger_privelege_user_jdbc_url": "[concat('jdbc:sqlserver://', parameters('rangerDbServerName'), ';databaseName=', parameters('rangerDbName'))]" "ranger-hive-security" : { "ranger.plugin.hive.service.name" : "common_repo" } © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Multi-user authentication in Premium Clusters Azure Active Directory With ADFS enabled Sync users Fetch Kerberos tickets Azure blob storage HDInsight cluster Azure data lake store

Multi-user authentication in Premium Clusters Azure Active Directory With ADFS enabled Sync users Fetch Oauth 2 token Azure blob storage HDInsight cluster Azure data lake store

Customer Scenarios Monitoring of clusters 9/21/2018 7:06 PM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

I want to ensure my clusters are always healthy 9/21/2018 7:06 PM I want to ensure my clusters are always healthy © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Default Monitoring We monitor your clusters. Every 5 minutes, we will ensure these services are running If monitoring reveals that services are down, automated alert gets raised to our engineers. 99.9% SLA is guaranteed. Monitoring service Cluster available Data nodes running Certificate valid Node managers up Oozie running Job submission working MapReduce running Rstudio running

HDInsight Monitoring in Azure Log Analytics