1. Enterprise security for big data solutions on Azure HDInsight
9/21/2018 7:06 PM
Enterprise security for big data solutions on Azure HDInsight
Saurin Shah
Sr. Program Manager
@saurinms
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4. 9/21/2018 7:06 PM
Azure HDInsight
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. Microsoft Tech Summit FY17
9/21/2018 7:06 PM
Fully-managed Hadoop and Spark for the cloud. 99.9% SLA
100% Open Source Hortonworks data platform
Clusters up and running in minutes Familiar BI tools, interactive open source notebooks
Multiple IDE tooling support, including remote debugging
63% lower TCO than deploy your own Hadoop on-premises* Scale clusters on demand
Secure Hadoop and Spark via Active Directory and Ranger
Best in class monitoring and predictive operations via OMS Native Integration with leading ISVs
Azure HDInsight
Open source analytics service for the Enterprise
*IDC study “The Business Value and TCO Advantage of Apache Hadoop in the Cloud with Microsoft Azure HDInsight”
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Demo Azure HDInsight cluster creation 9/21/2018 7:06 PM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Big Data security – rings of defense
Microsoft Ignite 2016
9/21/2018 7:06 PM
Big Data security – rings of defense
Perimeter level security
Virtual network
Network security (i.e. firewalls)
Gateway
Service Tunneling
Authentication
Kerberos
Active directory
Authorization
Hive policies
File and folder level ACLS
Data security
rest
HTTPS/TLS In-transit
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

8. Customer Scenarios Perimeter Security 9/21/2018 7:06 PM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. I want my data to be private .. always
9/21/2018 7:06 PM
I want my data to be private .. always
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. Virtual Networks & Network Security Groups
9/21/2018 7:06 PM
Virtual Networks & Network Security Groups
Malicious user
Azure blob storage
HDInsight cluster in a VNET
With NSG rules
Azure data lake store
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. Service Tunneling Azure blob storage HDInsight cluster in a VNET
From VNET to Storage
Azure blob storage
HDInsight cluster in a VNET
Azure data lake store

14. 9/21/2018 7:06 PM
Demo
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Customer Scenarios Authentication 9/21/2018 7:06 PM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. I want only authenticated users to see data
9/21/2018 7:06 PM
I want only authenticated users to see data
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

17. Multi-user authentication
Gateway layer to validate user
HDInsight cluster
Username/password
Azure blob storage
Azure data lake store
unauthenticated user

18. Multi-user authentication in Standard Clusters

19. Multi-user authentication in Premium Clusters
Azure Active Directory
With Domain Services enabled
Sync users
Fetch Kerberos tickets
Azure blob storage
HDInsight cluster
Azure data lake store

20. Demo Multi-user authentication 9/21/2018 7:06 PM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. Customer Scenarios Authorization – Role based access control
9/21/2018 7:06 PM
Customer Scenarios
Authorization – Role based access control
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. 9/21/2018 7:06 PM
I want only privileged users to access sensitive data and perform privileged operations
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

23. Authorization in Ambari portal

24. Authorization using Apache Ranger

25. Authorization using Apache Ranger
Azure Active Directory
With Domain Services enabled
Ranger database
Sync users
Fetch Kerberos tickets
Fetch authorization policies
Username password
HDInsight cluster

26. Auditing using Apache Ranger

27. Authorization using File and Folder level ACLs
Azure Storage
No built-in File & Folder ACLs
Apache Ranger plug-in available
Azure Data Lake Store
Built-in File & Folder ACLs
Seamless integration with built-in support*
* Available only for users that do not have Multi-factor authentication setup.

28. Demo Authorization 9/21/2018 7:06 PM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

29. Customer Scenarios Encryption of data 9/21/2018 7:06 PM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

30. I want my data to be encrypted .. Always (at rest and in-transit)
9/21/2018 7:06 PM
I want my data to be encrypted .. Always (at rest and in-transit)
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

31. Server-Side encryption
Azure Storage
Transparent Server-Side encryption using Microsoft managed keys
Transparent Server-Side encryption using customer keys coming soon
Azure Data Lake Store
Transparent Server-Side encryption using Microsoft managed keys
As well as customer managed keys.

32. In Transit encryption https https (TLS 1.2) Transparent Server Side

33. 9/21/2018 7:06 PM
To summarize …
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

34. Authorization & Auditing
9/21/2018 7:06 PM
Apache Ranger
RBAC for Admin
POSIX ACLs for Data Plane
Server-Side encryption at rest
HTTPS/TLS In-transit
Azure Active Directory
Kerberos authentication
Virtual Networks
Network Security Groups
Service Tunneling
Perimeter
Security
Multi-user
Authentication
Authorization & Auditing
Data Encryption
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

35. 9/21/2018 7:06 PM
“HDInsight as Big Data Platform has enabled our data engineers and scientists to focus on developing data and analytics products rather than managing infrastructure and troubleshooting day-day issues related to very large clusters. The heavy lifting of installing & managing clusters, providing robust security with Apache Ranger, data at rest encryption, monitoring and scaling up/down is taken care by HDInsight.  This platform is used for variety of use cases like real time streaming, machine learning, visualization, ETL. Overall a very positive experience with HDInsight engineering, product and support teams.” -- Navaljit Bhasin, Big Data Engineering Director, Honeywell
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

36. Get started with Azure HDInsight
Azure HDInsight overview
Azure HDInsight documentation
Azure HDInsight Training

37. 9/21/2018 7:06 PM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

38. Externalize Ranger database
9/21/2018 7:06 PM
Externalize Ranger database
"admin-properties": {
"audit_db_name": "[parameters('rangerDbName')]",
"audit_db_user": "[parameters('rangerDbUser')]",
"audit_db_password": "[parameters('rangerDbPassword')]",
"db_name": "[parameters('rangerDbName')]",
"db_user": "[parameters('rangerDbUser')]",
"db_password": "[parameters('rangerDbPassword')]",
"db_host": "[parameters('rangerDbServerName')]",
"db_root_user": "",
"db_root_password": "" },
"ranger-admin-site": {
"ranger.jpa.jdbc.url": "[concat('jdbc:sqlserver://', parameters('rangerDbServerName'), ';databaseName=', parameters('rangerDbName'))]"
"ranger-env": {
"ranger_privelege_user_jdbc_url": "[concat('jdbc:sqlserver://', parameters('rangerDbServerName'), ';databaseName=', parameters('rangerDbName'))]"
"ranger-hive-security" : {
"ranger.plugin.hive.service.name" : "common_repo" }
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

39. Multi-user authentication in Premium Clusters
Azure Active Directory
With ADFS enabled
Sync users
Fetch Kerberos tickets
Azure blob storage
HDInsight cluster
Azure data lake store

40. Multi-user authentication in Premium Clusters
Azure Active Directory
With ADFS enabled
Sync users
Fetch Oauth 2 token
Azure blob storage
HDInsight cluster
Azure data lake store

41. Customer Scenarios Monitoring of clusters 9/21/2018 7:06 PM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

42. I want to ensure my clusters are always healthy
9/21/2018 7:06 PM
I want to ensure my clusters are always healthy
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

43. Default Monitoring We monitor your clusters.
Every 5 minutes, we will ensure these services are running
If monitoring reveals that services are down, automated alert gets raised to our engineers.
99.9% SLA is guaranteed.
Monitoring service
Cluster available
Data nodes running
Certificate valid
Node managers up
Oozie running
Job submission working
MapReduce running
Rstudio running

45. HDInsight Monitoring in Azure Log Analytics