1. Hadoop in the Enterprise
Microsoft Ignite 2016
6/2/2018 9:16 PM
Hadoop in the Enterprise
DA335a
Erika Harris & Aaron Morrison
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. Session Objectives & Takeaways
6/2/2018 9:16 PM
Session Objectives & Takeaways
Capabilities
Why use Hadoop
What business needs will it enable
Security
How do we protect our enterprise data
Integration with Network & Identity
Enterprise
How do we operationalise this
What outcomes have we enabled
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. 6/2/2018 9:16 PM
Capabilities
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4. The 5 Vs of Big Data Volume Variety Velocity Variability Veracity
Microsoft Ignite 2016
6/2/2018 9:16 PM
The 5 Vs of Big Data
Volume
Variety
Velocity
Variability
Veracity
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. New analytic applications from big data
EXAMPLE SOLUTIONS
Marketing
Sales
Service
Finance
Operations
Workforce
Customer Insight
Customer Acquisition
Service Analytics
Financial Analytics
Demand Forecasting
Employee Insight
Customer Feedback Analytics
Cross sell & Upsell
Contact Center Analytics
Financial Forecasting
Inventory Optimization
HR Insight
Churn Analytics
Lead Generation & Opportunity Scoring
Fraud Management
IT Operations Insight
Pay for Performance
Product Innovation
Risk Management
Operational Efficiency
Personalization
Procurement Insight
Product Recommendation
Spend Insight
Marketing Optimization
Supplier Insight

6. Hadoop is a platform with portfolio of projects
Microsoft Ignite 2016
6/2/2018 9:16 PM
Hadoop is a platform with portfolio of projects
Governed by Apache Software Foundation (ASF)
Comprises core services of MapReduce, HDFS, and YARN
In addition to the core, includes functions across:
Data services which allow you to manipulate and move data (Hive, HBase, Pig, Flume, Sqoop)
Operational services which help manage the cluster (Ambari, Falcon, and Oozie)
Governance and integration
Data workflow, lifecycle and governance
Falcon
Atlas
Sqoop
Flume
NFS
WebHDFS
YARN: data operating system
Script
Pig
Search
Solr
SQL
Hive/Tez, HCatalog
Nosql
Hbase
Accumulo
Stream
Storm
Others
Spark, in-memory,
ISV engines 1 N
Batch
Map reduce
Data access
HDFS (Hadoop Distributed File System)
Data management
Authentication
Authorization
Accounting
Data protection
Ranger
Knox
HDFS Encryption
Security
Operations
Provision, manage, and monitor
Ambari
Zookeeper
Cloudbreak
Scheduling
Oozie
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Hadoop in Azure MapR, Cloudera & Hortonworks on IaaS
Microsoft Ignite 2016
6/2/2018 9:16 PM
Hadoop in Azure
MapR, Cloudera & Hortonworks on IaaS
HDInsight PaaS based on Hortonworks Data Platform
Match cluster type to workloads; Hadoop, Storm, Spark, HBASE, Hive Interactive, Kafka
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

8. Demo - HDInsight Erika Harris Microsoft Ignite 2016 6/2/2018 9:16 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. 6/2/2018 9:16 PM
Security
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. Securing enterprise data
6/2/2018 9:16 PM
Securing enterprise data
HDI
Data Access
User Access
Network Access
Service Access
Perimeter Security
Private networks
Role Based Access Control to Portal
Access & Audit
AD Authentication
Access and authorization
Security log management
Data Protection/Integrity
Shared Access Signature (eg source data read only)
Encryption at rest
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11. Roles – Service Administration
6/2/2018 9:16 PM
Roles – Service Administration
Used for separation of duties
Different roles
Owner
Contributor
Reader
Does not apply access control to nodes or data of a HDI instance
Report on access changes through Powershell:
\> Get-AzureRMAuthorizationChangeLog
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. 6/2/2018 9:16 PM
Network Access
HDI deployed in its own vNet using private IP range which can be an extension of on-prem
NSGs provide Layer 3 network filtering
Network traffic between Enterprise and Azure within ExpressRoute path
Load Balancer and Nodes managed as part of PaaS
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. Data Storage Access Use multiple storage accounts to segregate data
6/2/2018 9:16 PM
Data Storage Access
Use multiple storage accounts to segregate data
Shared Access Signature tokens to restrict access
RW vs RO
IP Filtering
Expiry time for temporary access
Storage encryption at rest
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

14. Microsoft Ignite 2016
6/2/2018 9:16 PM
HDI User Access
Domain joined HDI clusters enable user auth via Azure AD
Only specified users or groups can log into HDI
User Ranger to define user access policies
Audit user access and policy modification
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Demo - Ranger Erika Harris Microsoft Ignite 2016 6/2/2018 9:16 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. 6/2/2018 9:16 PM
Enterprise
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

17. Integrate with Enterprise process & operations
6/2/2018 9:16 PM
Integrate with Enterprise process & operations
Automated deployment
Configuration Management
Integration with test/dev lifecycle and process
Lifecycle maintenance
Product updates
Data backup and retention
Business as usual
Job submission
Cluster scaling
Monitoring
User experience (Developer, Data Scientist, BI)
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. Visual Studio Team Services: DevOps
Azure Tools for Visual Studio
Integrated DevOps—Team Build
Use Visual Studio Team Services (VSTS) for built in delivery of App Services and Cloud Services
Trigger builds from check-ins or schedule
Run custom script (e.g., PowerShell, Chef) tasks
Visual Studio
Eclipse/XCode
Code Repository
Dev and test in the cloud
Use and provision VMs to increase productivity and reduce infrastructure costs and wait time
Release Manager
Orchestrate Build, Test, and Deploy to multiple environments (Dev, QA, Staging, Prod.)
Move and promote complete application (data, config, ...)
Promote builds based on custom criteria
Works with VSTS, GitHub, Jenkins, and more
Application
Insights and
Hockey App
Load Testing
Azure
Release
Management

19. Maintenance Consider HDI as a disposable cluster
Externalise data and meta store from the cluster
Re-deploy a cluster to enable updates
Externalise management through scripting and automation
Backup
External meta stores are backed up through the Azure SQL database backup
Azure automation runbook to backup blob storage

20. Maintenance – Redeployment
Product updates:
Configure HDI with external meta store and storage account
Bootstrap new HDI cluster with scripted changes
Test the new cluster
Destroy the old cluster

21. Where to next? HDI On Demand – Data Factory
Orchestrating Big Data Pipelines with Azure Data Factory [DA332]
Monitoring – OMS (Log Analytics)
Power Up your Monitoring with OMS Insights and Analytics [INF223]
R Server Edge Nodes
Analytics Workflows with Microsoft R Server [DA334]
PowerBI
Power BI for the Enterprise, safely sharing your data [DA313]
Polybase
PolyBase: T-SQL Reaching Beyond the Database [DA336a]

22. Demo – Config Bootstrap
Microsoft Ignite 2016
6/2/2018 9:16 PM
Demo – Config Bootstrap
Aaron Morrison
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

23. Session Objectives & Takeaways
6/2/2018 9:16 PM
Session Objectives & Takeaways
Capabilities
Why use Hadoop
What business needs will it enable
Security
How do we protect our enterprise data
Integration with Network & Identity
Enterprise
How do we operationalise this
What outcomes have we enabled
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

24. Continue your Ignite learning path
6/2/2018 9:16 PM
Continue your Ignite learning path
HDInsight Documentation us/documentation/services/hdinsight/
Learning Map us/documentation/articles/hdinsight-learn-map/
Visit Microsoft Virtual Academy for free online training visit
Visit Channel 9 to access a wide range of Microsoft training and event recordings
Visit Channel 9 to access a wide range of Microsoft training and event recordings
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

25. 6/2/2018 9:16 PM
Thank you Chat with me in the Speaker Lounge Find me
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.