1. 4/11/2018 4:51 AM
BRK3118
Keeping your on-premises data up to date with the On-premises data gateway
Robert Bruckner
Principal Architect
Gateways & Connectors Team, Microsoft
Miguel Llopis
Senior Program Manager
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. Session Agenda Learn what is the On-premises Data Gateway
4/11/2018 4:51 AM
Session Agenda
Learn what is the On-premises Data Gateway
Use the gateway in Power BI
Best practices using the gateway
Gateway roadmap and “Coming Soon” demos
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. Azure Analysis Services
4/11/2018 4:51 AM
On-premises data gateway One gateway for multiple cloud services and experiences
Cloud services
Power BI
Azure Analysis Services
PowerApps
Microsoft Flow
Azure Logic Apps
Power Query Online
Gateway Management Cloud Service Data source connection credentials are encrypted
Azure Service Bus
On-premises Data Gateway
Data source connection credentials can only be decrypted by the gateway
On-premises data sources
Files, SharePoint
SQL Server
SQL Server Analysis Services
Other data sources
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4. What is an On-premises Data Gateway?
Secured, and optimized communication channel to use with your cloud services (Power BI, Azure Analysis Services, PowerApps, Microsoft Flow, Azure Logic Apps, Power Query Online/CDS)
The gateway is not a general purpose/bi-directional VPN
The gateway installs on any domain-joined machine
Outbound traffic only
You don’t have to install it on a DMZ, or open your firewall for inbound traffic
The gateway enables data/API connectivity
It’s orthogonal to how you consume Dashboards, Reports, Flows or Apps on top of these connections

5. What is an On-premises Data Gateway?
The Gateway uses an Azure Service Bus Relay to communicate between on-premises data sources and cloud services
Azure Service Bus infrastructure is provisioned and owned by the Gateway Cloud Service, it is not an additional offering customers need to buy/manage
This enables a self-service, low-friction installation, configuration, and runtime of gateways
Unless the gateway is up and running, connections & requests won’t pass through the gateway
No caching of requests or data on the on-premises gateway or the communication pipeline to the cloud

6. On-premises Data Gateway management
Direct query
Scheduled refresh
Cloud
On-premises
SQL
SSAS
SharePoint
On-premises data gateway
Manage & Monitor
Centralized way to refresh on- premises content
Access control to data sources
Users and admin management and controls

7. On-premises data gateway (personal mode) v/s On-premises data gateway
4/11/2018 4:51 AM
On-premises data gateway (personal mode) v/s On-premises data gateway
On-premises Data Gateway (Personal Mode)
On-premises Data Gateway
(aka Enterprise Gateway)
Target Persona
Business analyst sets up and uses the gateway for her data sources only
BI Admins set up the gateway for their department/ company
Multiple users use the gateway setup by the admins
Features
Import with scheduled refresh
Direct query and scheduled refresh
No additional data source configuration to set up a refresh
Gateway admin needs to configure each data source explicitly
Data source connections managed per user
Central data source management and access control
Services supported
Power BI
Power BI, PowerApps, Microsoft Flow, Azure Logic Apps, Azure Analysis Services, Power Query Online / CDS
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

8. Demo Gateway Setup & Configuration 4/11/2018 4:51 AM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. Architecture: Power BI Data Refresh on-premises
4/11/2018 4:51 AM
Architecture: Power BI Data Refresh on-premises
Gateway is installed & configured. During configuration, a corresponding service bus instance is also configured.
Credentials entered for the data source in the cloud services are encrypted then stored in the cloud. Only the gateway can decrypt the credentials. Personal Gateway windows credentials is stored in the Gateway only.
One of the supported cloud services kicks off a refresh or a live query
Data Movement Service analyzes the query and pushes to appropriate Service Bus communication channel
Gateway polls its Service Bus channel for pending requests. It takes the pending request
Gateway gets the query, decrypts the credentials, sends query to the data source for execution
After execution, gateway securely pushes the data to the cloud service
Gateway
Cloud Service DB
Data Movement Service
cred 2
Scheduler service 3 4
Service bus
Cloud
On-premises 7 5 1
cred 2 6
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. 4/11/2018 4:51 AM
Security in Gateway
Encryption keys based on recovery key never leave on-premises gateway machine.
Generates strong RSA and AES keys. Symmetric and private key never leave gateway machine, and are encrypted locally with Windows OS machine key.
The Gateway cloud service never knows the on-premises credential values encrypted / cannot intercept credentials
Web client encrypts the credential with a public key associated with the specific gateway it communicates with
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11. Connectivity in Power BI
4/11/2018 4:51 AM
Connectivity in Power BI
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. Data refresh in Power BI
TechReady 23
4/11/2018 4:51 AM
Data refresh in Power BI
Where is your data?
Cloud
On-premises
How do you connect?
Import data
Direct Query
How do you refresh?
Personal mode
Enterprise mode
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. Supported on-premises data sources
SQL Server, Teradata, Oracle, DB2, MySQL, Web, OData, PostgreSQL, SAP HANA, SAP BW, Access, Sybase, Informix, PostgreSQL, Impala, Snowflake, Custom ODBC Drivers
SQL Server Analysis Services (tabular and multi-dimensional)
Files/folder, SharePoint on-premises
Active Directory New, SAP BusinessWarehouse DirectQuery New , OLEDB New , IBM Netezza New
Coming soon: Spark, Google BigQuery, etc.
Roadmap: Custom Connectors Support in the Gateway

14. How do you want to refresh data?
4/11/2018 4:51 AM
How do you want to refresh data?
Import (cached mode)
Direct query or Live connection
Refresh frequency
Scheduled - hourly or daily
In Power BI Premium can be done every 30 minutes.
Real-time
Performance
No noticeable delay since data is already cached
Depends on how fast the data source is, as queries are executed in real-time
Data storage in Power BI
Since it is cached mode, data is stored in the cloud
No data is stored in Power BI. Data is always on-premises*
Data size
Current limit of 1 GB (compressed) per model; unlimited in Premium
The on-premises database is the limit; no Power BI limitation
Security
Can create row-level security on the Power BI dataset (import only)
Re-use on-prem row level security (for Analysis Services, additional SSO based on Kerberos)
Data sources
All on-prem data sources supported can be used in import mode
SQL, SSAS, Oracle, Teradata, SAP HANA, Redshift, SAP BW, Impala, etc. (more coming soon: Spark, etc.)
*Some data for visuals is cached for optimizing first-time load performance (refresh interval can be configured)
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Single Sign-on & Row Level Security
4/11/2018 4:51 AM
Single Sign-on & Row Level Security
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. Data Row Level Security (RLS) in Power BI
Microsoft Data Insights Summit
4/11/2018 4:51 AM
Data Row Level Security (RLS) in Power BI
Define RLS at a dataset level in Power BI
Leverage RLS defined in the underlying data source
Live Connect with SSAS on-premises, and Azure AS
DirectQuery with Single Sign-on
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

17. Single Sign-On / RLS from Power BI via Gateway
4/11/2018 4:51 AM
Single Sign-On / RLS from Power BI via Gateway
Power BI Service:
For each query by a Power BI AAD user to on-premises SSAS servers, or supported SSO DirectQuery sources, it passes along UPN with query: e.g.
Power BI
Cloud
On-premises
On-premises Data Gateway:
Map UPN to a local user (if needed)
Execute query on behalf of the original user.
SQL Server
SQL Server
Tera-data*
Oracle*
SAP HANA**
SAP BW**
SAP BW**
Impala **
Impala **
On-premises data sources
SQL Server Analysis Services
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. Demo / Demo Recap Three separate Power BI AAD accounts:
4/11/2018 4:51 AM
Demo / Demo Recap
Three separate Power BI AAD accounts:
On-prem test domain is an isolated domain without AAD DirSync configured; remapping UPNs is performed by the gateway via local Active Directory lookups.
AAD user
Role
Local AD domain
SQL DB Data permissions
GW Admin Dashboard owner
PBIEgwTest.local
RLS: all data
Dashboard consumer
PBIEgwTest.local
RLS: “Computer” category only
Dashboard was re-shared, but user should not see visuals -
Connection is denied at DB level
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. SSO User Mapping Options
4/11/2018 4:51 AM
SSO User Mapping Options
Your local Active Directory accounts match/are synchronized to Azure Active Directory.
Manual User Re-Mapping (UPN Mapping, SSAS Only)
Although manual mapping of each user might be possible using manual mapping, it can become too time-consuming and hard to maintain if there are too many rules.
Especially, when pattern matching does not suffice:
User Account names are different between AAD and on-prem AD
Example: 
On-prem Active Directory Property Lookup to remap UPNs to AD users
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

20. SSO User Mapping Options
4/11/2018 4:51 AM
SSO User Mapping Options
On-prem Active Directory Property Lookup to remap UPNs to AD users Gateway configuration for: AD User Lookup and AD Username Replacement Property
Lookup e.g. ‘ ’ attribute of AD Person based on incoming UPN string from Power BI Service.
If AD Lookup succeeds, retrieve ‘UserPrincipalName’ of that AD Person.
Replace the original Power BI service AAD account with ‘UserPrincipalName’ as EffectiveUserName to SSAS: e.g.
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. AD User Mapping with Custom Property Lookup
4/11/2018 4:51 AM
AD User Mapping with Custom Property Lookup
Power BI
Power BI Service:
For each query by a Power BI AAD user to on-premises SSAS servers, passes along UPN string:
Note: any manual UPN user mappings defined in the Power BI data source configuration are still applied before sending the user name string to the on-premises data gateway.
Cloud
On-premises
On-premises Data Gateway with configurable Custom User Mapping:
Find Active Directory to search (automatic, or configurable)
Lookup e.g. ‘ ’ attribute of AD Person based on incoming UPN string from Power BI Service.
If the AD Lookup fails, attempts to use the passed-along UPN
If AD Lookup succeeds, retrieve ‘UserPrincipalName’ of that AD Person.
Pass ‘UserPrincipalName’ as EffectiveUserName to SSAS: e.g.
SQL Server
SQL Server
Tera-data*
Oracle*
SAP HANA**
SAP BW**
SAP BW**
Impala **
Impala **
On-premises data sources
SQL Server Analysis Services
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. High Availability & Load-Balancing
4/11/2018 4:51 AM
High Availability & Load-Balancing
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

23. High Availability & Load Balancing
4/11/2018 4:51 AM
High Availability & Load Balancing
Today:
Manual takeover of gateway (switches all traffic as soon as takeover completed)
Manual workload partitioning E.g. Refresh of imported data vs. DirectQuery
Automatic High Availability, Load Balancing coming:
A gateway can join a cluster of gateways: need to be an admin on original gateway, and know recovery key.
A group of gateways will act effectively as a single unit (e.g. data source definitions)
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

24. Testing Load Balancing and Failover
4/11/2018 4:51 AM
Testing Load Balancing and Failover
GogoAir
Power BI (West Central US)
Gateway @ ft
Cloud
Gateways (Seattle, Chicago)
SQL DW
On-premises
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

25. Gateways (Seattle, Chicago)
4/11/2018 4:51 AM
Testing Load Balancing and Failover on an Airplane …
GogoAir
Power BI (West Central US)
Gateway @ ft
Cloud
Gateway Clustering
Gateways (Seattle, Chicago)
Data Access via VPN to data source VNet
SQL DW
On-premises
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

26. Demo Configuring Gateway Clusters: High Availability & Load-Balancing
4/11/2018 4:51 AM
Demo
Configuring Gateway Clusters: High Availability & Load-Balancing
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

27. Gateway Settings and Troubleshooting
4/11/2018 4:51 AM
Gateway Settings and Troubleshooting
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

28. Where to install the gateway
4/11/2018 4:51 AM
Where to install the gateway
Recommendation:
Start with an 8 core machine
Keep an eye on your performance counters
Depending on your usage, you can decide to scale up or down
Gateway doesn’t have to run on the same machine as the data source
Hardware needs depend on the usage scenarios for the gateway:
Frequency and data volume of data model refreshes in Power BI
DirectQuery: number of concurrent users, types of visualizations for interactive querying, whether RLS is used, number of pinned dashboard tiles over DirectQuery/LiveConnect keep updating in the background.
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

29. Gateway Network Communication
4/11/2018 4:51 AM
Gateway Network Communication
URI
Outbound Ports
Gateway Installation
Gateway Runtime
Purpose
*.download.microsoft.com 80 X
Download of Personal/Data Gateway installer package
*.msftncsi.com
443
Internet/Windows connectivity diagnostics check
*.login.windows.net
login.microsoftonline.com
*.microsoftonline-p.com
Azure AD login for PowerBI.com
*.powerbi.com
Configuration and registration of Gateway with Power BI & Gateway Management service
*.servicebus.windows.net
Tcp-Mode of the gateway for runtime communication for queries & results (encrypted, compressed).
Https-Mode of the gateway for runtime communication for queries & results (encrypted, compressed).
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

30. Proxy/Network settings: FQDN support
4/11/2018 4:51 AM
Proxy/Network settings: FQDN support
Forcing a gateway to communicate with Azure Service Bus using HTTPS-only instead of AutoDetect/TCP ports.
With locked-down network proxies present, this still required to whitelist Azure Data Center IP address ranges for the HTTPS traffic:
More info on proxies:
Since February 2017 Release:
When switching to HTTPS-mode, no network communication based on direct IP addresses anymore.
Instead all data traffic goes to FQDNs: *.servicebus.windows.net
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

31. Gateway logs Logs, logs, logs!
4/11/2018 4:51 AM
Gateway logs
Logs, logs, logs!
We’ve made it easier to retrieve all the logs.
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

32. Disaster recovery and gateway restore
4/11/2018 4:51 AM
Disaster recovery and gateway restore
When setting up a gateway, you are asked for a “Recovery Key”
Save it somewhere safe!
Gateway admin can use the recovery key to restore a gateway to a different machine.
Once a gateway is restored, all data sources and (encrypted) credentials will be restored automatically to the new gateway.
Restored gateway has the same name, and takes place of the original gateway across cloud services. No need for mapping!
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

33. Tips and Best practices
4/11/2018 4:51 AM
Tips and Best practices
Monthly updates: Update notification in Gateway Configurator
Keep it always on and credentials up-to-date
Can be installed on Windows Server or Client OS
Cannot be installed on the same machine with a local domain controller
Installing multiple gateways on the same computer
Can install Personal Mode and Enterprise Mode in a single machine
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

34. 4/11/2018 4:51 AM
More tips and guidance
Visit the FAQ and troubleshooting section in our documentation, leave comments/questions:
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

35. On-premises Data Gateway Roadmap
4/11/2018 4:51 AM
On-premises Data Gateway Roadmap
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

36. On-premises Data Gateway Roadmap (Power BI)
4/11/2018 4:51 AM
On-premises Data Gateway Roadmap (Power BI)
SSO via Kerberos for DirectQuery against SQL Server Database, Oracle Database, Teradata, Impala, SAP BW & SAP HANA – SQL Server available in October, others coming before EOY.
OAuth support for data mashup scenarios through Gateway:
Phase 1: Multiple sources in a single dataset, without mashup (September)
Phase 2: Multiple sources in a single dataset, including mashup (future)
High Availability & Basic Load Balancing (Public Preview in October)
General third-party extensibility in gateway for data connectors (before end of CY17)
Data Connector SDK:
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

37. Please evaluate this session
Tech Ready 15
4/11/2018
Please evaluate this session
From your
Please expand notes window at bottom of slide and read. Then Delete this text box.
PC or tablet: visit MyIgnite
Phone: download and use the Microsoft Ignite mobile app
Your input is important!
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

38. 4/11/2018 4:51 AM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.