Data Platform and Analytics Foundational Training Microsoft C+E Technology Training Data Platform and Analytics Foundational Training Solution Area Data Analytics Solution Business Intelligence Technology SQL Server EIM (SSIS, MDS, DQS) [Speaker Name]

Enterprise Information Management “Enterprise Information Management (EIM) is a best practice for creating, managing, sharing, and leveraging information in an enterprise, holistic manner that’s aligned with strategic, data-driven business objectives” TDWI – Philip Russom

Enterprise Information Management Through team collaboration and tool interoperability, EIM seeks to improve the “four C’s” of structured and unstructured data: Completeness Cleanliness Consistency, and Currency

SQL Server EIM SQL Server has evolved to deliver EIM with a rich set of services designed to improve the overall relevance and value of organizational information

SQL Server EIM Empowers people to: Discover and resolve problems with data Streamline existing data management processes using familiar tools like Excel

SQL Server EIM Provisions and manages enterprise information: Manage more effective data integration and management across the organization Deliver trusted information for Business Intelligence, data warehouse, and transaction processing workloads

SQL Server EIM SQL Server 2016 delivers EIM with three services: Integration Services Master Data Services Data Quality Services These services can be installed on-premises, or hosted inside a virtual machine (IaaS)

Complete, Clean, Consistent and Current Data SQL Server EIM Integration Services Master Data Services Data Quality Services Primarily designed to implement ETL processes Provides a robust, flexible, fast, scalable and extensible architecture Master Data Management Manages reliable, centralized data Broadens its reach with an Excel Add-in that can also use Data Quality Services for data matching Data Quality Management Knowledge-driven data cleansing Corrects and de-duplicates data Can correct data errors in the Integration Services data flow Complete, Clean, Consistent and Current Data

Integration Services

Integration Services SQL Server Integration Services (SSIS) is a service for building enterprise-level data integration and data transformation solutions It includes: A rich set of built-in tasks and transformations Tools for constructing packages, and The SSIS service for running and managing packages

Integration Services SSIS was primarily designed to implement Extract, Transform and Load (ETL) processes Provides a robust, flexible, fast, scalable and extensible architecture Its capabilities are useful in many other scenarios: Assessing data quality Cleansing and standardizing data Merging data from heterogeneous data stores Implementing repetitive and ad hoc data transfers Automating administrative tasks

Integration Services Architecture SQL Server Integration Services (SSIS) is a service The services has two different runtime engines: Control flow Data flow

Integration Services Architecture: Control Flow Control Flow is the process-oriented workflow engine A package consists of a single control flow Control flow elements: Variables Containers Tasks Precedence constraints, and Event handlers

Integration Services Architecture: Data Flow The Data Flow task encapsulates the Data Flow engine It is designed by connecting: Sources Destinations, and Transformations

Integration Services Architecture: Data Flow Extract Transform Load

Integration Services SQL Server 2016 New Features SQL Server 2016 SSIS includes many new features: Improved deployment Improved debugging Improved package management Control Flow package parts Expanded on-premises connectivity Expanded cloud connectivity And many more…

Integration Services SQL Server 2016 New Features: Azure Feature Pack Download and install the Azure Feature Pack New features include: Connection managers Enumerator Tasks, and Data Flow components

Integration Services SQL Server 2016 New Features: Azure Feature Pack

Integration Services SQL Server 2016 Support by Edition SSIS 2016 is supported by Enterprise, Standard and Developer editions Import/export Wizard, Built-in and Azure data source connectors and tasks are also supported by Web and Express editions Hadoop/HDFS connectors and tasks are also supported by Web edition Supported only by Enterprise edition: CDC Service and Designer for Oracle by Attunity Advanced adapters Advanced transforms

Integration Services Resources MSDN: SQL Server Integration Services https://msdn.microsoft.com/en-us/library/ms141026 MSDN: What’s New in Integration Services https://msdn.microsoft.com/en-us/library/bb522534.aspx MSDN: Features Supported by the Editions of SQL Server 2016 https://msdn.microsoft.com/en-us/library/cc645993.aspx?#IS SQL Server Integration Services (SSIS) Official Blog https://blogs.msdn.microsoft.com/ssis/

Integration Services Resources TechNet Virtual Lab: Exploring What's New in SQL Server 2016 Integration Services https://technet.microsoft.com/en-us/virtuallabs Use the search pane to locate the lab

Master Data Services

Master Data Services SQL Server Master Data Services (MDS) is a service designed to implement and manage Master Data Management (MDM) solutions v1: First released with SQL Server 2008 R2 v2: The SQL Server 2012 release included many new features and enhancements v3: The SQL Server 2014 release included no new capabilities or features v4: The SQL Server 2016 release included many new capabilities and features Supported by Enterprise edition only For non-production use, it is also supported by Developer and Evaluation editions

Master Data Services MDM MDM is about taking control of data, and making data work to the organization’s advantage It enables an organization to discover and define non-transactional lists of data, and to compile maintainable, reliable master lists A successful MDM solution delivers a master data hub that provides access to authoritative, standardized, and validated versions of data

Master Data Services MDM MDM can address several problems: No authoritative source of data ► Reports may not be trusted ► Poor decisions Systems with different versions of the same data Data spread across multiple systems can lack integrity and consistency Mergers and acquisitions that involve integrating data contribute to different versions No formal dimension maintenance capability Multiple stakeholders for the same record Inability to enable, monitor and enforce data stewardship processes

Master Data Services MDM MDM can be implemented as: Operational MDM Analytical MDM

Master Data Services Operational MDM Operational MDM delivers a Master Data hub as the System of Record (Master Data is created), and with a focus on integration achieved with centralized governance

Master Data Services Analytical MDM Analytical MDM delivers a single and consistent view of Master Data in the downstream data warehouse Having implemented an Analytical MDM solution, a master list of records can then become the dimension source

Master Data Services Architecture MDS is implemented as a WCF Data Service Internet Information Services (IIS) is required for the Master Data Services web application and web service MDS can be programmatically manipulated The repository for metadata and data is a SQL Server database The MDS database can be created on either the local or a remote computer Microsoft provides two user interfaces: Master Data Manager web application Excel add-in

Master Data Services Sample Implementation Source System(s) MDS Administration Data Steward via Master Data Manager or Excel WCF Data Service Data Changes SSIS SSIS Notification Outlook Data Warehouse SSIS

Master Data Services EIM Integration MDS can integrate with SSIS Importing master data is achieved by loading data into SQL Server tables Exporting master data is achieved by retrieving data from SQL Server views MDS can also integrate with DQS

Master Data Services SQL Server 2016 New Features SQL Server 2016 MDS includes many new features: Improved user interfaces Improved permission management Improved performance and scale Improved business rules, including custom T-SQL logic Merge conflict detection and resolution Changesets to enable approval workflows Improved hierarchies Entity Sync to share common entities between models And many more…

Master Data Services Resources MSDN: SQL Server Master Data Services https://msdn.microsoft.com/en-us/library/ee633763.aspx MSDN: What’s New in Master Data Services https://msdn.microsoft.com/en-us/library/ff929136.aspx MSDN: Features Supported by the Editions of SQL Server 2016 https://msdn.microsoft.com/en-us/library/cc645993.aspx?#MDS Master Data Services Blog https://blogs.msdn.microsoft.com/mds/

Master Data Services Resources TechNet Virtual Labs: 1: Developing a SQL Server 2016 Master Data Services Solution 2: Exploring What's New in SQL Server 2016 Master Data Services https://technet.microsoft.com/en-us/virtuallabs Use the search pane to locate the lab

Data Quality Services

Data Quality Services SQL Server Data Quality Services (DQS) is a knowledge-driven data cleansing product It is a product dedicated to data correction, standardization, enrichment and matching v1: First released with SQL Server 2012 v2: The SQL Server 2014 release included no new capabilities or features v3: The SQL Server 2016 release included no new capabilities or features Supported by Enterprise edition only For non-production use, it is also supported by Developer and Evaluation editions

Data Quality Services Architecture Source: MSDN https://msdn.microsoft.com/en-us/library/ff877917.aspx

Data Quality Services Benefits The benefits of using DQS include: Accumulating cleansing and matching knowledge in a knowledge base Updating cleansing and matching knowledge, as required Leveraging built-in functionality and algorithms to cleanse and match data Applying consistent cleansing logic Automating cleansing activities—reducing time, cost and the potential for “human” error

Data Quality Services EIM Integration DQS can integrate with SSIS The DQS Cleansing data flow transform is available with SSIS 2012, and later

Data Quality Services EIM Integration

Data Quality Services EIM Integration DQS can integrate with MDS MDS Add-in for Excel can be used to help identify duplicate members before they are published to an entity

Data Quality Services SQL Server 2016 New Features SQL Server 2016 DQS includes no new features

Data Quality Services Resources MSDN: Introduction to Data Quality Services https://msdn.microsoft.com/en-us/library/ff877917.aspx Data Quality Services Blog https://blogs.msdn.microsoft.com/dqs/ eBook: Data Quality Services Do a web search on “Data Quality Services.pdf”

Summary

Summary SQL Server 2016 continues to deliver a comprehensive, EIM platform with three services: Integration Services Master Data Services Data Quality Services They function independently, and also together in integrated designs

Complete, Clean, Consistent and Current Data SQL Server 2016 EIM Integration Services Master Data Services Data Quality Services Primarily designed to implement ETL processes Provides a robust, flexible, fast, scalable and extensible architecture Master Data Management Manages reliable, centralized data Broadens its reach with an Excel Add-in that can also use Data Quality Services for data matching Data Quality Management Knowledge-driven data cleansing Corrects and de-duplicates data Can correct data errors in the Integration Services data flow Complete, Clean, Consistent and Current Data

© 2016 Microsoft Corporation. All rights reserved © 2016 Microsoft Corporation. All rights reserved. Microsoft, Windows, Microsoft Azure, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION