SQL Server 2005 Integration Services Dave Glover Microsoft Australia

Agenda Cleanse Data Input Split an Output Channel Script to Branch Control Lookup Fuzzy Values Loop through Folder Configure and Deploy

Overview Part of SQL 2005 Std and Ent Editions: Integration Services with Basic Transforms. Provides graphical extract, transform, and load (ETL) capabilities Ent only: Integration Services Advanced Transforms Includes data mining, text mining, and data cleansing Collaborative Development Separate Management Tools Data Quality and profiling Scalable

SSIS Runtime WMI Sources Flat File Dests OLEDB Connection FTP HTTP FILE ODBC Managers CUSTOM FLATFILE Custom XML SQLServer Flat File FTP File OLEDB Connection FTP HTTP FILE ODBC Managers CUSTOM FLATFILE WEBFTP DB XML Task FILE FTP Task FTP Web Services Task HTTP Data Flow Task ODBC … FILE …Custom Containers WMI Event Task WMI Event Handlers Send Mail Task Send Mail Task Variables Log Providers Task Host Custom Task Variables Log Providers XML

Cleansing Data Input Connection-Related Objects Connection Manager Flow Types Data Adapter Metadata Validation Data Flow Components

Flow Types Two different types of flow Control Flow = Runtime = Tasks Data Flow = Pipeline = Transforms Managed in Designer Used to be single view in SQL 2000 Separate views in SQL 2005 Control Flow handles tasks and precedence Data Flow handles transformations - zoomed contents of Data Flow task

Data Flow Components Components Paths Data route between one component and the next Includes metadata about columns moving around Lineage Identifier tracks item transformations Pipeline Components connected by a path SourceTransformDestination

Demo Cleansing Data

Splitting an Output Channel Distributors and Collectors Precedence Constraints

Distributors and Collectors Distributor transformations Multicast Conditional Split Collector transformations Union All Merge Merge Join Src Tfm Dst DC Tfm Tfm

Precedence Constraints Connect one task to another Give sequential relationship to tasks Success/Failure/Completion workflow Establish concurrency Connected sequence of tasks is a task list Independent Task Lists execute concurrently Tasks within a list execute sequentially

Demo Cleansing Data & Splitting an Output Channel

Script and Branch Control Package Variables Script Task Complex Precedence Containers

Package Variables Scope Each container can have variables Define namespace for user variables Containers can access variables from higher levels Accessible from Expressions (such as loops and constraints) – Parameters in Execute SQL task Parent Package (as part of configuration) Script

Script Task Currently VB.Net Can read or modify properties throughout the package Can’t access inner workings of tasks or transforms Can’t modify pipeline metadata (e.g., number of columns piped)

Containers Container provides Grouping of task lists (list of one is allowed) Transaction scope Variable scope A package is a container Add your own tasks Insert your own containers Loops are containers, too

Fuzzy Lookup Transformation Proximity algorithm to find matches Builds index – Index can persist Creates metrics Similarity Confidence Uses a separate connection for reference table

Loops FOR LOOP Loops while expression is TRUE Manually add loop counter = 1 <= Execute tasks in container on each iteration More control—and more complex than For Each FOR EACH LOOP Loops over set of objects Files XML nodes Database objects Set variable (e.g. file name) for each iteration Execute tasks in container on each iteration

Demo Script and Branch Control

Configure and Deploy Configurations XML Customizability Deployment Execution

Package Lifecycle Support Data Visualizers Debugging Stop execution during package Stop control flow before and after Stop transformation during (visualizer) Logging Event Handlers Log Providers

Configurations Take something from the system Environment variables, registry, XML option file Apply it to some part of your package Run the package with the new setting Useful for multiple “similar” jobs Similar to Dynamic Properties from SQL 2000 Facilitate reusability, different environments Configurable at runtime or during execution

Deployment Same server deployment Deploy from BI Development Studio Multiple server deployment Deployment Utility Package configuration files with package Deploy to SQL Server (msdb) or file (dtsx)

Execution Command-line execution DTEXEC User Interface execution DTEXECUI Can generate command line for DTEXEC Scheduling SQL Server Agent

© 2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Report Builder Overview

Report Builder Architecture SQL Server Catalog Web Service / URL interface Report Server Delivery Rendering Query Security Report Builder Client ClientModelDesignerModelDesigner Data Sources (SQL Server, Analysis Services) Drill through report generation Query generation Model Security Report Processing ReportDesignerReportDesigner SQL Mgmt Studio ReportManagerReportManager

Why Report Builder? Report Consumers Business Users Report Viewer Report Builder Power Users Developers Report Designer

What is Report Builder? A new ad-hoc report design tool for SQL Server Reporting Services Targeted at business users who want to find and share answers to interesting questions Driven from a business model of the data so users do not need to understand the underlying data structures Not a full analytical client or replacement for Pivot Tables Fully integrated with Reporting Services and delivered in SQL Server 2005

Report Builder

Report Builder vs. Report Designer Report BuilderReport Designer Targeted at business usersTargeted at IT pros and developers Ad hoc reportsManaged reports Auto-generates queries using semantic layer on top of the source Native queries (SQL, OLE DB, XML/A, ODBC, Oracle) Reports built on templatesFree-form (nested, banded) reports Click-once application, easy to deploy and manage Integrated into Visual Studio Cannot import Report Designer reports Can work with reports built in Report Builder

Wrap-up Report Builder is… a new ad hoc report design tool for SQL Server Reporting Services targeted at business users who want to find and share answers to interesting questions driven from a business model of the data so users do not need to understand the underlying data structures is fully integrated with SQL Server Reporting Services and delivered in SQL Server 2005

© Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

SQL Server Integration Services SQL Server Analysis Services SQL Server Reporting Services Data acquisition from source systems and integration Data acquisition from source systems and integration Data transformation and synthesis Data transformation and synthesis Data enrichment, with business logic, hierarchical views Data enrichment, with business logic, hierarchical views Data discovery via data mining Data discovery via data mining Data presentation and distribution Data presentation and distribution Data access for the masses Data access for the masses Integrate Analyze Report SQL Server 2005 Business Intelligence

Report Builder Client Report Builder is a “Click Once” Windows application managed on the Report Server Users download and launch the client from the Report Manager or 3rd party applications Built on top of familiar Microsoft Office paradigms: Excel, PowerPoint Reports built with report templates (textboxes, tables, matrices, charts) Provides “infinite drill” for exploring data Finished reports can be saved on the server

Report Builder Client

Report Builder Reports Report Builder reports are just like any other Reporting Services report… Report Builder reports are defined in RDL Managed by the same APIs Secured by the same APIs Except: Report Builder reports contain semantic model metadata When an Report Builder report is requested by report processing, the metadata is used to generate a query in the source language A new component on the Report Server handles the query generation

Infinite Drill Automatic generation of new drillthrough reports allows users to explore data along model navigation paths Queries for navigation reports are generated on the fly, passing the context of the user’s current “location” Users can keep drilling as long as there is a path to follow from the current item Custom reports can be specified for model items Generated navigation reports are not saved unless the user explicitly saves them

Report Models Models define entities, fields and roles available for Report Builder Models reference a data source (either SQL Server or Analysis Services) Models are stored in the Report Server as a new item type Includes new management APIs Published models can be secured Items within models (entities, fields, roles) can also be secured Users only see items they have permissions to see Model security and management through SQL Server Management Studio

Model Design Model Designer is interface for designing/editing a model New project type in BI Development Studio Define fields and roles Define calculated fields Organize model Models can be auto-generated in Model Designer or on the Report Server Generates entities, fields and roles based on source metadata Auto-generation against SQL Server and Analysis Services databases

Model Designer

SQL Server 2005 The comprehensive, integrated data platform