DAT 360: DTS in SQL Server 2000 Best Practices Euan Garden Group Manager, SQL Server Microsoft Corporation

Technologies Covered SQL Server 2000 DTS SQL Server 2000 DTS SQL Server 2000 SQL Server 2000 SQL Server 2000 Analysis Services SQL Server 2000 Analysis Services Transact SQL Transact SQL ActiveX Scripting ActiveX Scripting SQL Server Accelerator for Business Intelligence (SSABI) SQL Server Accelerator for Business Intelligence (SSABI)

“Best Practices” Agenda Package Design Package Design Data Loading Data Loading Analysis Services Management Analysis Services Management Auditing and Error Handling Auditing and Error Handling Putting it all together - SSABI Putting it all together - SSABI

Package Design Practices Drive Packages through Metadata – Dynamic Properties Task Access to Property of any object Access to Property of any object Can get property values from.INI, SQL, etc. Can get property values from.INI, SQL, etc. – Global Variables Building blocks for communicating metadata Building blocks for communicating metadata Populated by the Dynamic Properties Task Populated by the Dynamic Properties Task – UDL Files Configurable File for storing Connection Information Configurable File for storing Connection Information

Package Design Practices Designing Modular Parent/Child Packages – Execute Package Task Allows joining of Transaction Allows joining of Transaction Supports passing of Globals Supports passing of Globals – Execute through COM Automation LoadfromSQLServer, LoadfromRepository, LoadfromStorageFile methods LoadfromSQLServer, LoadfromRepository, LoadfromStorageFile methods Eliminates Version Affinity Eliminates Version Affinity Programmatic access to Package objects Programmatic access to Package objects

Parent/Child Packages Parent Package Fact ETL DimensionETL Global Variables, Return Code CubeProcessing

Parent/Child Package(s) and UDL Files demo demo

“Best Practices” Agenda Package Design Package Design Data Loading Data Loading Analysis Services Management Analysis Services Management Auditing and Error Handling Auditing and Error Handling Putting it all together - SSABI Putting it all together - SSABI

Data Loading Practices Leveraging High Perf. Interface(s) Leveraging High Perf. Interface(s) – Execute SQL Task Rowset Parameter – ADO Disconnected Recordsets – DTS Multiphase Data Pump Benefits Benefits – Cache Rowset on Client (fast lookups) – Hold updates on Client – Conditional Inserts/Updates w/out Data Driven Query

Conditional Inserts and Updates demo demo

Advanced Practices Approaches Approaches – Multi-Threaded Data Loading – Task Looping – Communication through Global Variables Benefits Benefits – Higher Utilization during Data Load – Compressed Data Loading Window – Easy Orchestration within DTS

Multi-Threaded Data File Load demo demo

“Best Practices” Agenda Package Design Package Design Data Loading Data Loading Analysis Services Management Analysis Services Management Auditing and Error Handling Auditing and Error Handling Putting it all together - SSABIPutting it all together - SSABI

Analysis Services and DTS Analysis Services Processing Task Analysis Services Processing Task – Supports Processing of all OLAP Objects – Customizable Processing through TreeKey and other properties Prominent Properties Prominent Properties – TreeKey ServerName\DBName\CubeFolder\CubeName ServerName\DBName\CubeFolder\CubeName – ItemType 1=OLAP DB, 4=Cube, 7=Partition and 9=Dimension 1=OLAP DB, 4=Cube, 7=Partition and 9=Dimension – ProcessingOption 0=Full Process, 1=Refresh Data and 2=Incremental Update 0=Full Process, 1=Refresh Data and 2=Incremental Update

Analysis Services and DTS Decision Support Objects Decision Support Objects – Provide access to the AS Management Interfaces – Automated via AxScript – Supports Partition Management Creating Creating Cloning Cloning Dropping Dropping Merging of partitions Merging of partitions – Should be an OLAP Administrator to Use

DSO Object Hierarchy

Dynamic Partition Creation and Processing demo demo

Analysis Services and DTS Benefits – Opportunity to Dynamically Manage Partitions and other Objects – Support for Dynamic Processing – Inclusion in the BI Solution Load

“Best Practices” Agenda Package Design Package Design Data Loading Data Loading Analysis Services Management Analysis Services Management Auditing and Error Handling Auditing and Error Handling Putting it all together - SSABIPutting it all together - SSABI

Auditing and Error Handling Logging Options Logging Options – Logging to MSDB – Logging to Event Log – Logging to Text File Enabled through Properties Enabled through Properties – Granularity at Step and Package Level Enabled through Data Pump Properties Enabled through Data Pump Properties – Capture Erroneous Data

Auditing and Error Handling Scripting Options DTSPackageLog Object DTSPackageLog Object – WriteTaskRecord() – WriteStringToLog() Lineage Options DTS Lineage Global Variables DTS Lineage Global Variables – DTSLineage_Full – DTSLineage_Short

Custom Logging using Intrinsic DTS Options demo demo

Auditing and Error Handling Monitoring Execution – Simple as Writing SQL select package.name, package.description, step.stepname, step.errorcode, step.errordescription from msdb..sysdtspackagelog package, msdb..sysdtssteplog step where package.lineagefull = step.lineagefull and step.errorcode <> 0

“Best Practices” Agenda Package Design Package Design Data Loading Data Loading Analysis Services Management Analysis Services Management Auditing and Error Handling Auditing and Error Handling Putting it all together - SSABIPutting it all together - SSABI

ETLM Architecture Overview SourceSystems DTS ETL Subject Matter Db StagingDb KPI’sCubes *.txt DTS DTS

Loading Staging Tables Source Data Requirements Source Data Requirements – Cleansed, validated data – Separate data file for each table – Data files are | delimited Data Load Process Data Load Process – DTS Package calls the BULK INSERT T-SQL Statement – BULK INSERT parameters controlled by DTS Package global variables StagingDb *.txt 1|ABC Co.|110… DTS

Staging Data Options Staging Data Options – Inserts and Updates can be inferred or indicated – Slowly Changing Dimension support Data Load Process Data Load Process – DTS Package calls Stored Procedures – Stored Procedures provide error handling – Transform Data task uses Copy Column transformation Loading Dimension and Fact Tables StagingDb DTS

Processing Dimensions and Cubes Dimensions Dimensions – Incremental vs. Full Processing Cubes Cubes – Data Refresh – Incremental vs. Full Processing – Partitioning by Month DTS KPI’sCubes

SQL Server Accelerator for Business Intelligence demo demo

For More Information Encore Development Encore Development Professional Association for SQL Server Professional Association for SQL Serverwww.sqlpass.org Professional SQL Server 2000 DTS Professional SQL Server 2000 DTS wrox press, DTS newsgroup DTS newsgroupmicrosoft.public.sqlserver.dts SQLServer Central SQLServer Centralwww.sqlservercentral.com SQLDTS Site SQLDTS Sitewww.sqldts.com SQL Server Developers Network SQL Server Developers Networkwww.sqldev.net

Questions? 

Don’t forget to complete the on-line Session Feedback form on the Attendee Web site

© 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.