Moving Data from Oracle to SQL with Biml SQL Saturday #604, 4/8/2017 Arthur Wang

Topics What is Biml? Sourcing from Oracle Initial setup BimlExpress vs. Mist vs. BIDS Helper Walkthrough/demo Recap Links

What is Biml? XML dialect Generates SSIS XML Much cleaner than SSIS Lets you embed C# XML dialect Follows the same syntax rules as other XML dialects (e.g. opening and closing tags, with attributes and child elements) “Generates” – you’ll also see the terms “emits” and “compiles” Embedding C# Lets you encapsulate for repetitive tasks (i.e. building patterns)

Sourcing from Oracle OLE DB providers exist but are deprecated General recommendation: use Attunity drivers Some work is required to convert data types

Initial setup Install Oracle Data Access Components (ODAC) or Oracle Client Perform setup on ETL hosts Perform setup on dev machine Match bitness Install Attunity connector for Oracle Match to your version of SQL Server and Oracle Install Biml compiler ODAC contains all the drivers you need, but you may want client tools such as SQL Developer The machines that actually execute packages need to have ODAC/Client installed You will need to reboot before SSIS can use the drivers You also need it on your dev machine to establish connections to Oracle DBs Processor architecture: If your SSIS machine is running 64-bit, install 64-bit ODAC/Client You should install 32-bit on your dev machine, as Visual Studio is a 32-bit application You may wish to install 64-bit ODAC alongside 32-bit if you want to execute it outside of Visual Studio (e.g. deploy to a local SSIS catalog for testing) Attunity connector: match your version of SQL Server and Oracle 64-bit install should be fine, as it installs the 32-bit version as well Biml compiler: BimlExpress Mist/Biml Studio BIDS Helper

BimlExpress vs. Mist vs. BIDS Helper Mist: official IDE for Biml created by Varigence Reverse-engineers packages Can generate multiple SSIS projects Handles all elements in Biml schema BIDS Helper has a version of the Biml engine Add-in for VS 2015 doesn’t support Biml BimlExpress: official free add-in for Visual Studio Mist Most features “Handles all elements” – we’ll cover this during the demo BIDS Helper The link goes to the add-in for VS 2013 or earlier To install in VS 2015, download via VS Gallery The VS 2015 version doesn’t yet support Biml (as of 1/25/2017) BimlExpress Includes syntax highlighting (missing in BIDS Helper) Should update more frequently than the Biml engine included in BIDS Helper

Walkthrough/demo

Recap

Connecting to Oracle Use OracleConnection element Connection string: “Data Source=[service];User Id=[user ID]” Use an Easy Connect string for the service Use expressions for project connection manager passwords You really only need to use an expression if your project is DontSaveSensitive, and if you need to execute the package locally before deploying If you’re just going to deploy it right away, then you can configure the .conmgr to use an environment variable for the password Easy Connect strings mean you don’t have to worry about having tnsnames.ora and configuring the TNS_ADMIN environment variable However, it does require Oracle 10g net services installed on the client

Generating Oracle Source components Use CustomComponent syntax Copy most attributes and custom properties from .dtsx Need to declare all columns and paths Need to map data types from Oracle to Biml Mist will support third-party components (such as Attunity’s Oracle Source component), as long as they’re part of the Biml schema No guarantee that BimlExpress will ever support it If you try to do an implicit conversion (e.g. by taking an Oracle NUMBER column and declaring the Biml element as an Int64 – a.k.a. bigint), you’ll probably get an error You can identify what type SSIS will expect by manually adding an Oracle Source in the SSIS designer, and opening up the Advanced Editor to inspect metadata

Handling multiple packages Project connection managers get regenerated with new GUIDs Workaround: manually specify GUIDs, and don’t delete/overwrite connection managers Use the include directive to encapsulate There are other workarounds to the project connection manager issue – see Bill Fellows’ blog post: http://billfellows.blogspot.com/2016/09/resolving-biml-project-level-connection.html Manually specifying the GUID is probably the quickest and easiest way Save yourself manual effort by putting references in a separate file that gets included You can use the include directive in other ways as well – for example, by putting all custom functions inside an included file

Using metadata to generate packages Use Oracle Data Provider for .NET (ODP .NET) to get metadata from Oracle Pass in restrictions to limit amount of metadata returned Retrieve metadata to iterate through schemas, tables, columns, primary keys, etc. Restrictions = filters on your GetSchema method call

Handle Oracle data types ODP .NET returns the Oracle data type Write a custom function to convert Oracle to Biml data types Use the <#+ delimiter to define functions and classes Some additional data conversion may be required (e.g. through Data Conversion transformation) Oracle data types include NUMBER, VARCHAR2, DATE, etc. You can create a function that takes in the column’s DataRow, and returns the correct Biml data type (e.g. Double, AnsiString, DateTime) Enclose any function or class definitions inside <#+ and #> Certain types, such as float, are incompatible with the desired use case (such as the target of a Lookup transformation) In this case, apply a data conversion task You’ll need more code to handle the logic and additional XML

Links Initial setup: https://technet.microsoft.com/en-us/library/ee470675(v=sql.100).aspx Oracle Data Access Components (ODAC): 32-bit and 64-bit Installing ODAC: https://www.dba-resources.com/oracle/oracle-data-access-components-odac/ Oracle Client, Database, and Example files: http://www.oracle.com/technetwork/database/enterprise- edition/downloads/database12c-win64-download-2297732.html Attunity Connector: 2.0, 3.0, 4.0 BimlExpress: https://www.varigence.com/BimlExpress Biml language reference: https://www.varigence.com/Documentation/Language/Index Oracle schema collections: https://msdn.microsoft.com/en-us/library/cc716723(v=vs.110).aspx Despite the Microsoft white paper being ~7 years old, it’s still basically accurate (as of April 2017): Install Oracle client/data access components, matching your processor architecture Additionally install 32-bit Oracle client/data access components on your dev machine to match Visual Studio Install the Attunity Oracle connector

Credits Cathrine Wilhelmsen: SQL Server, SSIS and Biml Data Types Adeeb Mass'ad: Mapping Oracle NUMBER Data Types to SSIS Data Types Roelant Vos: making SSIS Project Connections generate correctly using BIML Express

Better Business Intelligence. Period. About Talavant There is a better way to make data work for companies. Better resources, strategy, sustainability, inclusion of the organization as a whole, understanding of client needs, tools, outcomes, better ROI. Better Business Intelligence. Period. STRATEGY ARCHITECTURE IMPLEMENTATION Sustainability Dependable, worry-free integration Accelerated planning, implementation and results Increased company-wide buy-in & usage VALUE WE PROVIDE By providing a holistic approach inclusive of a client’s people, processes and technologies - built on investment in our own employees and company growth. HOW WE DO IT