1. Moving Data from Oracle to SQL with Biml
SQL Saturday #604, 4/8/2017
Arthur Wang

2. Topics What is Biml? Sourcing from Oracle Initial setup
BimlExpress vs. Mist vs. BIDS Helper
Walkthrough/demo
Recap
Links

3. What is Biml? XML dialect Generates SSIS XML Much cleaner than SSIS
Lets you embed C#
XML dialect
Follows the same syntax rules as other XML dialects (e.g. opening and closing tags, with attributes and child elements)
“Generates” – you’ll also see the terms “emits” and “compiles”
Embedding C#
Lets you encapsulate for repetitive tasks (i.e. building patterns)

4. Sourcing from Oracle OLE DB providers exist but are deprecated
General recommendation: use Attunity drivers
Some work is required to convert data types

5. Initial setup
Install Oracle Data Access Components (ODAC) or Oracle Client
Perform setup on ETL hosts
Perform setup on dev machine
Match bitness
Install Attunity connector for Oracle
Match to your version of SQL Server and Oracle
Install Biml compiler
ODAC contains all the drivers you need, but you may want client tools such as SQL Developer
The machines that actually execute packages need to have ODAC/Client installed
You will need to reboot before SSIS can use the drivers
You also need it on your dev machine to establish connections to Oracle DBs
Processor architecture:
If your SSIS machine is running 64-bit, install 64-bit ODAC/Client
You should install 32-bit on your dev machine, as Visual Studio is a 32-bit application
You may wish to install 64-bit ODAC alongside 32-bit if you want to execute it outside of Visual Studio (e.g. deploy to a local SSIS catalog for testing)
Attunity connector: match your version of SQL Server and Oracle
64-bit install should be fine, as it installs the 32-bit version as well
Biml compiler:
BimlExpress
Mist/Biml Studio
BIDS Helper

6. BimlExpress vs. Mist vs. BIDS Helper
Mist: official IDE for Biml created by Varigence
Reverse-engineers packages
Can generate multiple SSIS projects
Handles all elements in Biml schema
BIDS Helper has a version of the Biml engine
Add-in for VS 2015 doesn’t support Biml
BimlExpress: official free add-in for Visual Studio
Mist
Most features
“Handles all elements” – we’ll cover this during the demo
BIDS Helper
The link goes to the add-in for VS 2013 or earlier
To install in VS 2015, download via VS Gallery
The VS 2015 version doesn’t yet support Biml (as of 1/25/2017)
BimlExpress
Includes syntax highlighting (missing in BIDS Helper)
Should update more frequently than the Biml engine included in BIDS Helper

7. Walkthrough/demo

8. Recap

9. Connecting to Oracle Use OracleConnection element
Connection string: “Data Source=[service];User Id=[user ID]”
Use an Easy Connect string for the service
Use expressions for project connection manager passwords
You really only need to use an expression if your project is DontSaveSensitive, and if you need to execute the package locally before deploying
If you’re just going to deploy it right away, then you can configure the .conmgr to use an environment variable for the password
Easy Connect strings mean you don’t have to worry about having tnsnames.ora and configuring the TNS_ADMIN environment variable
However, it does require Oracle 10g net services installed on the client

10. Generating Oracle Source components
Use CustomComponent syntax
Copy most attributes and custom properties from .dtsx
Need to declare all columns and paths
Need to map data types from Oracle to Biml
Mist will support third-party components (such as Attunity’s Oracle Source component), as long as they’re part of the Biml schema
No guarantee that BimlExpress will ever support it
If you try to do an implicit conversion (e.g. by taking an Oracle NUMBER column and declaring the Biml element as an Int64 – a.k.a. bigint), you’ll probably get an error
You can identify what type SSIS will expect by manually adding an Oracle Source in the SSIS designer, and opening up the Advanced Editor to inspect metadata

11. Handling multiple packages
Project connection managers get regenerated with new GUIDs
Workaround: manually specify GUIDs, and don’t delete/overwrite connection managers
Use the include directive to encapsulate
There are other workarounds to the project connection manager issue – see Bill Fellows’ blog post:
Manually specifying the GUID is probably the quickest and easiest way
Save yourself manual effort by putting references in a separate file that gets included
You can use the include directive in other ways as well – for example, by putting all custom functions inside an included file

12. Using metadata to generate packages
Use Oracle Data Provider for .NET (ODP .NET) to get metadata from Oracle
Pass in restrictions to limit amount of metadata returned
Retrieve metadata to iterate through schemas, tables, columns, primary keys, etc.
Restrictions = filters on your GetSchema method call

13. Handle Oracle data types
ODP .NET returns the Oracle data type
Write a custom function to convert Oracle to Biml data types
Use the <#+ delimiter to define functions and classes
Some additional data conversion may be required (e.g. through Data Conversion transformation)
Oracle data types include NUMBER, VARCHAR2, DATE, etc.
You can create a function that takes in the column’s DataRow, and returns the correct Biml data type (e.g. Double, AnsiString, DateTime)
Enclose any function or class definitions inside <#+ and #>
Certain types, such as float, are incompatible with the desired use case (such as the target of a Lookup transformation)
In this case, apply a data conversion task
You’ll need more code to handle the logic and additional XML

14. Links
Initial setup:
Oracle Data Access Components (ODAC): 32-bit and 64-bit
Installing ODAC:
Oracle Client, Database, and Example files: edition/downloads/database12c-win64-download html
Attunity Connector: 2.0, 3.0, 4.0
BimlExpress:
Biml language reference:
Oracle schema collections:
Despite the Microsoft white paper being ~7 years old, it’s still basically accurate (as of April 2017):
Install Oracle client/data access components, matching your processor architecture
Additionally install 32-bit Oracle client/data access components on your dev machine to match Visual Studio
Install the Attunity Oracle connector

15. Credits Cathrine Wilhelmsen: SQL Server, SSIS and Biml Data Types
Adeeb Mass'ad: Mapping Oracle NUMBER Data Types to SSIS Data Types
Roelant Vos: making SSIS Project Connections generate correctly using BIML Express

16. Better Business Intelligence. Period.
About Talavant
There is a better way to make data work for companies. Better resources, strategy, sustainability, inclusion of the organization as a whole, understanding of client needs, tools, outcomes, better ROI.
Better Business Intelligence. Period.
STRATEGY
ARCHITECTURE
IMPLEMENTATION
Sustainability
Dependable, worry-free integration
Accelerated planning, implementation and results
Increased company-wide buy-in & usage
VALUE WE PROVIDE
By providing a holistic approach inclusive of a client’s people, processes and technologies - built on investment in our own employees and company growth.
HOW WE DO IT