Oracle Data Integrator Data Quality (Integrity Control)

6-2 Objectives Know the different types of data quality business rules ODI manages. Be able to enforce data quality with ODI. Understand how to create constraints on datastores. After completing this lesson, you will:

6-3 When to Enforce Data Quality? The IS can be broken into 3 sub-systems Source application(s) Data integration process(es) Target application(s) Data Quality should be managed in all three sub-systems ODI provides the solution for enforcing quality in all three.

6-4 Data Quality Business Rules Defined by designers and business analysts Stored in the Metadata repository May be applied to application data Defined in two ways: Automatically retrieved with other metadata Rules defined in the databases Obtained by reverse-engineering Manually entered by designers User-defined rules

6-5 From Business Rules to Constraints De-duplication rules Primary Keys Alternate Keys Unique Indexes Reference rules Simple: column A = column B Complex: column A = function(column B, column C) Validation rules Mandatory Columns Conditions

6-6 Source CORRECTIONS File ORDERS LINES Target SALES Overview of the Data Quality System Integration Process Error Recycling is performed - by Interfaces Static Control is started - by Interfaces after integration - by Packages - manually Static Control is started - Automatically (scheduled) - manually Errors Flow Control is started - by Interfaces during execution Errors

6-7 Constraints in ODI Mandatory Columns Keys Primary Keys Alternate Keys Indexes References Simple: column A = column B Complex: column A = function(column B) Conditions

6-8 1.Double-click the column in the Models view. 2.Select the Control tab. 3.Check the Mandatory option. 4.Select when the constraint should be checked (Flow/Static). Mandatory Columns

6-9 1.Select the Constraints node under the datastore. 2.Right-click, select Insert Key. 3.Fill in the Name. 4.Select the Key or Index Type 5.Go to the Columns tab 6.Add/remove columns from the key. Keys

Go to the Control tab. 2.Select whether the key is Defined in the Database, and is Active 3.Select when the constraint must be checked (Flow/Static). 4.Click the Check button to perform a synchronous check of the key. Checking Existing Data with a New Key Number of duplicate rows

Select the Constraints node under the datastore 2.Right-click, select Insert Reference 3.Fill in the Name 4.Select the reference type User Reference Complex Reference 5.Select a Parent Model and Table Set the model and table to to manually enter the catalog, schema and table name. Creating a Reference

Go to the Columns tab 2.Click the Add button 3.Select the column from the Foreign Key table. 4.Select the corresponding column from the Primary Key table. 5.Repeat for all column pairs in the reference. Creating a User Reference

Go to the Expression tab 2.Set the Alias for the Primary Key table. 3.Code the Expression Prefix with the tables aliases Use the Expression Editor. Creating a Complex Reference

Go to the Control tab. 2.Choose when the constraint should be checked (Flow/Static). 3.Click the Check button to immediately check the reference. Not possible for heterogeneous references. Checking Existing Data with a New Reference

Right-click Constraints node, select Insert Condition 2.Fill in the Name. 3.Select ODI Condition type. 4.Edit the condition clause Use the Expression Editor 5.Type in the error message for the condition. Creating a Condition

Go to the Control tab 2.Select when the constraint must be checked (Flow/Static). 3.Click the Check button to perform a synchronous check of the condition. Checking Existing Data with a New Condition

6-17 How to Enforce Data Quality in an Interface The general process: 1.Enable Static/Flow Control 2.Set the options 3.Select the Constraints to enforce Table constraints Not null columns 4.Review the erroneous records

6-18 How to Enable Static/Flow Control 1.Go to the interface’s Flow tab. 2.Select the target datastore. The IKM properties panel appears. 3.Set the FLOW_CONTROL and/or STATIC_CONTROL IKM options to “Yes”. 4.Set the RECYCLE_ERRORS to “Yes”, if you want to recycle errors from previous runs

6-19 How to Set the Options 1.Select the interface’s Controls tab. 2.Select a CKM. 3.Set up the CKM Options. 4.Set the Maximum Number of Errors Allowed. Leave blank to allow an unlimited number of errors. To specify a percentage of the total number of integrated records, check the % option.

6-20 How to Select Which Constraints to Enforce For flow control: For most constraints: 1.Select the interface’s Controls tab. 2.For each constraint you wish to enforce, select Yes. For Not Null constraints: 1.Select the interface’s Diagram tab. 2.Select the Target datastore column that you wish to check for nulls. 3.In the column properties panel, select Check Not Null.

6-21 How to Review Erroneous Records First, execute your interface. To see the number of records: 1.Select the Execution tab. 2.Find the most recent execution. The No. of Errors encountered by the interface is displayed. To see which records were rejected: 1.Select the target datastore in the Models view. 2.Right-click > Control > Errors… 3.Review the erroneous rows.