Informatica & ETL Testing Rahul Parashar Confidential © 2008 Wipro Ltd

Case Study for Testing the Business requirements at Informatica workflow level by manipulating the file in Unix environment. Confidential © 2008 Wipro Ltd 2

Preface In projects where Informatica (or any other tool) is used to perform the data migration , it is driven by certain business rules. This is a case study to validate some of these business rules by manipulating the files in Unix server, it allows testers to validate the business requirements at Informatica workflow level also if Informatica server is installed in Unix, it becomes important for us to perform such validations. Confidential © 2008 Wipro Ltd 3

Validations In ETL Projects there are certain validations like Record Count check. Duplicate data check. Data check in the target file. We usually have to write certain SQL queries to test these validations in all our projects which sometimes is tedious job. To save our time every time when we have to perform such operations (which is generic to all ETL projects) and to make the testing more effective at the workflow level in Informatica we can perform certain steps at Unix level which is being discussed in this presentation.

Sample Business Requirements These are some sample business requirements which we normally come across in data migration projects. Source file should have two line header. 2. Source file should have the count of records in the trailer. 3. Source file should not have more then 50% of the records available in the file for previous business date (max count) 4. File taken as input should have min 10 records (min count). Confidential © 2008 Wipro Ltd

There are two ways of testing the above requirements :- Validation Procedures There are two ways of testing the above requirements :- FTP the file to local drive, make the necessary changes and then upload the file again and run the Informatica mapping B. Using the VI editor, open and edit the file in Unix server itself.

FTP Method Go the source location of the file using the run command

Get the file into the local drive H:\\, but before you get the file into the local drive please make sure that you have all the permissions to manipulate that file else you will see a 0 byte file being ftp’ed to your local drive.

Once you get the file into the H:\\ drive open the file and 1. Remove the header of the file. 2. Then save the file. 3. After that ftp the file into your local directory (using ‘put’ command) using the ftp server. 4. Then copy the same file into the Informatica source files location. 5. Run the Informatica workflow.

We can perform the other operations to validate the trailer check and record count check can be performed by modifying the file and uploading the file at source location using the FTP commands.

Using VI Editor VI Editor is file editor used in Unix to manipulate the files. To manipulate the data in the file we must have all the access rights to the file. To achieve this first copy the file in your local directory in Unix server by using the command cp sample.csv /home/local/ . Once the file is available in the local directory check for the permissions available for you on the file. Change the permissions to RWX by using the chmod command if you don't have all the access rights. Open the file with VI editor using command :- VI File name

Remove the header by ‘dd’ command in VI editor then save the changes in the file and put the file back into the Informatica source location and run the workflow The workflow should fail at the step where header validation is done. Similarly we can perform other validations by removing the trailer and then changing the number of records in the file by copying the data for other two business requirements.

Validation Results For The first business requirement that we are validating where the file is having two line header the workflow should run successfully when the file is having two line header Change the header of the file by inserting some junk values. The Workflow should fail when the header values are changed to some junk data Remove both the headers and run the workflow it should fail at the step where header validation is done. Remove one of the headers and run the workflow, the workflow should fail. These are the few validations that can be performed while validating the results and making sure the business rules are implemented properly in the mapping.

Validation Results Similarly for second validation where we have to check the trailer records we can perform following validations. Remove the trailer and run the workflow, the workflow should fail at the step where trailer validation is performed. Change the trailer value to some junk value (other then the total number of records), mapping should fail Keep the correct trailer record the mapping should Pass the trailer check step. Insert some alphanumeric values in the trailer record, the mapping should fail in this case. Similarly we can validate other business requirements. By changing the file using VI editor or using the FTP commands.

Sample Mapping This the sample mapping in Informatica for header trailer and count validation. When Header validation fails this step should fail in the mapping.

Informatica steps And for trailer, max count and Min count validations we have following mappings

Informatica View If workflow succeeds then in workflow monitor following is displayed

Informatica Workflow monitor View If header check fails in workflow monitor following is displayed

Important points We can verify which step in Informatica workflow has failed by checking the same in the workflow monitor. All set of business rules for the files to be migrated using Informatica (or any other tool used for ETL) can be verified following this approach. This approach allows the tester to validate the business requirements at the workflow level. These validations are specific to the files which are to be migrated using the ETL tool before the migration process.

Points to remember There are certain points that must be kept in mind while performing the operations on file in Unix. Always keep the backup of the file in your local drive. Always make sure that the file which you are referring to is taken as the input for the workflow being tested. While performing file manipulation in VI editor, make a copy as (sample_backup.csv) in your directory. Always keep in mind the target table whether it is truncate or progression load.

Thank You Confidential © 2008 Wipro Ltd 21