Automating the testing of your BI solutions with NBi Cédric L. Charlier

Thank you to our sponsors!

Who am I?  Business Intelligence, Data & Information Architect  MVP SQL Server from Belgium  Principal committer for NBi  Open source contributor  Agile enthusiast 

Agenda  Why automating is a must?  Tests with queries (Sql/Mdx/Dax)  Tests with ETL packages (SSIS)  Testing models (SSAS and DB engine)  Automating the creation of test-suites

What’s automated testing? A piece of software written to reproduce steps/actions defined in a manual process

Benefits of automation Allow testing to happen more frequently Speed up testing to accelerate releases Improve test coverage Ensure consistency

Tests automation doesn’t apply to Business Intelligence

Wednesday, November 24, 2012 Est Price 6d More than 33% of BI projects fail to deliver Lack of agility. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoqueCum sociis natoqueCum sociis natoque Low quality of deliverables Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturientAenean commodo ligula eget dolor. Aenean massa. Cum sociis natoqueCum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, Simply Too late. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, Member of the Asscoiated Press. Aenean commodo ligula eget dolor. Aenean. Aenean commodo ligula eget dolor. Aenhswse. Cejhciebce fcdcdcd. ILLUSTRATED WEEKLY NEWSPAPER

We haven’t time to create these tests

Time to execute a test-suite Exponential slope Why? -New objects -Old objects

Time to write a test-suite Initial gap

No existing framework for automating tests dedicated to BI … I’m not proficient enough in C#

This framework exists: NBi No C#, just xml syntax Open- source

Agenda  Why automating is a must?  Tests with queries (Sql/Mdx/Dax)  Tests with ETL packages (SSIS)  Testing models (SSAS and DB engine)  Automate the creation of test-suites

Compare queries … result-sets System-under-test Assert Previous release Two different releases of a same data warehouse Tabular model versus Csv file DAX … or any brilliant idea Cube and data warehouse MDX SQL

Demo: compare queries’ result  Compare cube/operational database  101  Compare cube/flat files  Defaults and References  Tolerance  Intervals

Other testing goals for queries  Performance of a query  Executed in less than x seconds  Cache management  Syntax of a query  Extract queries from reports and assemblies

Query’s result and format  Result’s format  Example: all the cells have effectively 2 digits after the decimal separator and this separator is a comma.

Check calculations in your query  Assert a logical test, row by row, for you result-set  Example: Calculate the total price from the unit price, quantity and discount percentage.  If at least one row doesn’t validate this logical test, the whole test shift to red

Assert row count  Assert how many rows are returned in the result-set.  Equal, more-than, less-than

Agenda  Why automating is a must?  Tests with queries (Sql/Mdx/Dax)  Tests with ETL packages (SSIS)  Testing models (SSAS and DB engine)  Automate the creation of test-suites

Demo SSIS  Assert the execution status of an ETL  Alternatively test performances

Inside this ETL Dimension Calendar Dimension Scientist insert Unique index violation Dimension Geography

Setup and Cleanup  Setup:  Define the state of your testing environment before executing a test  Cleanup:  Remove consequences of your tests on the testing environment

Tasks and behaviors  Available tasks  Truncate/Load tables with csv files  Run sql batch, exe or batches  Run Etl packages  Move or delete files  Start/Stop Windows services  Run-once: execute only once this set of tasks  Parallel: run a set of tasks in parallel

Demo Setup  Setup to register a clean state before test’s execution  Assert database’s state after the execution of an ETL

Limitations  ETLs must be stored  SSIS Catalog  File share  SQL Server  Tests must be executed on the server running the ETL  No remote testing  v1.10 supports SQL Server 2014  v1.11 supports SQL Server 2008R2, 2012 and 2014 Currently in preview

Agenda  Why automating is a must?  Tests with queries (Sql/Mdx/Dax)  Tests with ETL packages (SSIS)  Testing models (SSAS and DB engine)  Automate the creation of test-suites

Testing dimension’s members  “Norway” is a member of dimension “Country”  “Norway” is not a member of the set “European Union”  Dimension “Country” has more than 190 members.  All members of dimension “Country” have at least 4 characters  Members of hierarchy “Month” are ordered chronologically (and not alphabetically)  Members are equivalent to members of this other instance  …

Example: members’ pattern

Testing structure (SSAS)  Assert that a dimension, hierarchy, measure, set, … is visible for end-user in a given perspective.  Assert the existence of a relation between a measure-group and a dimension.

Testing structure (DB engine)  Assert existence of a  schema,  Table  Column  Routine (stored procedure or function)  Assert the type of a column  Full type: decimal(10,3)  Light type: decimal Currently in preview

Agenda  Why automating is a must?  Tests with queries (Sql/Mdx/Dax)  Tests with ETL packages (SSIS)  Testing models (SSAS and DB engine)  Automate the creation of test-suites

Some tests are really repetitive It should be possible to automate the creation

It exists! Take a look at Genbi A template for the tests A set of test- cases

First steps with Genbi File loaded from Genbi Variables to be substituted (surrounded by $)

Demo: Genbi  Load test-cases  From a flat file  Select a template  Generate the test-suite  Grouping option

I’m scripting macros to automate the actions performed by genbi … With genbiL!

Sample for genbiL Subject Action Parameters

Demo for genbiL  Command line

Conclusions  Benefits of automated tests  Save Time (Time-on-market)  Better Quality (end-result)  More confidence  NBi  Supports large range of different tests  No need of compiler (neither C#)  Good feedback when some tests are failing  Open-Source  Automate the creation of your test-suites (genbi / genbiL)

Read further  Download releases  Read documentation  Report bugs  Suggest new features

Your feedback is important! Please let us know what you thought of this session

Thank you to the organizers and volunteers!