Automating the testing of your BI solutions with NBi Cédric L. Charlier
Thank you to our sponsors!
Who am I? Business Intelligence, Data & Information Architect MVP SQL Server from Belgium Principal committer for NBi Open source contributor Agile enthusiast
Agenda Why automating is a must? Tests with queries (Sql/Mdx/Dax) Tests with ETL packages (SSIS) Testing models (SSAS and DB engine) Automating the creation of test-suites
What’s automated testing? A piece of software written to reproduce steps/actions defined in a manual process
Benefits of automation Allow testing to happen more frequently Speed up testing to accelerate releases Improve test coverage Ensure consistency
Tests automation doesn’t apply to Business Intelligence
Wednesday, November 24, 2012 Est Price 6d More than 33% of BI projects fail to deliver Lack of agility. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoqueCum sociis natoqueCum sociis natoque Low quality of deliverables Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturientAenean commodo ligula eget dolor. Aenean massa. Cum sociis natoqueCum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, Simply Too late. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, Member of the Asscoiated Press. Aenean commodo ligula eget dolor. Aenean. Aenean commodo ligula eget dolor. Aenhswse. Cejhciebce fcdcdcd. ILLUSTRATED WEEKLY NEWSPAPER
We haven’t time to create these tests
Time to execute a test-suite Exponential slope Why? -New objects -Old objects
Time to write a test-suite Initial gap
No existing framework for automating tests dedicated to BI … I’m not proficient enough in C#
This framework exists: NBi No C#, just xml syntax Open- source
Agenda Why automating is a must? Tests with queries (Sql/Mdx/Dax) Tests with ETL packages (SSIS) Testing models (SSAS and DB engine) Automate the creation of test-suites
Compare queries … result-sets System-under-test Assert Previous release Two different releases of a same data warehouse Tabular model versus Csv file DAX … or any brilliant idea Cube and data warehouse MDX SQL
Demo: compare queries’ result Compare cube/operational database 101 Compare cube/flat files Defaults and References Tolerance Intervals
Other testing goals for queries Performance of a query Executed in less than x seconds Cache management Syntax of a query Extract queries from reports and assemblies
Query’s result and format Result’s format Example: all the cells have effectively 2 digits after the decimal separator and this separator is a comma.
Check calculations in your query Assert a logical test, row by row, for you result-set Example: Calculate the total price from the unit price, quantity and discount percentage. If at least one row doesn’t validate this logical test, the whole test shift to red
Assert row count Assert how many rows are returned in the result-set. Equal, more-than, less-than
Agenda Why automating is a must? Tests with queries (Sql/Mdx/Dax) Tests with ETL packages (SSIS) Testing models (SSAS and DB engine) Automate the creation of test-suites
Demo SSIS Assert the execution status of an ETL Alternatively test performances
Inside this ETL Dimension Calendar Dimension Scientist insert Unique index violation Dimension Geography
Setup and Cleanup Setup: Define the state of your testing environment before executing a test Cleanup: Remove consequences of your tests on the testing environment
Tasks and behaviors Available tasks Truncate/Load tables with csv files Run sql batch, exe or batches Run Etl packages Move or delete files Start/Stop Windows services Run-once: execute only once this set of tasks Parallel: run a set of tasks in parallel
Demo Setup Setup to register a clean state before test’s execution Assert database’s state after the execution of an ETL
Limitations ETLs must be stored SSIS Catalog File share SQL Server Tests must be executed on the server running the ETL No remote testing v1.10 supports SQL Server 2014 v1.11 supports SQL Server 2008R2, 2012 and 2014 Currently in preview
Agenda Why automating is a must? Tests with queries (Sql/Mdx/Dax) Tests with ETL packages (SSIS) Testing models (SSAS and DB engine) Automate the creation of test-suites
Testing dimension’s members “Norway” is a member of dimension “Country” “Norway” is not a member of the set “European Union” Dimension “Country” has more than 190 members. All members of dimension “Country” have at least 4 characters Members of hierarchy “Month” are ordered chronologically (and not alphabetically) Members are equivalent to members of this other instance …
Example: members’ pattern
Testing structure (SSAS) Assert that a dimension, hierarchy, measure, set, … is visible for end-user in a given perspective. Assert the existence of a relation between a measure-group and a dimension.
Testing structure (DB engine) Assert existence of a schema, Table Column Routine (stored procedure or function) Assert the type of a column Full type: decimal(10,3) Light type: decimal Currently in preview
Agenda Why automating is a must? Tests with queries (Sql/Mdx/Dax) Tests with ETL packages (SSIS) Testing models (SSAS and DB engine) Automate the creation of test-suites
Some tests are really repetitive It should be possible to automate the creation
It exists! Take a look at Genbi A template for the tests A set of test- cases
First steps with Genbi File loaded from Genbi Variables to be substituted (surrounded by $)
Demo: Genbi Load test-cases From a flat file Select a template Generate the test-suite Grouping option
I’m scripting macros to automate the actions performed by genbi … With genbiL!
Sample for genbiL Subject Action Parameters
Demo for genbiL Command line
Conclusions Benefits of automated tests Save Time (Time-on-market) Better Quality (end-result) More confidence NBi Supports large range of different tests No need of compiler (neither C#) Good feedback when some tests are failing Open-Source Automate the creation of your test-suites (genbi / genbiL)
Read further Download releases Read documentation Report bugs Suggest new features
Your feedback is important! Please let us know what you thought of this session
Thank you to the organizers and volunteers!