1. Automating the testing of BI Solutions wih NBi
Cédric L. Charlier
@Seddryck

2. Open-sources contributions
Skills & Interests
Cédric L. Charlier
@Seddryck
In a nutshell
Database architecture
Data modeling
Business intelligence
Data warehousing
ETL
Olap/Tabular
No SQL
Big data
Data analysis
Machine learning
seddryck.wordpress.com
Data & BI architect
Agile data warehouse
Quality-oriented
Automation fanboy
Open-sources contributions
Testing framework for BI
Other projects on github
Deployment SSRS - RsPackage
Starting SSIS from .Net - Cassis
Toolset for special cases of modeling: ERMine et Tibre
And much more
Recognitions
International speaker
Microsoft MVP Data Platform award
Experience
15 years avec SQL Server
Former consultant, working internally for Elia

3. Agenda Why automating is a must? Tests with queries (Sql/Mdx/Dax)
Tests with ETL packages (SSIS)
Testing models (SSAS and DB engine)
Automating the creation of test-suites

4. What’s automated testing?
A piece of software written to reproduce steps/actions defined in a manual process

5. Benefits of automation
Speed up testing to accelerate releases
Allow testing to happen more frequently
Improve test coverage
Ensure consistency

6. Tests automation doesn’t apply to Business Intelligence

7. More than 33% of BI projects fail to deliver
Member of the Asscoiated Press . Aenean commodo ligula eget dolor. Aenean. Aenean commodo ligula eget dolor. Aenhswse. Cejhciebce fcdcdcd.
ILLUSTRATED WEEKLY NEWSPAPER
Est. 1869
Wednesday, November 24, 2012
Price 6d
More than 33% of BI projects fail to deliver
Lack of agility. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoqueCum sociis natoqueCum sociis natoque
Low quality of deliverables Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturientAenean commodo ligula eget dolor. Aenean massa. Cum sociis natoqueCum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec,
Simply Too late.
Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec,

8. We haven’t time to create these tests

9. Time to execute a test-suite
Exponential slope
Why?
New objects
Old objects

10. Time to write a test-suite
Initial gap

11. No existing framework for automating tests dedicated to BI
… I’m not proficient enough in C#

12. This framework exists: NBi
No C#, just xml syntax
This framework exists: NBi
Open-source

13. Agenda Why automating is a must? Tests with queries (Sql/Mdx/Dax)
Tests with ETL packages (SSIS)
Testing models (SSAS and DB engine)
Automate the creation of test-suites

14. Compare queries … result-sets
System-under-test
Assert
Cube and data warehouse
MDX
SQL
… or any brilliant idea
Previous release
Two different releases of a same data warehouse
Tabular model versus Xmlfile
DAX
Tabular model versus Csv file
DAX

15. Demo: compare queries’ result
Compare cube/operational database
101
Compare cube/flat files
Defaults and References
Tolerance
Intervals

16. Other testing goals for queries
Performance of a query
Executed in less than x seconds
Cache management
Syntax of a query
Extract queries from reports and assemblies

17. Query’s result and format
Result’s format
Example: all the cells have effectively 2 digits after the decimal separator and this separator is a comma.

18. Assert row count Assert how many rows are returned in the result-set.
Equal, more-than, less-than

19. Agenda Why automating is a must? Tests with queries (Sql/Mdx/Dax)
Tests with ETL packages (SSIS)
Testing models (SSAS and DB engine)
Automate the creation of test-suites

20. Demo SSIS Assert the execution status of an ETL
Alternatively test performances

21. Unique index violation
Inside this ETL
insert
Unique index violation
Dimension
Calendar
Dimension
Geography
Dimension Scientist

22. Setup and Cleanup Setup: Cleanup:
Define the state of your testing environment before executing a test
Cleanup:
Remove consequences of your tests on the testing environment

23. Tasks and behaviors Available tasks
Truncate/Load tables with csv files
Run sql batch, exe or batches
Run Etl packages
Move or delete files
Start/Stop Windows services
Run-once: execute only once this set of tasks
Parallel: run a set of tasks in parallel

24. Demo Setup Setup to register a clean state before test’s execution
Assert database’s state after the execution of an ETL

25. Limitations ETLs must be stored
SSIS Catalog
File share
SQL Server
Tests must be executed on the server running the ETL
No remote testing
Supports SQL Server 2008R2, 2012 and 2014

26. Agenda Why automating is a must? Tests with queries (Sql/Mdx/Dax)
Tests with ETL packages (SSIS)
Testing models (SSAS and DB engine)
Automate the creation of test-suites

27. Testing dimension’s members
“Italy” is a member of dimension “Country”
“Italy” is not a member of the set “Nordic countries”
Dimension “Country” has more than 190 members.
All members of dimension “Country” have at least 4 characters
Members of hierarchy “Month” are ordered chronologically (and not alphabetically)
Members are equivalent to members of this other instance …

28. Example: members’ pattern

29. Testing structure (SSAS)
Assert that a dimension, hierarchy, measure, set, … is visible for end-user in a given perspective.
Assert the existence of a relation between a measure-group and a dimension.

30. Testing structure (DB engine)
Assert existence of a
schema,
Table
Column
Routine (stored procedure or function)
Assert the type of a column
Full type: decimal(10,3)
Light type: decimal

31. Agenda Why automating is a must? Tests with queries (Sql/Mdx/Dax)
Tests with ETL packages (SSIS)
Testing models (SSAS and DB engine)
Automate the creation of test-suites

32. Some tests are really repetitive
It should be possible to automate the creation

33. It exists! Take a look at Genbi
A set of test-cases
It exists! Take a look at Genbi
A template for the tests

34. Variables to be substituted (surrounded by $)
First steps with Genbi
File loaded from Genbi
Variables to be substituted (surrounded by $)

35. Demo: Genbi Load test-cases Select a template Generate the test-suite
From a flat file
Select a template
Generate the test-suite
Grouping option

36. I’m scripting macros to automate the actions performed by genbi …
With genbiL!

37. Sample for genbiL
Subject
Action
Parameters

38. Demo for genbiL
Command line

39. Conclusions Benefits of automated tests NBi Save Time (Time-on-market)
Better Quality (end-result)
More confidence
NBi
Supports large range of different tests
No need of compiler (neither C#)
Good feedback when some tests are failing
Open-Source
Automate the creation of your test-suites (genbi / genbiL)

40. Read further http://www.nbi.io/ Download releases Read documentation
Report bugs
Suggest new features