1. Automating the testing of BI Solutions wih NBi
Cédric L. Charlier
@Seddryck

2. Our Sponsors
If you think, that a SQL Saturday is a nice possibility to learn from and network with fellow SQL Server enthusiasts FOR FREE,
I just ask you one thing: Visit the sponsor booths and chat with the sponsors!
They are covering the expenses for each and every of you, with is around EUR 60 …

3. Open-sources contributions
Skills & Interests
Cédric L. Charlier
@Seddryck
In a nutshell
Database architecture
Data modeling
Business intelligence
Data warehousing
ETL
Olap/Tabular
No SQL
Big data
Data analysis
Machine learning
seddryck.wordpress.com
Data & BI architect
Agile data warehouse
Quality-oriented
Automation fanboy
Open-sources contributions
Testing framework for BI
Other projects on github
Deployment SSRS – RsPackage
TmBundle for a few languages
Starting SSIS from .Net - Cassis
Toolset for special cases of modeling: ERMine et Tibre
And much more
Recognitions
Microsoft MVP Data Platform award
Experience
15 years with SQL Server
Former consultant, working internally for Elia

4. Agenda Why automating is a must? Tests with queries (Sql/Mdx/Dax)
Tests with ETL packages (SSIS)
Testing models (SSAS and DB engine)
Automating the creation of test-suites

5. What’s automated testing?
A piece of software written to reproduce steps/actions defined in a manual process

6. Benefits of automation
Speed up testing to accelerate releases
Allow testing to happen more frequently
Improve test coverage
Ensure consistency

7. Tests automation doesn’t apply to Business Intelligence

8. More than 33% of BI projects fail to deliver
Member of the Asscoiated Press . Aenean commodo ligula eget dolor. Aenean. Aenean commodo ligula eget dolor. Aenhswse. Cejhciebce fcdcdcd.
ILLUSTRATED WEEKLY NEWSPAPER
Est. 1869
Wednesday, November 24, 2012
Price 6d
More than 33% of BI projects fail to deliver
Lack of agility. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoqueCum sociis natoqueCum sociis natoque
Low quality of deliverables Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturientAenean commodo ligula eget dolor. Aenean massa. Cum sociis natoqueCum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec,
Simply Too late.
Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec,

9. We haven’t time to create these tests

10. Time to execute a test-suite
Exponential slope
Why?
New objects
Old objects

11. Time to write a test-suite
Initial gap

12. No existing framework for automating tests dedicated to BI
… I’m not proficient enough in C#

13. This framework exists: NBi
No C#, just xml syntax
This framework exists: NBi
Open-source

14. Agenda Why automating is a must? Tests with queries (Sql/Mdx/Dax)
Tests with ETL packages (SSIS)
Testing models (SSAS and DB engine)
Automate the creation of test-suites

15. Compare queries … result-sets
System-under-test
Assert
Cube and data warehouse
MDX
SQL
… or any brilliant idea
Previous release
Two different releases of a same data warehouse
Tabular model versus Xmlfile
DAX
Tabular model versus Csv file
DAX

16. Demo: compare queries’ result
Compare cube/operational database
101
Compare cube/flat files
Defaults and References
Tolerance
Intervals

17. Other testing goals for queries
Performance of a query
Executed in less than x seconds
Cache management
Syntax of a query
Extract queries from reports and assemblies

18. Query’s result and format
Result’s format
Example: all the cells have effectively 2 digits after the decimal separator and this separator is a comma.

19. Assert row count Assert how many rows are returned in the result-set.
Equal, more-than, less-than

20. Agenda Why automating is a must? Tests with queries (Sql/Mdx/Dax)
Tests with ETL packages (SSIS)
Testing models (SSAS and DB engine)
Automate the creation of test-suites

21. Inside this ETL insert Dimension Calendar Dimension Geography
Dimension Scientist

22. Demo SSIS Assert the execution status of an ETL
Alternatively test performances

23. Unique index violation
Inside this ETL
insert
Unique index violation
Dimension
Calendar
Dimension
Geography
Dimension Scientist

24. Setup and Cleanup Setup: Cleanup:
Define the state of your testing environment before executing a test
Cleanup:
Remove consequences of your tests on the testing environment

25. Tasks and behaviors Available tasks
Truncate/Load tables with csv files
Run sql batch, exe or batches
Run Etl packages
Move or delete files
Start/Stop Windows services
Run-once: execute only once this set of tasks
Parallel: run a set of tasks in parallel

26. Demo Setup Setup to register a clean state before test’s execution
Assert database’s state after the execution of an ETL

27. Remarks and limitations
ETLs must be stored
SSIS Catalog
File share
SQL Server
Supports SQL Server 2008R2, 2012, 2014 and 2016
Tests must be executed on the server running the ETL
No remote testing

28. Agenda Why automating is a must? Tests with queries (Sql/Mdx/Dax)
Tests with ETL packages (SSIS)
Testing models (SSAS and DB engine)
Automate the creation of test-suites

29. Testing dimension’s members
“Austria” is a member of dimension “Country”
“Austria” is not a member of the set “Nordic countries”
Dimension “Country” has more than 190 members.
All members of dimension “Country” have at least 4 characters
Members of hierarchy “Month” are ordered chronologically (and not alphabetically)
Members are equivalent to members of this other instance …

30. Example: members’ pattern

31. Testing structure (SSAS)
Assert that a dimension, hierarchy, measure, set, … is visible for end-user in a given perspective.
Assert the existence of a relation between a measure-group and a dimension.

32. Testing structure (DB engine)
Assert existence of a
schema,
Table
Column
Routine (stored procedure or function)
Assert the type of a column
Full type: decimal(10,3)
Light type: decimal

33. Agenda Why automating is a must? Tests with queries (Sql/Mdx/Dax)
Tests with ETL packages (SSIS)
Testing models (SSAS and DB engine)
Automate the creation of test-suites

34. Some tests are really repetitive
It should be possible to automate the creation

35. It exists! Take a look at Genbi
A set of test-cases
It exists! Take a look at Genbi
A template for the tests

36. Variables to be substituted (surrounded by $)
First steps with Genbi
File loaded from Genbi
Variables to be substituted (surrounded by $)

37. Demo: Genbi Load test-cases Select a template Generate the test-suite
From a flat file
Select a template
Generate the test-suite
Grouping option

38. I’m scripting macros to automate the actions performed by genbi …
With genbiL!

39. Sample for genbiL
Subject
Action
Parameters

40. Demo for genbiL
Command line

41. Conclusions Benefits of automated tests NBi Save Time (Time-on-market)
Better Quality (end-result)
More confidence
NBi
Supports large range of different tests
No need of compiler (neither C#)
Good feedback when some tests are failing
Open-Source
Automate the creation of your test-suites (genbi / genbiL)

42. Read further http://www.nbi.io/ Download releases Read documentation
Report bugs
Suggest new features

43. Please give us feedback!
How did you like it?
Please give us feedback!
to the event:
to me as a speaker:

44. Resources SQL Server 2016 in 15 Minuten SQL PASS Austria Homepage
SQL PASS Austria Homepage
SQL PASS Austria Meeting Archive

45. Thank You!
If you think, that a SQL Saturday is a nice possibility to learn from and network with fellow SQL Server enthusiasts FOR FREE,
I just ask you one thing: Visit the sponsor booths and chat with the sponsors!
They are covering the expenses for each and every of you, with is around EUR 60 …