1. Testing of XBRL files for the supervisory reporting
5 May2014
Katrin Heinze
Testing of XBRL files for the supervisory reporting
Eurofiling Workshop in Rome

2. Objectives The objective of this presentation is
to provide an overview on the testing strategy of the SUBA project
to explain the test plan and sample test cases
to inform about the next steps
Eurofiling Workshop in Rome, 5th of May 2014

3. Agenda Overview 1 SUBA* test strategy 2 Sample XBRL test cases 3
Next steps
* SUpervisory BAnking Data System
Eurofiling Workshop in Rome, 5th of May 2014

4. Agenda Overview 1 SUBA Test Strategy 2 Sample XBRL test cases 3
Next steps
Eurofiling Workshop in Rome, 5th of May 2014

5. Technical Test Execution Status
SUBA Test Strategy
Functional Tests
Non-Functional Tests
Regression Tests
Based on
User requirements
Use cases
Experience of users
Technical requirements
Source code (unit tests) …
Categorised following
Load tests (number of users)
Performance tests (processing time)
Volume test (quantity of data)
Stress test
Test of the (data) security
Usability tests …
Done after changes on
Components
XBRL taxonomy
Validation rules
Interfaces
EXDI
RIAD
Infrastructure …
Technical Test Execution Status
Eurofiling Workshop in Rome, 5th of May 2014
Page 5

6. HP ALM usage within the SUBA project
SUBA Test Strategy
HP ALM usage within the SUBA project
Scope
Planning of releases and test cycles
Enable traceability between requirements, tests, defects and according code changes
Increase visibility with dashboards on analysis and KPIs
Achievements
Definition of test cases linked to the defined requirements
Execution of test cases based on defined test data
Defect tracking and documentation of bug fixes
Way forward
Increase the level of automation on test execution
Customise reports and analysis based on the project needs
Preparation of test cases for the next release
Eurofiling Workshop in Rome, 5th of May 2014
Page 6

7. Test plan structure Conventions on numbering 1. level: Categories
2. level: Use cases
3. level: Test cases
Conventions on numbering
Structure in the Test Lab
Eurofiling Workshop in Rome, 5th of May 2014
Page 7

8. Tracking of the test progress in HP ALM
Eurofiling Workshop in Rome, 5th of May 2014
Page 8

9. Agenda Overview 1 SUBA Test Strategy 2 Sample XBRL test cases 3
Next steps
Eurofiling Workshop in Rome, 5th of May 2014

10. Positive functional test cases
Sample XBRL test cases
Positive functional test cases
Creation of test data for
each module of the COREP taxonomy
which is a valid XBRL instance document.
Valid according to
XML specification
XBRL 2.1 specification
XBRL Dimension 1.0 specification
XBRL Formula 1.0 specification
EBA disabled validation rules
EBA Filing Rules for COREP and FINREP
By using different file sizes
small
medium
large
Eurofiling Workshop in Rome, 5th of May 2014
Page 10

11. Negative functional test cases
Sample XBRL test cases
Negative functional test cases
Tests that ensure that the application shows a proper behaviour when invalid files are received.
Creation of test data where the system is expected to provide correct error messages.
Possible test cases : A reporting entity sends a file which
is not a well-formed XML document.
has not the correct format, i.e. jpg, txt, xls etc.
has not the expected encoding.
includes neither facts nor contexts.
misses a namespace in the xbrli root element
includes letters where numbers are expected as fact value
misses an explicit member in the scenario element
Includes a fact with no value but also no nil attribute is provided
Refers to a non-existing module
Refers to two different modules.
Eurofiling Workshop in Rome, 5th of May 2014
Page 11

12. Agenda Overview 1 SUBA Test Strategy 2 Sample XBRL test cases 3
Next steps
Eurofiling Workshop in Rome, 5th of May 2014

13. Voluntary functional tests with NCAs
Next steps
Voluntary functional tests with NCAs
Test scope
Test period: 13th to 15th of May
Test activities:
Acquisition
NCA sends data to EXDI via WebUI or standard adapter.
EXDI system sends a delivery message.
SUBA system sends an acknowledgement message to NCA with the XBRL validation result
NCA receives the messages in the scheduled time periods.
Eurofiling Workshop in Rome, 5th of May 2014
Page 13

14. Next steps
Load tests
First estimation about the load expected for the first remittance date
Number of reported entities
128
First remittance date NCAs to EBA/ECB
Modules reported
Number of files (per institute)
Max. file size in MB
Max. number of observations
COREP consolidated
Own Funds 1
115
Large Exposures
264
NSFR 9
7.510
Total
512
49.664
*) Estimations based on information of the Deutsche Bundesbank for one big German bank
IT infrastructure for establishing a supervisory function at the ECB

15. Organisation and approach – Test case preparation
Next steps
Organisation and approach – Test case preparation
Specification workshops
Specification sign-off
Implementation
Development of software
Definition of test cases (HP ALM)
Tests execution
Tests runs (HP ALM)
Defects workflow (HP ALM)
Bug fixing
Control point A
Control point B
Control point C
Control point
Metric
Target (by priority)
Average
High
Medium
Low A
% reviewed requirements
100% B
% requirements test coverage
75%
50% C
% tests run
90%
% tests succeeded
Defect resolution time (working days) 3 5
Defects reopened ratio 5%
10%
Eurofiling Workshop in Rome, 5th of May 2014
Page 15

16. Key issues and challenges
Next steps
Key issues and challenges
Functional tests
Supporting the voluntary functional tests with the NCAs, organized by WG SIM
Load tests
Defining scenarios for load tests on XBRL processing via automated test, simulating also peaks
Preparation for next releases
Defining test cases based on more precise user requirements
EBA testing
Participating in tests scheduled by EBA
Eurofiling Workshop in Rome, 5th of May 2014
Page 16