1. SDMX Tools Interactive demonstrations Structural Validation Service
ESTP Workshop on SDMX in National Accounts
Luxembourg, March 2016

2. Summary: Current ESA business process
National Production
Generate SDMX-ML
PUSH Transmission
Structural Validation
Content Validation
Processing & Calculation
Data Sharing & Dissemination
Data provider
Eurostat
Structural Validation

3. Possible future architecture (NAPS-S)
Shared / Replicated Services
For NA
and others
Structural Validation
Content Validation
Statistical Service A
Statistical Service B
SDMX Registry
VTL Repository
Common Repository
Common Repository

4. Structural Validation
Proof of concept setup
Regular production
All countries
EDAMIS
Selected TF members
Eurostat
Process Manager
Step 1:
Call Structual Validation Svc
If OK:
deliver to "reduced" production system
If Not OK:
deliver report to EDAMIS feedback channel
Structural Validation
Svc PoC

5. Demo: launching Struval after EDAMIS

6. ? Structural Validation SDMX Registry Content Validation
SDMX compliance
Valid SDMX-ML file
Coded according to the DSD
Mandatory fields present
Correct data types
Dataflow definition
Basic logical checks
Sender ID and REF_AREA
Table ID is present
Value "NaN" and OBS_STATUS
EMBARGO_DATE and CONF_STATUS
PRICES and REF_YEAR_PRICE
Basic content checks
Missing or unexpected series
Hole in series
Zero values
Negative values
General plausibility and consistency (within file)
Additivity of breakdowns
Outliers
Consistency between prices
Unadjusted and adjusted series
Advanced plausability and consistency (across files)
Revisions
Quarterly versus Annual
Same series across tables
Cross-domain or source checks
Balance of Payments
Trade statistics
Labour market statistics
Data pulished by NSI or IO
Structural Validation
SDMX Registry ?
Content Validation
VTL Repository

7. ? Structural Validation SDMX Registry Content Validation
SDMX compliance
Valid SDMX-ML file
Coded according to the DSD
Mandatory fields present
Correct data types
Dataflow definition
Basic logical checks
Sender ID and REF_AREA
Table ID is present
Value "NaN" and OBS_STATUS
EMBARGO_DATE and CONF_STATUS
PRICES and REF_YEAR_PRICE
Structural Validation
SDMX Registry
General plausibility and consistency (within file)
Additivity of breakdowns
Outliers
Consistency between prices
Unadjusted and adjusted series
Basic content checks
Missing or unexpected series
Hole in series
Zero values
Negative values ?
Content Validation
VTL Repository
Advanced plausability and consistency (across files)
Revisions
Quarterly versus Annual
Same series across tables
Cross-domain or source checks
Balance of Payments
Trade statistics
Labour market statistics
Data pulished by NSI or IO

8. Example of the STRUVAL error report
SDMX compliance
Valid SDMX-ML file
Coded according to the DSD
Mandatory fields present
Correct data types
Dataflow definition
Example of a mistake
Example of the STRUVAL error report

9. Example of the STRUVAL error report
SDMX compliance
Valid SDMX-ML file
Coded according to the DSD
Mandatory fields present
Correct data types
Dataflow definition
Example of a mistake
Example of the STRUVAL error report

10. Example of the STRUVAL error report
SDMX compliance
Valid SDMX-ML file
Coded according to the DSD
Mandatory fields present
Correct data types
Dataflow definition
Example of a mistake
? CONF_STATUS ?
Example of the STRUVAL error report

11. Example of the STRUVAL error report
SDMX compliance
Valid SDMX-ML file
Coded according to the DSD
Mandatory fields present
Correct data types
Dataflow definition
Example of a mistake
Example of the STRUVAL error report

12. SDMX Dataflow defintion
1 DSD covers multiple Dataflows
NA_MAIN DSD:
Table 1: T0101_A, T0102_A, T0103_A …
Table 3: …
Table 5: … …
Each of those Dataflows gives a partial picture
"cutting the DSD": SDMX cube region
"listing each series": SDMX key set

13. SDMX Cube Region Constraint
The DSD has an Activity code list with >170 items
In table 101 we only report A*10
SDMX cube region constraint (linked to dataflow):
List of allowed codes by concept ?

14. SDMX Key Set Constraint
Which series are allowed / forbidden?

15. Example of the STRUVAL error report
SDMX compliance
Valid SDMX-ML file
Coded according to the DSD
Mandatory fields present
Correct data types
Dataflow definition (Cube)
Example of a mistake
Example of the STRUVAL error report

16. STO=YA1 is allowed in T0101 (statistical discrepancy)
SDMX compliance
Valid SDMX-ML file
Coded according to the DSD
Mandatory fields present
Correct data types
Dataflow definition (Key)
Example of a mistake
STO=YA1 is allowed in T0101 (statistical discrepancy)
ACTIVITY=L is allowed in T0101
But the combination is not:
Statistical disrepancy is not broken down by activity
Other example: expenditure _Z or _T depends on STO

17. Example of the STRUVAL error report
SDMX compliance
Valid SDMX-ML file
Coded according to the DSD
Mandatory fields present
Correct data types
Dataflow definition (Key)
Example of a mistake
Example of the STRUVAL error report
Not yet activated 
Data flow defintions need to be extended and tested
Planned to be added during Feb / Mar

18. Planned improvements Faster turnaround time (currently ~10 minutes)
Change presentation of error node from list to XML tags
Grouping of errors by error type instead of once per occurance

19. Presentation of error node
<Series FREQ="Q" ADJUSTMENT="N" REF_AREA="LU" COUNTERPART_AREA="W2" REF_SECTOR="S1" COUNTERPART_SECTOR="S1" ACCOUNTING_ENTRY="B" STO="YA1" INSTR_ASSET="_Z" ACTIVITY="L" EXPENDITURE="_Z" UNIT_MEASURE="XDC" PRICES="V" TRANSFORMATION="N" />

20. Grouping of errors by type
Now
Wrong code B5N in series X.X.X
Wrong code B5N in series X.X.Y
Wrong date format in series X.X.Z
Wrong code B5N in series X.Y.Z
Then
Wrong code B5N in X.X.X X.X.Y X.Y.Z
Wrong date format in X.X.Z

21. NAMAIN DSD Scope for phase 1 GDP NASEC DSD
Main Aggregates: 0101, 0102, 0103, 0120, 0121
Sector Accounts: 0800, 0801
0800
NASEC DSD
0801
GDP
0101
0102
0103
0120
NAMAIN DSD
0121

22. Timeline for structural validation
Pilot of structural validation until end March
Improvements following suggestions from VALTF
Presentation of results to NAWG
Staged go-live throughout 2016