1. Data validation at DESTATIS
ESTP course on Data Validation
Item 12 – German validation system

2. Federal Production Validation Data Collection
Data Processing and Analysis
Publication
© Federal Statistical Office of Germany (Destatis) | Department C3

3. Common data formats DatML/RAW DatML/SDF XML DatML/ASK DatML/EDT
Micro data
DatML/RAW
XML
Data model
Range checks
DatML/SDF
PL-Spezifikationssprache
(Validation specification language)
Web form
DatML/ASK
DatML/EDT
Hierarchical data structures
Validation/editing rules
Validation procedures
© Federal Statistical Office of Germany (Destatis) | Department C3

4. Common software portfolio
II + III IV V VI
VII
Run exclusively at DESTATIS
Survey editor
IDEV
Web forms
.CORE
Web services
Data Validation and Editing Runtime
Statspez
Validation rules editor
Genesis
Open data portal
Input database
Domain applications
SAS
Web forms editor
Micro data
Validated micro data
Metadata
Macro data
Survey database
Metadata DB
© Federal Statistical Office of Germany (Destatis) | Department C3

5. Overview of metadata system
II + III
Define data model for survey
Validation on data items (e.g. range checks)
Survey editor
Define data model for validation
Write validation and automatic editing rules
All kinds of data validation
Arrange data validation procedures
Validation editor
Design web forms
Determine when data validation procedures are executed
Web forms editor
Store and provide resources where needed
Survey database
Metadata DB
© Federal Statistical Office of Germany (Destatis) | Department C3

6. Overview of the data collection system
II + III IV
Web form based data collection
Validation on data items
Validation within data set
Some validation against reference material
Survey editor
IDEV
Web forms
.CORE
Web services
Interpret rules in PL- Spezifikationssprache
Validation rules editor
Web service based data collection
Otherwise just structural validation
Input database
Web forms editor
Survey database
Metadata DB
© Federal Statistical Office of Germany (Destatis) | Department C3

7. Overview of the data collection system
II + III IV
Survey editor
IDEV
Web forms
.CORE
Web services
Validation rules editor
Stores incoming DatML/RAW files
Transports files to …
… the data owner
… the central production office
Input database
Web forms editor
Survey database
Metadata DB
© Federal Statistical Office of Germany (Destatis) | Department C3

8. Overview of the processing system
II + III IV V
Translate rules to JAVA at runtime (OVIS)
Survey editor
IDEV
Web forms
.CORE
Web services
Data Validation and Editing Runtime
Validation rules editor
All kinds of data validation
Input database
Domain applications
Web forms editor
Use validation procedures as JAVA classes
Survey database
Metadata DB
© Federal Statistical Office of Germany (Destatis) | Department C3

9. Output of the validation procedure
Micro data
Validated micro data
Validation DB
Editing
One tool
Error list
Long validation report
Short validation report
© Federal Statistical Office of Germany (Destatis) | Department C3

10. Micro, macro, …? II + III IV V VI VII .CORE Validation rules editor
Web forms editor
Survey editor
Survey database
Metadata DB
Statspez
SAS
Data Validation and Editing Runtime
Domain applications
EDIT
Other software
Other
Print or digital
Data recipients Parliament, etc.
IDEV
Web forms
.CORE
Web services
Input database
Data Validation and Editing Runtime
Domain applications
Statspez
SAS
Genesis
Open data portal
© Federal Statistical Office of Germany (Destatis) | Department C3

11. We’re working on it! A complete system? Common data formats
Common software portfolio covering the complete production process
Common data validation language covering the complete production process
Standardized validation reports
Standardized review process
We’re working on it!
(up to 5.4)
(mostly)
© Federal Statistical Office of Germany (Destatis) | Department C3

12. General process Decide Produce Evaluate process, decide on changes
Design rules
Production process
Methodology dept. advises
Review and accept rules
© Federal Statistical Office of Germany (Destatis) | Department C3

13. Centralization in a federal system?
Länder
DESTATIS
Not centralized
Centralized
Approval of validation rules
Structural validation
Content validation (4.3)
Content validation (5.3)
Data editing (5.4)
Content validation (6.2)
Data editing (6.2)
Review of production process
Validation rule design
Content validation (5.3)
Data editing (5.4)
Content validation (6.2)
Data editing (6.2)
Review of production process
Validation methodology (in an advisory role)
Metadata distribution
Structural validation
Content validation (4.3)
Content validation (5.3)
Data editing (5.4)
Quality reports
© Federal Statistical Office of Germany (Destatis) | Department C3

14. Questions?
© Federal Statistical Office of Germany (Destatis) | Department C3