A handbook on validation methodology. Metrics. Nadežda Fursova, Jūratė Petrauskienė 9–11 November 2015, Wiesbaden Workshop ValiDat Foundation

Metrics. General insight Metrics for validation - indicators to measure the quality of a validation procedure (tuning, evaluating validation procedure...) to achieve efficient validation process. A set of well defined metrics can be used to ensure the adequacy of the validation process to the objective quality gains in final data and the validation procedure itself. Data validation is an important part of statistical production and data exchange, therefore the data validation rules important object of study. The objectives of the good design of a set of validation rules is to achieve a satisfactory quality that would permit the statistical officers to have a reasonable confidence that the outcome of the validation process is free of important errors and that the cost of that process is satisfying some requirements of efficiency and completeness. 9–11 November 2015, Wiesbaden Workshop ValiDat Foundation

Evaluation of validation rules. Efficiency, non-redundancy, completeness, feasibility and complexity are the main factors for validation quality. Efficiency: rules flagging a maximum of “true” error Non-redundancy: no double check by different rules Completeness : all checks are satisfied Feasibility: no contradictory rules in a rule set Complexity: variety of information necessary to compute a validation rule is regarded and involved 9–11 November 2015, Wiesbaden Workshop ValiDat Foundation

Evaluation of validation rules. Efficiency. Efficient validation rules - detect most true errors and miss few of them. Removing those that do not detect many true errors made rules more efficient. An efficient set of validation rules favors automated validation and limits human actions. 9–11 November 2015, Wiesbaden Workshop ValiDat Foundation

Evaluation of validation rules. Non-redundancy. Non-redundancy or independence. The validation rules must be as independent as possible one from each other. e.g. if “Total” is computed as male + female, the validation rule male+ female = total is redundant There are two reasons to remove redundancies from a set of rules: First, redundancy removal yields a more compact and elegant rule set that expresses the necessary restrictions in some sense minimally. Secondly, the time and/or memory consumption of some algorithms that make use of rule sets can depend strongly on the number of rules provided. 9–11 November 2015, Wiesbaden Workshop ValiDat Foundation

Evaluation of validation rules. Completeness. Completeness or sufficiency. Completeness refers to a limited interpretation as sufficiency. A rule is assumed to be sufficient when the values that not rejected are plausible. A sufficient set of validation rules can be established: By analyzing relations amongst the dataset variables; Ensure that for each variable there is a rule checking it; Introduce at least one validation rule referring to an external value known as valid. Two obvious completeness-related problems may occur: incompleteness overcompleteness 9–11 November 2015, Wiesbaden Workshop ValiDat Foundation

Evaluation of validation rules. Feasibility. A rule set is called feasible, or consistent, when the acceptance range defined by a rule set is nonempty. Infeasibility occurs for instance when a rule set contains a rule that is contradictory in itself, for example the rule that states x > x is clearly contradictory. Or consider the following rule set has rules that are perfectly feasible by themselves but their combination is contradictory: {x > 1, x < 0} Clearly, there is no number that can satisfy both rules. In practice, rule sets can be much more complicated than this and contradictions rarely present themselves in such a clear form. 9–11 November 2015, Wiesbaden Workshop ValiDat Foundation

Evaluation of validation rules. Complexity. There is no single definition of the complexity of a set of validation rules. Complexity is related to the variety of information that is necessary to evaluate a validation rule. In the most simple case, a rule can be evaluated by comparing a single data value with a fixed range of values. A minimal set of four sources of information are identified that characterize complexity: the domain/ type of statistical objects, the time of measurement, the actual unit on which a measurement was performed, the variable that was measured. The levels of complexity correspond to the number of sources of information that must be varied in order to compute the validation function. 9–11 November 2015, Wiesbaden Workshop ValiDat Foundation

Evaluation of validation rules. Designing and maintaining a set of validation rules is a dynamic learning validation process as it can be improved through the experiences drawn from the checking of successive data vintages. Evaluation of the existing validation rules should be periodically performed on the basis of the validation results (flag and hit rates and any external feedback on presence of errors that have survived the validation process). The following actions are performed: Less efficient rules are replaced More efficient rules are incorporated to detect systematic problems New rules may be added to detect errors that escaped from previous checks 9–11 November 2015, Wiesbaden Workshop ValiDat Foundation

Indicators Indicators measure the efficacy and efficiency of a data validation procedure. The indicators may refer either to a single rule or to the whole data validation procedure. Indicators may be distinguished in : Indicators taking into account only validation rules Indicators taking into account only observed data Indicators taking into account both observed and plausible data (imputed data). The first ones are generally used to fine tune the data validation procedure. The second ones are used to obtain a more precise measure of the effectiveness of a data validation procedure, but are dependent on the method chosen to obtain amended plausible data. The method is composed of a localization error procedure and an imputation procedure. 9–11 November 2015, Wiesbaden Workshop ValiDat Foundation

Indicators taking into account only observed data The indicators belonging to this group exploit only information related to the application of rules to data. They can be calculated overall or per variable/observation. For example: Number of failed records; Minimum number of variables to be changed in order to make records pass the given set of rules (cardinality of solution); Counts of records that passed, missed and failed for each rule; Distribution of records that passed missed and failed k rules, etc. Good examples of indicators developed for measuring the impact of validation rules on observed data can be found in BANFF, a processing system for editing and imputation developed at Statistics Canada (presented in the handbook). 9–11 November 2015, Wiesbaden Workshop ValiDat Foundation

Indicators taking into account both observed and plausible data (imputed data) While the assessment of the impact of a data validation procedure requires only the availability of the observed data, measures of efficiency and efficacy should in principle involve comparison between “true” data and observed data. Since true data are usually not available (otherwise no validation procedure would be necessary), some set of reference data have to be used in place of the true data. True values as reference data A possible approach is to accurately revise a dataset of the observed data and to use the resulting cleaned dataset as reference dataset. 9–11 November 2015, Wiesbaden Workshop ValiDat Foundation

Indicators taking into account both observed and plausible data (imputed data) Plausible data as reference data When true data are not available, even for a small subsample of the data, a common approach to analyse a validation procedure is applying indicators to “plausible” data in place of true data. Simulation approach Simulation is another strategy for the assessment of a validation procedure based on comparing observed data with reference data. In practice, one tries to artificially reproduce a situation which is likely to be similar in terms of true data distribution and error mechanism. Specifically, there are two elements for the simulation approach: a set of true reference data some procedure for perturbing data by introducing errors. 9–11 November 2015, Wiesbaden Workshop ValiDat Foundation

Workshop ValiDat Foundation 9–11 November 2015, Wiesbaden Workshop ValiDat Foundation

Open problems hard to measure and compare true data and observed data, since true data are usually not available difficult to perform validation of administrative data important, not to overedit data, to keep the balance validation of inter-record rules deserves further studies 9–11 November 2015, Wiesbaden Workshop ValiDat Foundation

THANK YOU FOR YOUR ATTENTION! 9–11 November 2015, Wiesbaden Workshop ValiDat Foundation