Quality Reporting with JD+ Where are we? Where are we going? SAEG – June 7th, 2016 SACE

The Current JD+ Quality Report: Principle n numerical quality indicators. Each of them is translated into a qualitative indicator. This set of qualitative indicators is then summarized in a qualitative global indicator. Use of colors: green, orange, red, severe Palate Jean (2014), JD+ Quality Indicators, NBB SACE

Current JD+ Quality Report SACE

JD+ QR: Selected Indicators Diagnostics on: The coherence of the decomposition ("Basic checks" group), Visual spectral inspection ("Visual spectral analysis"), The residuals of the RegArima pre-processing ("RegArima residuals" group), The residual seasonality ("Residual seasonality" group) The decomposition ("Seats" group for Tramo-Seats, M-Statistics group for X12). Most of them use parameters (usually thresholds) that can be modified Each group of diagnostics might be disabled, when it is considered as meaningless. SACE

Pros and Cons Pros Cons Very simple to read Very useful for the producer Cons Could be misunderstood (Warnings vs. Quality) Tramo/Seats “biased” Too many indicators (7 different seasonality tests) But output csv matrix incomplete SACE 5

The “Quality Report Plug-In Project” 2 very different ideas/objectives Technical side: Disseminating the knowledge on JD+ to make people able to contribute Statistical side: Developing a customized Quality Report The challenges Getting basic programming skills What is a Quality Report? Selecting and weighting statistical indicators SACE 6

Quality Report: A Quite Fuzzy Concept “Quality report” are words widely used in the SA context but they cover various meanings and concepts. When is it used, by whom, and for which purposes? Producing, Disseminating, Using (the series) SACE

Producing the series To define the process To run the process choosing the best strategy/model To run the process using the old/previous parameters To check/improve the process Annual campaign, production time To prepare the series for publication & storage SACE

Disseminating and storing the series Various publications Informing on overall quality and revisions etc. Various databases, various metadata Producer databases Institute databases Other databases Often a need for other quality diagnostics (timeliness for example) SACE

Using the series “Basic” user Researcher/Econometrician Direct use of the series (indexing contract) Use of the series to comment the evolution of the sector/Economy Usually “trust the series”; basic QR Researcher/Econometrician Focus on very specific criteria: timeliness, revisions, noise, dynamics of the series (model) etc. Use the QR ….. or not: redo its own adjustment SACE

Various objectives → Various QR? Yes Metadata ≠ Publication ≠ Producer ≠ Researcher …. and No Almost every information we can think about can be output by JD+. Metadata and Publication: “just a reporting problem” Some problems: JD+ cannot contain all necessary information (timeliness, real time revisions, classifications etc.) Quality of indirect aggregates (chain-linked or not) SACE

What can do JD+? Lots of things ….. But not everything The output matrix could be extended / customized The signal alerts (red lights) can be adapted/tuned Reports can be automatically created using standard formats (SDMX) But not everything It might consume resources (computing time, space) if too many indicators are required. JD+ does not store all the information useful for a QR JD+ does not handle “direct vs. indirect” aggregates. SACE

A Generic Structure A set of descriptive data (ex: method, model, parameters, length etc.) Weight of the descriptive data A set of quality statistics (ex: M1-M11 statistics in X-12) Weight of the statistics and quality statistics Weight of the series SACE

The “descriptive data” Typically information necessary for the metadata “Clarity” and “transparency” aspects of quality Information on the series (name, length etc.), on the process (method, regressors etc.) and on the result of the process (model, # of outliers, TD effect etc.) Could be also used to rerun the process Weight Typically, each item is present (1) or not present (0). Allows summarizing the overall quality (presence) of the “metadata” SACE

The “quality statistics” Typically information necessary to check the efficiency of the process for a series “accuracy” aspect of quality No residual seasonality, TD effect, M1-M11 statistics, JD+ set of quality statistics Also used to improve the seasonal adjustment Weight To derive a global quality index: Q-statistics, JD+ alerts. SACE

Weight of the series Important in production time Examples: weight of the sector in the NACE, contribution of the sector to the growth of the aggregate, weight of the country in the UE etc. Permits a selective editing: if something is going wrong during production, you focus on the most important series SACE

The real problems to solve The choice of the information to output Revisions: the last value of the series can be kept Some metadata cannot be included (see example) Too many indicators (7 seasonality tests) The choice of the weights Simple (?) for metadata (1/0) More complex for quality statistics How to use the QR in production time? SACE

A First Plug-In for Eurostat Unit F3 To report on the SA of Job Vacancy Survey series First specifications during a Hackathon session organized by the SAUG (08/12/2015) Several iterations to agree on the quality indicators A “final product” delivered end of 02/2016. Used in production by Member States SACE 18

The “command” SACE 19

The “final” version SACE 20

JD+ Quality Report Joint work with Istat and NBB First steps: Learning to develop simple plug-ins Output a very complete csv matrix for both TS & X12 First statistical studies using this basic plug-in to select relevant quality indicators Checking the probable redundancy of seasonality tests SACE 21