1. Thorsten TÜMMLER Federal Statistical Office of Germany
Peer Review from a national perspective – preparation, procedure, consequences
Thorsten TÜMMLER
Federal Statistical Office of Germany
As you all know, we are in the middle of the 2nd round of Peer Reviews.
The first phase, the self-assessment is finished.
My speech covers the process of answering the SAQ and it is divided in three parts: Preparation for the SAQ, the procedure of answering the SAQ and consequences from this first phase of the Peer Peview.

2. The German statistical system
Federal structure of Germany Independence of the state statistical offices Division of tasks between federal and state level
In order for you to understand why we did things the way we did, it is essential to know a few key facts about the German statistical system.
Germany is a federal republic. That means that Germany consists of 16 federal states (Länder). From the basic law follows that the Länder are responsible for executing federal laws – including those on statistics. So each federal state has a statistical office of its own that is independent from the FSO. Those statistical offices vary greatly in size as well as organizational and legal form. The statistical offices of the Federation and the Länder cooperate closely and form the so called “System of Official Statistics”. Within this system exists a variety of boards and bodies as well as regulations and agreements to support cooperation.
As a consequence of this, almost all European Statistics are a common product of the statistical offices of the Federation and the Länder, where the FSO generally is responsible for methodology and coordination while the SOL do the actual field work and data processing. They then send aggregated data on Länder-level to the FSO which compiles results for Germany.
So the German statistical system is quite similar to the ESS.

3. Preparation for the Peer Review (I)
Review of the impact of the Quality Assurance Framework (QAF)
Goal
Identify weaknesses
Determine actions for improvement
Results
Inventory of activities/methods/tools
14 measures for improvement
The preparation for the Peer Review started shortly after the ESSC adopted the QAF in 9/2011: Since the QAF would be a major instrument for the next round of Peer Reviews, the heads of the statistical offices of the Federation and the Länder decided in 5/2012 to review the impact of the QAF on the System of Official Statistics.
The goal was to identify weaknesses in the implementation of the QAF (and therefore the CoP) and to develop relevant actions for improvement even in advance of the Peer Review.
This analysis was finished im 9/2013 and resulted in an inventory of activities/methods/tools of the QAF that are already in use in Germany as well as 14 measures of improvement to adress the identified weaknesses.

4. Self-Assessment Questionnaire (I)
Involvement
Federal Statistical Office as NSI
14 state statistical offices as 1 ONA
1 common self-assessment questionnaire
2 other ONAs
Challenge to harmonize answers
For the SAQ of the 14 state statistical offices
Between the SAQs of the Federal Statistical Office and of the state statistical offices
On December 20 the actual Peer Review started with the receipt of the SAQ. As you know, the whole National Statistical System is subject to the Peer Review. However, only the NSI and 3 ONAs are to be involved in order to limit burden. For Germany the limitation of ONAs posed the problem, that the System of Official Statistics could not be adequately described if the majority of the SOL are not involved. So the Conference of Heads of the Statistical Offices of the Federation and the Länder decided, that all SOL are to be treated as 1 single ONA and that they complete 1 common SAQ.
The other 2 ONAs involved in the Peer Review are the Federal Employment Agency and the Federal Office for Motor-Traffic.
The main challenge was to produce harmonized answers:
-Firstly, for the SAQ of the 14 SOL. These answers should reflecting their similarities but also their differences and all 14 SOL must agree to that answer.
-Secondly, answers needed to be harmonized with the SAQ to be filled in by the FSO for all questions that relate to the System of Official Statistics, where common rules and agreements ensure unified procedures so that all statistical offices ought to give identical answers.
As a result, both SAQs, the one for the SOL and the one for the FSO are identical for around 2/3 of answers.
This task was fulfilled by April 30.

5. Consequences from the Peer Review (I)
14 Measures to improve compliance with CoP
Negotiations with Federal Ministry for Economics
First publication of the production index and index of new orders by Federal Statistical Office
The preparation for the Peer Review and the completing of the SAQ already brought about some improvements concerning the compliance with the CoP.
In the preparation phase 14 measures for improvement were proposed, 6 of which were integrated in the working schedule of the WG Quality. The remaining 8 will be assigned to other units for further processing.
The upcoming Peer Review also helped the Federal Statistical Office with the first publication of the production index and the index of new orders. In the past, these have been published by the Federal Ministry of Economics, which was a clear breach of the rules of the CoP and which was a major weakness in the first Peer Review.

6. Consequences from the Peer Review (II)
Promote culture of quality
Raised awareness for aspects of quality management
Exchange of information within and between statistical offices
Comparison of different practices
Adoption of best practices
The third improvement from the Peer Review is the promotion of a culture of quality.
The extensive work with the CoP, the QAF and the SAQ raised the awareness for various aspects of quality and quality management that hitherto were largely neglected.
All the open discussions within institutions and between institutions, brought about a lot of exchange of information and increased transparency. It enabled easy comparison of different practices and in stimulated a series of small improvements in the SLO by adopting best practices:
-Example 1: The SO BBB now publishes its vision on the homepage
-Example 2: several SOL reconsider their publication procedures and plan to publish a release calendar as well as publish the information on pre-release access.
-Expamle3: several SOL plan to adopt the guideline on how to deal with publication errors from the FSO

7. Consequences from the Peer Review (III)
Suggestions for refining the QAF
Rephrase some actions/methods/tools
Expand to include principles 1-3 and 5, 6
Go beyond CoP and include “co-ordination”
Suggestions for future Peer Reviews
Reduce costs of Peer Review
Estimated 100 working months full time equivalent
Leave more time to complete SAQ
Possible solution: Drop questions on product level
On European level, the work so far brought some suggestions for refining the QAF:
-Firstly the QAF should be revised in order to increase understandability and reduce the need for interpretation
-Secondly, the QAF should be expanded in order to also include the principles dealing with the institutional environment (1-3, 5, 6).
-Thirdly, the aspect of „co-ordination“ is an important issue in the Peer Review; there even is a separate questionnaire dealing with co-ordination. But it is not reflected in the CoP and as consequence it is also missing in the QAF. Since co-ordination will become more and more important, the demand to reflect that significance in the CoP and the QAF will arise.
Cost and burden of the Peer Review already were an issue for this round, but for the future it absolutely requires some more attention. We estimate that we invested about 100 working months full time equivalent into the preparation and the completing of the SAQ, 30 of those in the FSO alone. Of course, we are aware that the German approach of effectively involving 17 institutions in the Peer Review is especially costly.
Also, the originally planned time schedule for completing the SAQ within 3 months would have been impossible even though we were well prepared and conducted a stock-taking exercise for preparation. We absolutely needed the 4 ½ months until end of April.
Both issues could be addressed by shortening the SAQ. A starting point for this could be to concentrate on questions on institutional level and to drop the questions on product level. Answering these was not only difficult, but many of those answers are insubstancial because it effectively it was an answer on institutional level where actually an answer on product level would have been required. But that could have been different for every single statistic. So my proposal for future rounds of Peer Reviews: Drop the questions on product level from the SAQ.

8. Thank you for your attention!
Thorsten TÜMMLER