1. Working Group on Quality Reviews of Energy Data
Kevin Roberts, WG Chair
10th Meeting of the
Oslo Group on Energy Statistics
May 2016
Welcome to this session on energy data quality
Introductions: Kevin Roberts, Chair of the Working Group (WG)
Topic: Quality
An ongoing challenge of all statistical programs: finding ways to ensure the quality of data being collected and disseminated
There are many dimensions of quality: completeness, timeliness, accuracy, objectivity, integrity, comparability, access, etc.
Our WG was given the responsibility to find practical ways to help energy programs to assure the quality of their data, with the broader goal being to promote and support the collection of high quality energy data at the international level
Thanks to WG members for their ongoing support and contributions to this project
I look forward to some interesting and productive discussions today, leading to some concrete plans for the future work of this group

2. Overview of Today’s Session
Review the efforts of the WG since the 10th OG meeting
Present the WG recommendations for the next projects to be conducted
Reach agreement on the topics to be addressed
Get commitments from members to contribute to the upcoming work
To start, I would like to provide you with an overview of my presentation, and of the discussions that we will be having during this session
I will begin with a brief summary of the efforts of the WG since the last meeting of the Oslo Group in June 2014
Next, I will present the recommendations of the WG in terms of the proposed work going forward
After that, we will discuss those recommendations and (hopefully) come up with an agreement on a workplan for the future efforts on this topic
Finally, I hope that we can get some commitments from you to provide support and inputs into the work that will be done in the months to come
To be successful, we will need to have contributions from many countries and organizations – to share the workload, to gather a broad and useful set of information and advice
10th Meeting of the Oslo Group on Energy Statistics

3. Summary of Activities of the WG
Established at the 9th Meeting of the OG (June 2014)
Prepared a mandate for the WG (fall 2014)
Solicited suggestions on topics, approaches
Developed 6 potential projects for consideration
Gathered feedback on these potential projects
Today: Proposing a workplan for moving forward
As you will recall, this WG on Quality was established at the last meeting of the OG in Abu Dhabi in June I agreed to take on the responsibility of Chair for this group.
We drafted a mandate for the WG, which was sent out in the fall of 2014 for review and comment. Based on the good input we received from members, the mandate was then finalized by December 2014.
In 2015, Members were then asked to suggest possible topics for the WG to address. These would represent practical projects to help and support energy statistics programs to ensure the quality of their data.
In the end, 6 potential projects were identified. These proposals were written up and sent out in an from me in February 2016 to WG members to gather their feedback. Members were asked for their views and preferences. Thanks again to all of you who were able to provide your comments.
We gathered up this feedback and today, I will be telling you what we heard. I will list the 6 proposals and then identify the ones that we are proposing to start with. That would represent our proposed workplan between now and the next OG meeting.
10th Meeting of the Oslo Group on Energy Statistics

4. The 6 Projects Under Consideration
Promote the sharing of country practices on ensuring data quality
Serve as a clearinghouse for information on data quality tools and practices
Prepare guidelines for conducting periodic quality reviews
Collaborate with existing energy data quality initiatives
Develop a manual on energy data quality
Support regional efforts to improve quality
Here are the 6 projects that were put forward for consideration.
I will provide a little more about each project in a minute
10th Meeting of the Oslo Group on Energy Statistics

5. Feedback from the Consultations
Key messages:
Start with priority topics
Select options that are practical, doable, helpful
Avoid duplication of efforts with other organizations
Build on the good work already being done
Start with the following 3 projects
As I mentioned earlier, I sent out a note asking for feedback on these 6 project proposals. When we got our feedback from members, there were a number of over-arching messages that came out.
There was general support for these proposals as good ideas that would help to address quality issues. But at the same time:
It was suggested that we focus on a few, priority projects. Do not try to do them all. Do not over-extend. Go for the low-hanging fruit where can demonstrate quick and useful progress.
Pick topics that are practical, constructive and doable. We should strive for outputs that would be helpful to energy statistics programs.
We should avoid duplication of efforts with other organizations. The projects we undertake should not recreate or contradict the work already being done by bodies like, for example, the IEA, Eurostat, the UNSD.
Rather, we should build on, or supplement, the good work that is already being done out there. In other words, create value added outputs.
This is good advice. Based on that input, and on the comments from members about the proposals, we are recommending that the following 3 projects be undertaken by the WG.
10th Meeting of the Oslo Group on Energy Statistics

6. 1. Promote the Sharing of Country Practices on Energy Data Quality
Objective: to assemble and disseminate examples of good country practices for ensuring quality energy data
Strategies:
Seek out more good practices
Organize them by theme
Actively disseminate them to countries & organizations
The first project being proposed: sharing country practices on data quality
Goal: to assemble & disseminate examples of good country practices for ensuring quality energy data
Currently, the Oslo Group already shares country practices that have been submitted on a broad range of topics related to energy statistics. These are stored and available on the OG site. Several of these discuss dimensions of quality. But we need to do more.
Under this proposal, the WG would adopt a proactive approach whereby good quality practices would be sought out, written up, made available on the OG site, and organized by particular theme.
Practices could be categorized, for example, by fuel type, by data collection strategy (survey vs admin data), by stage of collection process (questionnaire design, follow-up practices, data validation strategies).
Examples could also be organized according to whether the energy statistics program is a new and developing one, or a mature and advanced one. As we work on this, we will learn about the best ways to do this.
When enough practices have been assembled by theme, the working group could develop dissemination strategies for targeted audiences in need of that type of information. Examples of possible dissemination strategies could include: compendiums of practices by theme; workshops; or sets of recommendations.
10th Meeting of the Oslo Group on Energy Statistics

7. 2. Serve as a Clearinghouse for Information on Energy Data Quality
Objective: to improve access to information on energy data quality from a broad range of sources by enhancing the OG site and its content to serve as a clearinghouse
Strategies:
Identify links to other sites and sources
Establish partnerships with other organizations to share information
Broaden the OG site to house this new function
The 2nd proposed project…
Under this proposal, the Oslo Group web site could be enhanced to serve a clearinghouse function. The objective would be to improve access to already existing information on energy data quality from a broad range of sources – countries, international organizations, academic institutions, etc.
How would we do this?
A section on the site could be dedicated to storing documents from, or providing links to, other sites in countries or organizations around the world with valuable information and tools related to improving the quality of energy data.
This clearinghouse would serve to publicize and provide access to these helpful sources, thereby building on the good work already done by others.
At the same time, we would negotiate with those groups to provide on their sites links to the OG site – thereby providing two-way links. This would improve the visibility, profile and awareness of the OG site (to broaden its users and usefulness)
The efforts of the working group would therefore focus on organizing the Oslo site to serve this function, identifying useful sites in countries and organizations, negotiating permissions to provide links, and advertising this new clearinghouse role.
This would mean broadening the awareness of the OG site beyond just its members. To be a useful clearinghouse, the site would have to be well-known, easy-to-find and easy-to-use.
10th Meeting of the Oslo Group on Energy Statistics

8. 10th Meeting of the Oslo Group on Energy Statistics
3. Prepare Guidelines for Conducting Quality Reviews of Energy Programs
Objectives: to develop and share guidelines for conducting periodic quality reviews
Strategies:
Gather examples from countries that do this
Create a generic set of guidelines that can be shared
Disseminate these guidelines
The 3rd project that the WG is proposing is the preparation of a set of guidelines for conducting periodic quality reviews.
Some statistical programs have implemented a policy of conducting periodic, comprehensive reviews of particular surveys for the purpose of evaluating and improving data quality.
These reviews can be very useful for identifying risks, problems or opportunities for improvements.
Sometimes statistical programs do not have the chance to conduct self-assessments on an ongoing basis due to workload, costs and other constraints. As an alternative, these types of periodic quality reviews (e.g. once every five years) can allow for a regular, structured evaluation.
Under this proposal, the working group would assemble information on periodic quality review processes that various countries have adopted. These country approaches could be compared for the purpose of designing a specific process for energy surveys.
The recommended approach could then be shared broadly, for other programs to consider and implement.
For those unfamiliar with this concept, STC does this for all major surveys, including energy surveys, every 5 years
Senior staff are brought in from other programs to conduct an objective, independent review of processes, tools, outputs to identify risks or opportunities for improvement
This can lead to changes, investments, improvements, sharing of good practices
So, for example, these types of review processes can be shared between countries. And a generic model for energy surveys could be produced.
10th Meeting of the Oslo Group on Energy Statistics

9. Projects not to be launched at this time
4. Collaborate with existing quality initiatives 5. Develop a manual on energy data quality 6. Support regional efforts to improve quality
So, those were the 3 projects being recommended by the WG for immediate attention.
The other 3 proposals are all good ideas too. We may want to consider them again in the future. But the current recommendation is to focus on 3 projects only.
A few words on the 3 projects not being recommended at this time:
Collaborate with existing energy data quality initiatives. As mentioned earlier, there are other organizations that are involved in delivering support and services relating to energy data quality (e.g. IEA, UNSD). Under this proposal, the working group would seek to identify opportunities to build on those valuable activities through collaboration. In our view, this idea may have merit. But at this point, a lot of research and consultation would have to be done to identify possibilities. Too vague and not practical enough, at this time. Later in this session today, we will have presentations from 4 organizations about the work they are already doing on enhancing the quality of energy data. This may give us some ideas about how we as a group might be able to collaborate with or support them in the future.
Develop a manual on energy data quality. The objective of this proposal would be to produce an energy statistics manual that would focus on providing practical, detailed support to statistical programs in the collection and dissemination of quality energy data. The work could be organized and delivered in separate components, allowing products to be disseminated in modules, as each one is completed.
Different approaches could be taken when organizing such a manual.
One approach would be to base the manual on one of the existing, generic Quality Assurance Frameworks that have been developed by various organizations. The manual could organized according to the components of the framework (e.g. timeliness, relevance, comparability), but prepared specifically for energy statistics programs. The working group could focus on the highest priority components first, and release the chapters incrementally when ready.
Another approach would be to follow the Generic Statistical Business Process Model (GSBPM) that was used in the Energy Statistics Compilers Manual, but again, to tailor the content to be energy specific. Modules could be produced and disseminated independently, when ready, and could prioritized in a number of ways – for example, by each stage of the process, by fuel type, etc.
In the end, we believe that this would be an extremely ambitious project that could require ongoing efforts over a substantial period of time. In a few minutes, we will have a presentation by the UNSD about their current work on quality frameworks. That may give us some ideas about what we may want to do as a group in the future to build on those efforts.
Support regional efforts to compare and collaborate. Sometimes, it may be easier for energy statistics programs to select and implement quality improvement strategies by working with others – i.e. by collaborating with other energy programs facing similar challenges and circumstances. For example, this collaboration could be with programs in neighboring countries, with other countries with energy statistics programs at similar stages of development, or with programs in other countries with similar energy sectors or trade flows. Working together could reduce duplication of effort, enable the sharing of good ideas and experiences, harmonize practices and outputs, etc.
Under this proposal, the working group could identify countries or regions with needs for assistance, help bring their energy statistics programs together to collaborate, and provide them with support to make improvements. This support could come in the form of sharing experiences, expertise and tools to help them in their efforts. Of course, the OG does not have dedicated resources to provide to programs to conduct this work. Rather, the support would come in the form of advice and guidance.
In the end, we thought this would be a large and ambitious project. However, as one example, at the moment, there is a tri-lateral project underway with Canada, the USA and Mexico to improve and harmonize energy data for North America. Mike Scrim made a presentation about this project yesterday. Perhaps we could report at the next OG meeting about the ongoing progress in that work. It might lead to a discussion about how similar initiatives could be launched in other regions.
10th Meeting of the Oslo Group on Energy Statistics

10. 10th Meeting of the Oslo Group on Energy Statistics
Next Steps of the WG
Begin work on the 3 proposed projects:
Establish a workplan with concrete deliverables
Allocate work among the WG members
Get contributions from a broad range of countries and organizations
Put these in place prior to the next OG meeting
Discuss the potential future activities of the WG
In conclusion, the recommendation of the WG is to focus on the 3 identified projects in the short term. The WG would:
Establish a workplan for each project, with concrete deliverables and timeframes
Determine who would take the lead
Be proactive in getting the participation and contributions of others – from the WG, from the OG, and beyond – to identify best practices, good sources of information for the clearinghouse, countries with quality review processes in place, and so on. We will need this broad input to be successful.
Hopefully, we will have made good progress by the next OG meeting. We will report back on our work, our outputs to date and the issues encountered.
The WG could also take some time to look ahead, beyond the current projects, to identify what other possibilities are out there. I already mentioned the trilateral work in Canada, USA and Mexico, as one example. Another may be the ongoing work of the UNSD on quality frameworks.
10th Meeting of the Oslo Group on Energy Statistics

11. Questions for Discussion
Do you have questions or comments about any of these proposed projects?
Do you agree with the recommendation to start with the 3 identified projects?
Do you have suggestions about best practices, documents or other quality initiatives that the WG should consider?
Any other questions, comments or ideas?
Now, we will move into our discussion period. I have listed some topics for your consideration.
10th Meeting of the Oslo Group on Energy Statistics