Quality Frameworks: Implementation and Impact Notes by Michael Colledge

Terms of Reference Some organisations are using quality frameworks others are not. Does it really make a difference? What is important, the existence of a quality framework, support of top management, quality of the organisation?

Content What is a quality (assurance) framework? Who are stakeholders in a quality framework? What are costs and benefits? Which organisations have a quality framework? What is impact on quality of a quality framework? To what extent can quality frameworks converge?

What is a quality (assurance) framework? Umbrella for quality practices Refers to survey programme rather than single survey Covers range of survey operations rather than single subject Involves template that can be used for quality assessment

Scope of a quality framework? Transformation Input Output

Scope of a quality framework? Product (output data, services) Process (collection transformation, dissemination) Input data Organisation Difference in scope limits convergence

Dimensions of Data Quality Generic set: Relevance, accuracy, timeliness, accessibility, interpretability, comparability, coherence Could be standardised Could be validated by feedback from users Check against ISO 20252

The other dimension: cost/performance Quality Effectiveness Product “Doing the right thing” Quality indicators Performance Efficiency Process “Doing the thing right” Performance indicators

Quality Related Mechanisms and Topics Mechanisms Review (self, peer, external) Reporting (indicators, reports) Labelling, declarations Sharing best practices Topics Metadata management Programme monitoring /evaluation, audit TQM Management

Types of Quality Framework by Owner and Applier NSO owned and applied Statistics Canada quality framework IO owned and applied OECD quality framework IO owned NSO applied IMF DQAF

Types of Quality Framework by Owner and Applier NOS owned IO applied What value does IO add? Is every data item actually used? What efforts have been made to harmonise across IOs? How does IO quality assure its outputs? Questions would be answered by IO owned and applied framework

Stakeholders in Quality Framework Designers/owners Organisational units that apply framework Users of data collected via framework template

Costs and Benefits Benefits Increased awareness of quality concepts, components, best practices Systematic quality assessment Identification of quality problems Comparisons over time, across organisational units Indications of additional resource and training needs Costs - staff time

Which Organisations Have Quality Frameworks? National International/Supranational

Impacts of Quality Frameworks? Statistics Canada Quality Guidelines Statistics Canada Quality Framework ESS Self-Assessment Checklist (DESAP) OECD Quality Framework IMF Data Quality Assessment Framework (DQAF) Australian Statistical Clearing House

Impacts of Quality Frameworks: Conclusions Quality frameworks serve range of useful purposes Does not have to be developed from scratch Can adapt existing ones (best practice) Development process itself has benefits

Convergence of Quality Frameworks Benefits Common terminology Better understanding of concepts Promoting current best practice Reducing reporting burden Limitations Differing objectives, scope, users For example, process orientation v product orientation