1 European Conference on Quality in Official Statistics - Helsinki. Finland 3-6 May 2010 The use of R-indicators in responsive survey design – Some Norwegian.

Slides:



Advertisements
Similar presentations
1 Measuring data quality by the use of a routine re-interview module Experiences from the Norwegian European Social Survey Øyvin Kleven and Frode Berglund.
Advertisements

Brian A. Harris-Kojetin, Ph.D. Statistical and Science Policy
Quality indicators for measuring and enhancing the composition of survey response Q2008 – Special topic session, July 9 Jelke Bethlehem and Barry Schouten.
TM PEN: CDC Guidance for Developing Plans and Reports 2011 TB ETN/PEN Conference Awal Khan, PhD Lead, Program Evaluation Team Field Services and Evaluation.
1 1 Effects of attrition in the Norwegian Survey on statistics on income and living conditions Marit Wilhelmsen Statistics Norway Q European Conference.
Survey Design Steps in Conducting a survey.  There are two basic steps for conducting a survey  Design and Planning  Data Collection.
Documentation and survey quality. Introduction.
Chapter 3 Preparing and Evaluating a Research Plan Gay and Airasian
RESEARCH METHODS Lecture 19
Needs Analysis Instructor: Dr. Mavis Shang
Marketing Research and Information Systems
1 Module 6 Putting It All Together. 2 Learning Objectives At the end of this session participants will understand: The monitoring and evaluation process.
18/08/2015 Statistics Canada Statistique Canada Responsive Collection Design (RCD) for CATI Surveys and Total Survey Error (TSE) François Laflamme International.
RESEARCH DESIGN.
4.04 Understand marketing- research activities to show command of their nature and scope.
The Future of Statistical Data Collection? Challenges and Opportunities Johan Erikson (Statistics Sweden) Gustav Haraldsen (Statistics Norway) Ger Snijkers.
Chapter 9 Marketing Research And Information Systems
1 1 Management Tools for Enhancing the Composition of Survey Response Q2008 European Conference on Quality in Official Statistics Rome, July 2008 Anne.
1 APPLYING MARKETING RESEARCH IN MARKETING STRATEGY FORMATION: THE CASE OF SECONDARY EDUCATION IN CROATIA Mario Vučić mag.oec.univ.spec.oec Agency for.
Exploring the use of QSR Software for understanding quality - from a research funder’s perspective Janice Fong Research Officer Strategies in Qualitative.
1 Metagora: Current Progress and the Way Forward PARIS21 Steering Committee Paris, 13 November 2007.
Monitoring and Evaluation in MCH Programs and Projects MCH in Developing Countries Feb 10, 2011.
PBIS Data Review: Presented by Susan Mack & Steven Vitto.
Descriptive and Causal Research Designs
Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts.
Evaluating a Research Report
Eurostat Overall design. Presented by Eva Elvers Statistics Sweden.
Market research in Business
KEY MANAGEMENT ROLES. POLC  There are four key management roles.  Say in your head 5 times: management roles = POLC.  DO NOT FORGET THIS!  Very easy.
1 Renee M. Gindi NCHS Federal Conference on Statistical Methodology Statistical Policy Seminar December 4, 2012 Responsive Design on the National Health.
Stop the Madness: Use Quality Targets Laurie Reedman.
20081 E-learning Lecture-10: EVALUATING THE IMPACTS OF E-LEARNING week 12- Semester-4/ 2009 Dr. Anwar Mousa University of Palestine Faculty of Information.
Understanding the Respondent – a Key to Improve our Data Collection Strategies? Seminar on Statistical Data Collection 26 th September 2013 Anton Johansson,
1 Non-Response Non-response is the failure to obtain survey measures on a sample unit Non-response increases the potential for non-response bias. It occurs.
Quality Assessment and Improvement Methods in Statistics – What works? Hans Viggo Sæbø, Statistics Norway Quality frameworks Tools.
A Theoretical Framework for Adaptive Collection Designs Jean-François Beaumont, Statistics Canada David Haziza, Université de Montréal International Total.
ICES 2007 Labour Cost Index and Sample Allocation Outi Ahti-Miettinen and Seppo Laaksonen Statistics Finland (+ University of Helsinki) Labour cost index.
Handbook on Precision Requirements and Variance Estimation for ESS Household Surveys Denisa Florescu, Eurostat European Conference on Quality in Official.
European Conference on Quality in Official Statistics Session 26: Census 2011 « Helsinki, 6 May 2010 « Census quality control with BSC: the Portuguese.
Process Quality in ONS Rachel Skentelbery, Rachael Viles & Sarah Green
CES Seminar on “Organization of data collection and sharing, and the management challenges for the implementation of SDMX” Session 1 Hank Hermans CES /
Monitoring and Evaluation in MCH Programs and Projects MCH in Developing Countries Feb 24, 2009.
Part Three Using Technology and Information to Build Customer Relationships 7 Marketing Research and Information Systems.
Survey is a strategy that involves the collection of data from a pre-determined sample.
Representativity Indicators for Survey Quality Programme: Cooperation Theme: Socio-economic sciences and Humanities Activity: Socio-economic and scientific.
United Nations Oslo City Group on Energy Statistics OG7, Helsinki, Finland October 2012 ESCM Chapter 8: Data Quality and Meta Data 1.
Kathy Corbiere Service Delivery and Performance Commission
Paradata and Survey Management in the National Survey of Family Growth (NSFG), William D. Mosher, NCHS; James Wagner, Mick Couper, & Nicole Kirgis,
Chapter 7 Data for Decisions. Population vs Sample A Population in a statistical study is the entire group of individuals about which we want information.
Chapter 06 Marketing Research and Information Systems Part Three Target Market Selection and Research.
Senior European Volunteers Exchange Network 3rd Network Meeting Vienna November 30 – December 3, 2009 Charlotte Strümpel Status quo of the on-going evaluation.
Q2010 – special topic session 33 - Page 1 Indicators for representative response Barry Schouten (Statistics Netherlands) Natalie Shlomo and Chris Skinner.
Plan for Today: Chapter 1: Where Do Data Come From? Chapter 2: Samples, Good and Bad Chapter 3: What Do Samples Tell US? Chapter 4: Sample Surveys in the.
1 1 Using administrative registers to evaluate the effects of proxy interviews in the Norwegian Labour Force Survey Øyvin Kleven, Ib Thomsen and Ole Villund.
Comparison between the EQAVET process and the ISO 9001 and the EFQM Model Hungarian experience Katalin Molnar-Stadler Information Seminar for National.
Comparison between the EQAVET Quality Cycle and the ISO 9001 and the EFQM Model Hungarian experience Katalin Molnar-Stadler Information Seminar for National.
TOPIC - Page 1 Representativity Indicators for Survey Quality R-indicators and fieldwork monitoring Koen Beullens & Geert Loosveldt K.U.Leuven.
Project Management PTM721S
Chapter 9 Designing Experiments
Chapter 2: The nonresponse problem
The European Statistical Training Programme (ESTP)
The European Statistical Training Programme (ESTP)
The European Statistical Training Programme (ESTP)
RESEARCH METHODS Lecture 19
Chapter 9 Designing Experiments
Sampling and estimation
The European Statistical Training Programme (ESTP)
Chapter 6: Measures of representativity
Chapter 2: The nonresponse problem
Workshop on best practices for EU-SILC revision, −
Presentation transcript:

1 European Conference on Quality in Official Statistics - Helsinki. Finland 3-6 May 2010 The use of R-indicators in responsive survey design – Some Norwegian experiences Øyvin Kleven, Johan Fosen, Bengt O. Lagerstrøm and Li-Chun Zhang Statistics Norway

European Conference on Quality in Official Statistics - Helsinki. Finland 3- 6 May 2010 It is the objective of the RISQ project to develop R-indicators, to explore their characteristics and to show how to implement and use them in a practical data collection environment.

European Conference on Quality in Official Statistics - Helsinki. Finland 3- 6 May 2010 R-indicators as quality-indicators Product quality –The R-indicator indicates bias conditioning on several auxiliary variables (X’s) –Will be used for a fixed set of X’s to report development in time for a given survey and between surveys Process quality –R-indicators will be used as a monitoring device to aid the surveillance of surveys during field work –R-indicators gives information on which survey to focus on –Partial R-indicators gives information on what subsample to allocate more recourses to, and what subsample we can decrease the recourses

European Conference on Quality in Official Statistics - Helsinki. Finland 3- 6 May 2010 Responsive design Computer assisted interviewing with a fully integrated computer exchanging system provides rich data on the interviewing process…and gives the opportunity to analyze paradata and the data from the survey itself during the fieldwork period (Thomsen et al. 2006). Real-time access to information about the survey process enables the survey organisation to analyse the quality of the fieldwork process and to make mid-course decisions and design alterations during the fieldwork period (Hapuarachchi, March and Wronski 1997; Couper 1998; Scheuren 2001). This is often known as “responsive survey design” (Heeringa and Groves 2005).

European Conference on Quality in Official Statistics - Helsinki. Finland 3- 6 May 2010 To explore the use of R-indicators in a “natural” survey setting: We had one pilot in The Netherlands and one in Norway. The purpose of the pilot in Norway was to gain experience of using the R-indicators in an ongoing survey in a realistic setting. The pilot was implemented in the ongoing Level of Living Survey (LLS) 2009, ( respondents, aged 16 to 69 years).

European Conference on Quality in Official Statistics - Helsinki. Finland 3- 6 May 2010 Design of the pilot From the gross sample, respondents where randomly selected for the pilot sample before the data collection started. The pilot sample was then randomly split in two groups respondents served as the experiment group while respondents formed the control group. The pilot fieldwork was conducted during six weeks in October and November 2009 The experiment group was subjected to rigorous supervision with respect to the R-indicator and partial R-indicators, while the control group was treated as the rest of the regular LLS.

European Conference on Quality in Official Statistics - Helsinki. Finland 3- 6 May 2010

Partial R-Indicators P2: Deviation between the mean response propensity within group_k and the overall mean response propensity P3: Sum of squared conditional P2_k given the other explanatory variables. Large P3 indicates that the effect of the variable of concern can not be 'conditioned away' by the other exp. variables

European Conference on Quality in Official Statistics - Helsinki. Finland 3- 6 May 2010 Attempts to intervene during fieldwork Treatment groupRegular group Prioritize young adults in the CATI call schedule. Use mobile phone numbers based instead of land-line No specific attempt

European Conference on Quality in Official Statistics - Helsinki. Finland 3- 6 May 2010 Measuring the effects of the attempts on the R-indicator

European Conference on Quality in Official Statistics - Helsinki. Finland 3- 6 May 2010 Conclusions The R-indicator is a useful tool for evaluating responsive survey design and data collection process In our pilot the responsive survey design was weak but the R-indicator was strong, it’s a standardised tool The interpretation and use of R-indicators requires some training of field staff In most cases R-indicators can be used as a monitoring device for deciding what kind of respondents to put in the parking lot (harder to invent smart strategies to persuade refusalers)

European Conference on Quality in Official Statistics - Helsinki. Finland 3- 6 May 2010 Future use of R-indicators in a systematic quality approach –Database (historic)  For contact. Number of contact achieved and attempts made  For Corporation. Number of response, refusal and absentee –Planning  Retrieve from database relevant propensities  Iterative optimization –Allocation (relevant anticipated response rates) –Fit and evaluate target criterion (say, R-indicator with pre-fixed X) –Adjust provided improvement is feasible –Adapting: repeated optimization during data collection  With historic propensities  Allow for updating of certain propensities based on fresh data

European Conference on Quality in Official Statistics - Helsinki. Finland 3- 6 May 2010 Thank you for your attention!