Non-response weighting at NatCen/ScotCen - a brief (and biased) history Susan Purdon Survey Methods Unit, NatCen.

Slides:

Advertisements

Similar presentations

Using weighted data on the HSE Susan Purdon Survey Methods Unit, NatCen.

Advertisements

Outline of talk The ONS surveys Why should we weight?

Longitudinal LFS Catherine Barham and Paul Smith ONS.

Traps and pitfalls in medical statistics Arvid Sjölander.

GrowingKnowing.com © Sampling A sample is a subset of the population In a sample, you study a few members of the population In a census, you study.

1 Sampling Telephone Numbers and Adults, and Interview Length, and Weighting in the California Health Survey Cell Phone Pilot Study J. Michael Brick, Westat.

Sampling Partially Adapted from The Research Methods Knowledge Base, William Trochim (2006). & Methods for Social Researchers in Developing Counries, The.

Statistics 100 Lecture Set 7. Chapters 13 and 14 in this lecture set Please read these, you are responsible for all material Will be doing chapters

GATHERING DATA Chapter Experiment or Observe?

Experimental Design Statistics Introduction Remember, population and sample Samples –1523 randomly chosen voters –6 Black capped chickadees –The.

Questions:  Are behavioral measures less valid and less reliable due to the amount of error that can occur during the tests compared to the other measures?

2.2: Sampling methods (pp. 17 – 20) Probability sampling: methods that can specify the probability that a given sample will be selected. Randomization:

Why sample? Diversity in populations Practicality and cost.

Sampling and Experimental Control Goals of clinical research is to make generalizations beyond the individual studied to others with similar conditions.

American Psychological Association’s Task Force on Statistical Inference TFSI 1999.

C82MCP Diploma Statistics School of Psychology University of Nottingham 1 Overview Central Limit Theorem The Normal Distribution The Standardised Normal.

Variables and Measurement (2.1) Variable - Characteristic that takes on varying levels among subjects –Qualitative - Levels are unordered categories (referred.

Understanding sample survey data

Error and Sample Sizes PHC 6716 June 1, 2011 Chris McCarty.

~ Draft version ~ 1 HOW TO CHOOSE THE NUMBER OF CALL ATTEMPTS IN A TELEPHONE SURVEY IN THE PRESENCE OF NONRESPONSE AND MEASUREMENT ERRORS Annica Isaksson.

SAMPLING AND STATISTICAL POWER Erich Battistin Kinnon Scott Erich Battistin Kinnon Scott University of Padua DECRG, World Bank University of Padua DECRG,

FINAL REPORT: OUTLINE & OVERVIEW OF SURVEY ERRORS

1 COMM 301: Empirical Research in Communication Kwan M Lee Lect5_1.

CHAPTER 7, the logic of sampling

PEAS wprkshop 2 Non-response and what to do about it Gillian Raab Professor of Applied Statistics Napier University.

1 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Sampling.

1 Social Research Methods Surveys. 2 Survey Characteristics Collecting a SMALL amount of data in STANDARDISED form from RELATIVELY LARGE NUMBERS OF INDIVIDUALS.

Scot Exec Course Nov/Dec 04 Ambitious title? Confidence intervals, design effects and significance tests for surveys. How to calculate sample numbers when.

How survey design affects analysis Susan Purdon Head of Survey Methods Unit National Centre for Social Research.

Inferential Statistics & Test of Significance

Sampling : Error and bias. Sampling definitions  Sampling universe  Sampling frame  Sampling unit  Basic sampling unit or elementary unit  Sampling.

Chapter 12: AP Statistics

● Midterm exam next Monday in class ● Bring your own blue books ● Closed book. One page cheat sheet and calculators allowed. ● Exam emphasizes understanding.

Chapter 5 Section 3 Part 1.  Often when we hear of a sample we do not know the truth behind the sampling process.  The whole truth about opinion polls.

Sample Design Establishments Surveys Stuart Brown Research, Design & Evaluation January 2013 STATISTICAL INSTITUTE OF JAMAICA.

Sampling. Concerns 1)Representativeness of the Sample: Does the sample accurately portray the population from which it is drawn 2)Time and Change: Was.

From Sample to Population Often we want to understand the attitudes, beliefs, opinions or behaviour of some population, but only have data on a sample.

Chap 20-1 Statistics for Business and Economics, 6e © 2007 Pearson Education, Inc. Chapter 20 Sampling: Additional Topics in Sampling Statistics for Business.

1 Psych 5500/6500 Populations, Samples, Sampling Procedures, and Bias Fall, 2008.

Data Collection and Sampling

Sampling Class 7. Goals of Sampling Representation of a population Representation of a population Representation of a specific phenomenon or behavior.

Instructor Resource Chapter 8 Copyright © Scott B. Patten, Permission granted for classroom use with Epidemiology for Canadian Students: Principles,

Scot Exec Course Nov/Dec 04 Survey design overview Gillian Raab Professor of Applied Statistics Napier University.

1 Introduction to Survey Data Analysis Linda K. Owens, PhD Assistant Director for Sampling & Analysis Survey Research Laboratory University of Illinois.

Chapter 41 Sample Surveys in the Real World. Chapter 42 Thought Question 1 (from Seeing Through Statistics, 2nd Edition, by Jessica M. Utts, p. 14) Nicotine.

Handling Attrition and Non- response in the 1970 British Cohort Study Tarek Mostafa Institute of Education – University of London.

Using administrative registers in sample surveys European Conference on Quality in Official Statistics 3-6 May 2010 Kaja Sõstra Statistics Estonia.

GROUP 2 Practical C. Question 1 Cut off will depend on the country situation : 1 pig may be significant Frequency distribution – take the lower 10 – 20%

The Challenge of Non- Response in Surveys. The Overall Response Rate The number of complete interviews divided by the number of eligible units in the.

Lecture # 6:Designing samples or sample survey Important vocabulary Experimental Unit: An individual person,animal object on which the variables of interest.

Copyright 2010, The World Bank Group. All Rights Reserved. Reducing Non-Response Section A 1.

5-4-1 Unit 4: Sampling approaches After completing this unit you should be able to: Outline the purpose of sampling Understand key theoretical.

Chapter 22: Comparing Two Proportions. Yet Another Standard Deviation (YASD) Standard deviation of the sampling distribution The variance of the sum or.

Sources of Errors M&E Capacity Strengthening Workshop, Addis Ababa 4 to 8 June 2012 Arif Rashid, TOPS.

Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee.

Bias in research Objective 1.6 Prologue. Building Context Bias is a form of systematic error that can affect scientific investigations and distort the.

 When every unit of the population is examined. This is known as Census method.  On the other hand when a small group selected as representatives of.

Errors in Sampling Objective: to identify the types of nonsampling errors.

Status for AP Congrats! We are done with Part I of the Topic Outline for AP Statistics! (20%-30%) of the AP Test can be expected to cover topics from chapter.

1 Chapter 11 Understanding Randomness. 2 Why Random? What is it about chance outcomes being random that makes random selection seem fair? Two things:

Statistical Inference for the Mean Objectives: (Chapter 8&9, DeCoursey) -To understand the terms variance and standard error of a sample mean, Null Hypothesis,

MATH Section 6.1. Sampling: Terms: Population – each element (or person) from the set of observations that can be made Sample – a subset of the.

Evaluating imputation of sex and age for substitutes in substitute households Michael Ryan 2008 UNECE Work Session on Statistical Data Editing.

Presented: 2009 Canadian Users Stata Group Meeting

Nonresponse adjustments and calibration: a comparison between two methods to weight the Labour Force Survey Tania Borg Principal Statistician Labour Market.

The Scientific Method.

MATH 2311 Section 6.1.

Inferential Statistics & Test of Significance

Inference for Sampling

What do Samples Tell Us Variability and Bias.

Presentation transcript:

Non-response weighting at NatCen/ScotCen - a brief (and biased) history Susan Purdon Survey Methods Unit, NatCen

The case against 10 years ago non-response weighting not the norm Reasons: Response rates pretty high on govt sponsored surveys Beyond age and sex, not many control totals around

The case against Many surveys had long histories: weighting would introduce discontinuities Non-response is subjective and cosmetic! No two statisticians would create the same set of non-response weights. Unscientific. Weighting makes analysis more complex and error prone.

The counter-arguments Non-response rates have now grown (c. 1%/year) The under-representation of certain groups is a constant and clearly biasing Trends in response rates undermine the argument against introducing discontinuities ONS have concluded that all national statistics should be calculated on a consistent basis (same age-sex-region distn)

The implications Most surveys now come with non-response weights New industry of calculating weights for old surveys Not too comfortable a position (what if new method around the corner?)

How it’s done on gen pop surveys Approach kept as simple as possible. Adjust (standardise) for age-sex; no major attempt to eliminate other biases. Usual approach = calibration weighting where (a) adjust to national age-sex totals; but (b) give all household members same weight.

Calibration weighting Biases observed for individuals, but non- response is a household process (yes/no determined by whoever answers the door); Implies that probability of response depends upon who you live with (e.g. young men living with parents v. other young men) So only under-represent young men in certain types of households

Calibration weighting Calibration weighting attaches a probability of selection per household that ‘explains’ difference between survey and population age/sex. Assumes that non-response is not an independent process

More elaborate approaches Non-response hierarchies internal to the survey Panel attrition Studies with informative sampling frames

Suggestions for analysts Non-response weights are subjective. You don’t have to trust them. Check that observed differences are not attributable to weighting If you think the survey organisation has missed a trick then tell them!