1. A poor prognosis for survey methods ? Assessing the health of survey methods in social research McCaughey Centre Survey Methods Seminar May 2008

2. 2 The Topic Survey methods have been a stalwart tool for social science research, however, traditional approaches to survey methods are becoming increasingly difficult to undertake. Letterboxes are overflowing with ‘junk’ mail, households are fatigued by unsolicited telephone calls and electronic home protection systems are turning homes into fortresses. With low response rates - what does this mean for ideals of ‘representativeness’ and what does ‘representativeness’ mean in contexts of increasing diversity within populations? For survey methods to adapt to these changing contexts for social science research and to continue to provide reliable and valid insights into people’s lives there is a need for innovative approaches. Sample frames Response Rates Methodological Innovations

3. 3 Credentials About the Social Research Centre Established in 2000 as a specialist provider of survey research services to government and academia Operates as a social research supplier under the umbrella of the market and social research industry 25 permanent staff 180 telephone interviewers 110 seat Call Centre Our aim is to enhance knowledge and contribute to informed decision-making by undertaking high quality social and health research

4. 4 About the Social Research Centre large-scale survey design, conduct and management longitudinal surveys population health research community attitudes research campaign evaluation research research amongst persons of culturally and linguistically diverse backgrounds Programme evaluation and policy development research Services provided

5. 5 About the Social Research Centre Research Areas

6. 6 Random Digit Dialling (RDD)

7. 7 RDD Background Up until quite recently the Electronic White Pages was considered complete and accurate for most survey research purposes The final version of the Desktop Marketing Systems (DtMS) Electronic White Pages (EWP) product was released in July 2004 and is still widely used by market research companies but is no longer used for high quality social research Random Digit Dialing (RDD) methodologies are now the norm rather than the exception One of the main barriers to the efficient use of RDD has been overcome by the development of automatic dialing technologies The rise of RDD and the demise of the Electronic White Pages (EWP)

8. 8 EWP versus RDD Methods of RDD generation Launched in 2003 Government’s 5 year plan for Queensland women Initiatives to address areas where women experience inequity, under-representation and need Generation process involves Undertaking a random selection of “seed” numbers from the 2004 Electronic White Pages Retaining the eight digit stem and deleting the last two digits Randomly generating two numbers to append to the eight digit stem Washing the resultant randomly generated ten digit numbers against business listings Send those records for which we get a confirmed address a primary approach letter

9. 9 EWP versus RDDSelected characteristics CharacteristicsListedUnlisted VPHS 2005 (n=7,560)5,4492,111 Employed50.760.3 Retired33.215.4 University educated22.928.3 Household income <$40k pa49.746.6 Language spoken at home - English only89.884.5 Dependent children in household32.042.3 Divorced / separated11.016.4 Resident in area less than one year1.718.9 Renting10.630.9 Selected demographic characteristics of persons with listed and unlisted phone numbers

10. 10 EWP versus RDDSelected characteristics VPHS 2005 (n=7,560)5,4492,111 Holds private health insurance52.543.6 Daily smoker14.622.0 Has an alcoholic drink every day18.913.6 Sought professional help for mental health related illness8.612.2 Could raise $2,000 within 2 days in an emergency85.079.4 ICVS 2004 (n=6,000)4,7301,270 Victim of at least one crime in last 5 years51.358.0 Victim of violent crime in last 5 years17.524.4 Feels safe waiting for public transport after dark60.755.9 All of the results are statistically significant at the 95% confidence interval. Selected survey measures for persons with listed and unlisted phone numbers CharacteristicsListedUnlisted

11. 11 RDD Improved representation Launched in 2003 Government’s 5 year plan for Queensland women Initiatives to address areas where women experience inequity, under-representation and need RDD samples can access different persons in terms of: Age distribution Demographic characteristics Key usage, attitudinal or behavioural variables Inclusion of persons with unlisted numbers in the sample universe by using RDD techniques can achieve a more representative sample of the general population BUT IN THE SPACE OF A DECADE THE REPRESENTATIVENESS OF SAMPLES OBTAINED FROM RDD FRAMES IS STARTING TO BE ERODED…

12. 12 RDD Limitations of RDD frames Launched in 2003 Government’s 5 year plan for Queensland women Initiatives to address areas where women experience inequity, under-representation and need Little or no coverage of areas poorly represented in source listing e.g. emerging growth areas The emergence of ‘mobile only’ households. Non coverage of households without a landline (used to be 1-2% but now probably around 13%) 1,2 Decreasing ability to obtain address matches and send out primary approach letters 112.8% of households in the US in 2006. National Health Interview, Public Opinion Quarterly, Vol. 71 2One in four adults aged 18 to 24 years, Ibid.

13. 13 Response Rates Are response rates declining? Launched in 2003 Government’s 5 year plan for Queensland women Initiatives to address areas where women experience inequity, under-representation and need Generally, probably yes but not inevitably so In market research it has been predicted that … “Reach is increasing but co-operation is declining and non- contact is increasing, leading to a 3% decline in response rate per annum” (Bednall, 2000). Is the same true in social research …Possible yes but not inevitably so and certainly not to the same extent …. It depends on how hard you work…

14. 14 Response Rates Selected findings Government’s 5 year plan for Queensland women

15. 15 Response Rates The current state of play Launched in 2003 Government’s 5 year plan for Queensland women Extended call regimes Longer fieldwork periods Increased use of Primary Approach Letters Increased reliance on bilingual interviewing More sophisticated interviewer training Leaving messages on answering machines Increased reliance on survey sponsors to promote their surveys Creating an internet presence for surveys More calls to mobiles to obtain interviews with selected respondents Overall only marginally lower than they were several years ago or at least not substantially lower. Certainly no precipitous decline. This has been achieved by:

16. 16 Population Research The future? Government’s 5 year plan for Queensland women Advocacy is needed Further research into advances in survey methodology is needed Survey methods are certain to adapt to the changed environment as the alternative is sub standard population research Access to the Integrated Public Number Data Base is needed to overcome these deficiencies Mixed / multi modes of data collection On line access panels 1 Access to mobile phone directories 1. 2006 Census, 58% of households had an internet connection, 37% had broadband.

17. 17 Population Research The future? No, just continued innovation … A poor prognosis for survey methods?

18. 18 CATI Computer Assisted Telephone Interviewing Not completely random More disconnected, non-household, fax Reaches silent numbers and recent connections Random Digit Dialling (RDD) Computerised survey management system (SMS) Direct data entry Good response rates Economical Quick CATI   ??    ? __ ??

19. 19 RDD Random Digit Dialing The proceedings from the first CATI Population Health Forum in 1998 record the following advantages of CATI and RDD: CATI  quick  economical  good response rates  direct data entry  computerised survey management system (SMS) Random Digit Dialling (RDD)  reaches silent numbers and recent connections  more disconnected, non-household, fax  not completely random