Are interviews costing £0. 08 a waste of money Are interviews costing £0.08 a waste of money? Reviewing Google Surveys for Wisdom of the Crowd projects Geoffrey Roughton & Iain MacKay X-MR Limited ASC conference, September 2016

Introduction

Table 1: Surveys & Interviews Row Labels Surveys Interviews UK 125 65514 US 30 15076 Grand Total 155 80590

Table 2: Surveys & Interviews - II Subject Surveys UK 100 EU Referendum 46 Labour leadership 7 London Mayor 8 UK General election 2015 39 US 21 Democratic candidate 13 Republican candidate Grand Total 121

Figure 1:Specimen question presentation

Table 3: Length of interview (seconds) Subject Length in seconds UK 19 UK General election 2015 Labour leadership 18 London Mayor 26 EU Referendum US 23 Democratic candidate 22 Republican candidate 24 Grand Total 20

Table 4: Impressions & completes

Table 5: Source of interviews

Table 6: Example Google weighting

Table 7: UK Population& Google

Wisdom of crowds methodology

Why wisdom of crowds? Scottish referendum 2014 – much less close than polls predicted Maybe people report other’s behaviour more reliably than their own Surowiecki’s 2004 book – “The Wisdom of Crowds”

Criteria for a wise crowd Criterion Explanation How addressed by sample surveys Diversity of opinion Each person should have private information even if it's just an eccentric interpretation of the known facts. We draw a random sample; people are not selected based on any relationship to the issue in question. Independence People's opinions aren't determined by the opinions of those around them. This is more questionable, especially if we are asking people what they believe is the consensus opinion of others. Decentralization People are able to specialize and draw on local knowledge. We (attempt to) draw a nationally representative sample Aggregation Some mechanism exists for turning private judgments into a collective decision. We arrive at the collective decision by simple aggregation of responses.

Studies of UK political events

General Election outcome – initial wording

General Election outcome – revised wording “Who do you think most likely to be UK Prime Minister after the May 2015 election (whatever your personal preference)?” David Cameron Ed Miliband Nick Clegg Nigel Farage Others (specify) …

General election time series

Conventional polls showing party prospects Labour Conservative Liberal Democrat UKIP Green 15-day average trend line of poll results from 6 May 2010 to 7 May 2015 (Wikipedia)

UK general election 2015 - results Who will be next PM? Party X-MR final % Share of vote Share of seats David Cameron Conservative 48% 37% 51% Ed Miliband Labour 35% 30% 36% Nick Clegg Liberal Democrat 5% 8% 1% Nigel Farage UKIP 10% 13% 0% Other Other Parliamentary 3% 9% 12%

Labour Leadership Election 2015 – X-MR polls “Who do you think is most likely to be elected Leader of the Labour Party in the September 2015 election (whatever your personal preference)?”

Labour Leadership Election 2015 (results) Candidate X-MR final poll YouGov poll Result 04-Sep 10 Aug 12-Sep Jeremy Corbyn 52% 53% 59% Andy Burnham 20% 21% 19% Yvette Cooper 16% 18% 17% Liz Kendall 12% 8% 5%

London Mayoralty Election 2016 Respondents in England Respondents in London Zac Goldsmith Sadiq Khan

London Mayoralty Election 2016 (results) First round result Final X-MR poll (Londoners) Final X-MR poll (all England) Final conventional Poll 26/4 -1/5 Final poll, adjusted Caroline Pidgeon (Lib Dem) 5% 9% 6% 3% 4% Peter Whittle (UKIP) 11% Sadiq Khan (Labour) 44% 38% 39% 35% 48% Siân Berry (Green) 15% 10% Zac Goldsmith (Conservative) 23% 32% 26% Other 8% 2%

Brexit referendum 2016 – initial wording “In a "UK in/out of the EU" referendum, what do you think most voters in the UK will choose (whatever your own preference)?”

Brexit referendum 2016 – final wording “In the referendum on Thursday 23rd June, which answer do you think most voters in the UK will choose (whatever your own preference)?”

Brexit - result Outcome X-MR results on 20-June USurv on 21-June Final poll-of-polls Leave 52% 39% 41% 48% Remain 61% 59%

Other and ongoing studies Exploratory studies leading up to the general election US Presidential primaries Ongoing Labour leadership 2016 US Presidential Election

Conclusions We have demonstrated that £0.08 interviews are not a waste of money Google are not alone in offering bargain basement prices Large scale general population survey with relatively low data collection costs are with us now Setting these up, analysing & interpreting them are an opportunity for market researchers. We have a solution. All we need now is to find the problems it helps solve.