1. The Performance of The Polls John Curtice whatscotlandthinks.org @whatscotsthink

2. Why Polls Matter Can affect the extent and nature of media coverage Can affect the ability of campaigns to raise money and motivate activists Can influence the campaigns strategy and tactics Can influence whether people vote

3. The Challenge Regular monthly political polling in Scotland ended in 2003. Partly thanks to perceived poor performance, and partly the result of worsening newspaper finances. Thereafter mostly episodic and concentrated at election times. So companies having to estimate the attitudes of a population most have not been regularly monitoring. And for a vote intention that they have not previously attempted to estimate. While the industry is still coming to terms with the internet

4. The Performance - 1

5. The Performance - 2

6. A Time Trend? Based on 9 polls conducted Feb-May, 10 polls July-Sept, 8 polls Oct-Dec, and 7 polls in Jan-mid Feb and 10 polls mid-Feb- Mar. Dont Knows excluded.

7. House Differences Based on all polls since Sept. Dont Knows excluded

8. Panelbases Record

9. Ipsos MORIs Record

10. The Dont Knows Source: 1 st 2 cols: Average of all polls since September

11. Methodological Differences CompanyMethodPolitical Weighting/Selectio n Question PanelbaseInternet2011 Holyrood Vote; 8/10 likely to vote Was current vote; now intend to vote Sept ICMInternet2011 Holyrood VoteThink will vote Sept SurvationInternet2011 (1 st poll: 2010) & likelihood of voting Vote now TNS BMRBFace to face quota2011, incl abstention rate Intend to vote Sept YouGovInternet (18+ only)2011, + extra Lab 10, SNP 11 group Vote now Ipsos MORITelephone (RDD)None. Figures based on certain to vote Vote now

12. Record Panelbase – no previous polling close to election day ICM – previously all election polling done by phone Survation – new company since 2010 TNS BMRB – still using System Threes (not always successful) approach YouGov – over 10 years experience. Panellists previous vote collected on joining/after election Ipsos MORI – using same approach as for GB polls since 2008

13. How Their Last Poll Performed In 2011 % SNP ConstError% SNP ListError TNS BMRB45038-6 YouGov42-335-9 Ipsos MORI*45042-2 * Last Poll Conducted 14-17.4.11

14. The Extent and Impact of Weighting CompanyMen16-34C2DEImpact on % Yes Panelbase1.161.171.40+2 ICM1.031.341.33+3 Survation1.171.421.01+3 TNS BMRB1.031.150.96+3 YouGov0.901.21*1.26+1 Ipsos MORI0.981.08-+2 * Based on 16-39

15. Reported Turnout Source: Ipsos MORI (Scottish & British Polls)

16. Polls of The Interested? Survation & YouGov figures may include a few Other party voters

17. Tentative Conclusions Particularly large house effects on % Yes vote Do not simply correspond to internet vs non- internet But do correspond to experienced vs. less so Internet samples more heavily weighted But weighting etc has pushed all recent polls in a pro-Yes direction Some polls (at least) look set to be wrong – but which ones?