WERS the Validity: What’s The Current State of Play? Dr Andrew R Timming Reader in Management University of St Andrews

Preface WERS is the single greatest source of data on work and employment in the history of not just the United Kingdom, but also the world No other dataset comes close But scientific advancement depends on critique

Background From WIRS (1980, 1984, 1990) to WERS (1998, 2004, 2011) WERS 2011: (i) management questionnaire, (ii) financial performance questionnaire, (iii) worker representative questionnaire, (iv) survey of employees and (v) employee profile questionnaire 2,680 face-to-face interviews with managers; 1,002 interviews with employee reps; 21,981 respondents completed the survey of employees

Structure of Presentation Look at my top 10 “best practices” for survey design Evaluate how the WERS instruments stand in relation to those top 10 practices Focus mainly on survey of employees Summarize key points to draw from the presentation

Tip 1 Surveys should be relevant and timely Relevant according to whose judgement? Evidence of timeliness is clear: workplaces in the shadow of recession But we still have to ask how much value is added by some of the questions.

Timeliness

Relevance

Tip 2 The layout should be clear, not cluttered If the survey is too spread out, it reduces response rate; too cluttered, it reduces the quality of the data On this point, the Survey of Employees gets an A+ Nice aesthetics, clear instructions, inviting, good boundaries and delineation between items, clear flow

Clear Layout

Tip 3 Strike a balance on length of the survey MQ response rate=46.3%; WR response rate=63.9%; Survey of Employees=54.3%; financial performance questionnaire= 31.8% Face-to-face administration is usually much higher than postal, but in WERS 2011 we see the opposite Was the length of the instruments driving away respondents?

Tip 4 Items should be short and simple Principle of parsimony Express what you need to express in as few words as possible and as clearly as possible On this point, WERS does a pretty good job, but not perfect...

Tip 5 Avoid leading questions ‘Do you agree that Scotland should be an independent country?’ vs ‘Should Scotland be an independent country?’ How we ask a question can result an an expected response

Overall, how satisfied or dissatisfied are you with …

Tip 6 Response categories should be exhaustive and mutually exclusive Exhaustive: all possibly choices are listed Mutually exclusive: no two choices overlap WERS is generally good on this point

Mutually exclusive?

Tip 7 Respondents should be knowledgeable about the topic or topics. Albanian foreign policy towards China... No problem with survey of employees, but there may be concerns with management and employee rep questionnaires

Are workers’ reps qualified to speculate?

Can a manager really know the answer to this question?

Tip 8 Items must not be double-barreled ‘Double-barreled’ implies that a single item is asking two questions in one Double-barreled items lead to serious questions about whether we’re measuring what we say we’re measuring On this point, WERS could be improved

Double-barreled items

Tip 9 Complex constructs require multiple items Multiple items allow for the construction of latent variables On this point, WERS does a really good job

Tip 10 Survey instruments should always be piloted and reformed according to results Obviously, the fact that there are different iterations of the survey implies sufficient piloting But on the question of reform, we need to continue to refine the instrument, even though this may lead to a break in continuity from previous versions

Summary On the whole, WERS 2011 is a fantastic resource and a very well designed set of instruments... but it’s not perfect What is included and excluded from surveys is a political choice and influenced by power Do we tolerate imperfection for the sake of continuity?

Counter-Arguments ‘We shouldn’t change or reduce the amount of data gathered because they are always relevant to someone’. ‘Other surveys have used these same measures, so why shouldn’t we as well’? ‘Our response rates are, in fact, much better than other comparable surveys’. ‘Effecting changes now will make it difficult to do comparable longitudinal analyses’. ‘You’ve cherry picked a few examples to make the whole survey look bad’. ‘Making changes would cost too much money without an effective return’.