1 Some reflections on quality in research applications Sharon Witherspoon Deputy Director Nuffield Foundation

2

3 The Nuffield Foundation   Endowed charitable trust   Endowed in Morris Motors shares, 1943   General objective: ‘to advance social well-being, particularly by means of scientific research’

4 Main Areas of Activity   Social research, social science and social innovation (welfare)   Education   Science (mainly new capacity and some areas of science policy)

5 We fund  Critical ‘synthetic’ reviews, drawing together and evaluating evidence from a range of studies  Evaluation of particular interventions – both quantitative and qualitative  Self-contained research projects: oQuantitative and qualitative oPrimary data-collection and secondary analysis oDescriptive and experimental

6 Key Criteria for Project Grants   Is the topic of interest to the Foundation?   Does project have implications for policy or practice?   Does the plan for dissemination back this up? Will it be disseminated to practitioners or policy makers?   Is the research design rigorous and appropriate? Methodology of highest standard, precisely described? Is methodology apt for the questions being asked?   Are the results truly of interest, or is it ‘fancy that’ research?   Is the right sort of team assembled?

7 A ‘judgement’ framework  Explicitly based on judgements oBy ‘scientific’ referees oBy referees judging importance, relevance, timeliness oBy Foundation Trustees (zero-sum)  But not arbitrary: oProcedural quality does not guarantee perfect judgements, but is correlated with it

8 Indicators of quality (1) Precision of concepts – operationalisations Precision of hypothesis or of exploration of meaning, taking meaning as problematic (not just what people say). Sceptical scrutiny: consideration of alternatives Alternative hypothesis (disproving hypotheses, rather than proving in classic mode), or consideration of alternative meanings and mechanisms. Some detachment from preferred explanations.

9 Indicators of quality (2) Clarity about epistemological aims: Descriptive: What being described, why description matters (lack of previous knowledge, refuting commonly held views) May be useful to think of contrast groups (is the phenomenon really specially important to group being studied?) Explanatory Here contrast or comparison groups, or control groups, or longitudinal study almost essential

10 Indicators of quality (3) Clarity about how methods illuminate the underlying questions: Are questions of incidence, incidence, prevalence, distribution, spread, correlations crucial to your argument? …..Qualitative alone unlikely to suffice Are questions of meaning, understanding, motivation, affect crucial to your argument? …..Quantitative alone unlikely to suffice Do you need both? In what way: preliminary; joint; elucidation of mechanism ….all this before consideration of more technical questions about research methods, analysis techniques (though in fact iterate back and forth)

11 Some examples (1) Use of qualitative to illuminate concepts, how people understand them, language used: o oHazel Genn, et al: Paths to Justice o oWellings, Field, Johnson, Wadsworth. Sexual behaviour in Britain Use of qualitative to get “beneath” superficial justifications o oSuccessive projects by Carol Smart and others on cohabitation and commitment

12 Some examples (2) Use of quantitative descriptive study to put this into context o o"Just a piece of paper? Marriage and cohabitation" by Anne Barlow, Simon Duncan, Grace James and Alison Park in British Social Attitudes: the 18th Report Use of qualitative to adduce evidence about a mechanism – why a correlation ‘causes’ something: o oJohn H. Laub, Robert J. Sampson. Shared Beginnings, Divergent Lives: Delinquent Boys to Age 70

13 Judgements of ‘import’   Clearly less objective, more room for discussion   Policy/practical import: oNOT tied to what government or powerful feel is important oRational consideration of balancing setting long-term agenda with short- term effect oPower to stop – consider the discussion of why government doesn’t do more pilots in the report ‘Trying it out’ ….versus influence…. versus agenda-setting   Scientific importance: oTakes the field forward oNew concept, new operationalisation, new site of study oInnovation ….. extension ….. replication

14 Some personal predictions and predilections…..  Will (and should) see more invested in larger, quantitative or longitudinal studies  Will (and should) see more concern with outcomes  Will (and should not) see diminution in government enthusiasm for much research

15 And even more controversially …  Will (and should) see that researchers who get good quantitative training early on have a head-start in thinking through analytic framework and posing big questions for future research programmes (as opposed to projects)  Refuse to predict though how this will work structurally with research concentration, RAE, funding changes ahead …..

16