Question-led systematic reviews: implications for searching Methods Festival 2006 Sandy Oliver Institute of Education, University of London

Question-led approach Starts with the nature of the research questions rather than the mechanisms for finding the answers. artificial boundaries between qualitative and quantitative research methods; potential for bias in asking questions as well as finding answers. identify the most pressing conceptual and technological issues and what aspects of MRS are common across review questions and which are question specific. resonates with the way in which users tend to view research. They consult research evidence in order to inform decisions relating to their policy or practice.

The proposal Develop a typology of research questions for synthesis Further develop a question-driven Methods for Research Synthesis framework for all types of review questions, specifying the generic issues and issues specific to types of review question and research method.

The presentation Three approaches to exploring question- led review methodology –a) Existing typologies of questions –b) Analysis of Social Science questions –c) Questions and methods of diverse types of reviews Model for question-led syntheses Implications for searching

What are the characteristics of X (and Y etc)? Is X related to Y (and Z etc)? How and why are X and Y (and Z etc) related? etc What types of questions?

What are the characteristics of X (and Y etc)? Is X related to Y (and Z etc)? How and why are X and Y (and Z etc) related? etc Elements of different questions: What? When? How much? Etc. Asked of single, multiple and related items. In order to generate, explore, test knowledge. What types of questions?

What are the characteristics of X (and Y etc)? Is X related to Y (and Z etc)? How and why are X and Y (and Z etc) related? etc Elements of different questions: What? When? How much? Etc. Asked of single, multiple and related items. In order to generate, explore, test knowledge. Spectrum of quantitative, qualitative and mixed methods What types of questions

What are the characteristics of X (and Y etc)? Is X related to Y (and Z etc)? How and why are X and Y (and Z etc) related? etc Elements of different questions: What? When? How much? Etc. Asked of single, multiple and related items. In order to generate, explore, test knowledge. Spectrum of quantitative, qualitative and mixed methods Question-led approach

(a) Typologies of questions Education, Coe (2004) Tabulating common and unique types of questions Descriptive: What exactly is X? What are the characteristics of X? Historical: what happened at X? Evaluative: How good is X? What is its impact? Philosophical: what is the justification/ value of X? Theoretical: what is the relationship between X and Y? Causal: what are the effects/ results/ benefits of X? Emancipatory: how can X be empowered? Ameliorative: How can X be improved?

(a) Typologies of questions Public health, Petticrew and Roberts (2005) Tabulating common and unique types of questions Salience: does it matter? Process of service delivery: how does it work? Acceptability: Will people be willing to or want to take up the service offered? Appropriateness: Is this the right service for these people? Satisfaction: Are users, providers, and other stakeholders satisfied with the service? Effectiveness: Does this work? Does doing this work better than doing that? Safety: will it do more harm than good?

(a) Typologies of questions Diffusion, dissemination,& sustainability of innovations in health services delivery and organisation, Greenhalgh, Robert & Bate (2004) Tabulating common and unique types of questions What are the beliefs, perceptions, experiences of group G? What is the nature of process P and is it transferable to context Q? Do attributes A, B, C etc account for event D? Does intervention X produce pre-defined outcome Y (and how large is the effect)? Cost effectiveness: Is it worth buying this service? What research has been done into complex field F?

(a) Typologies of questions Sociology, Lofland & Lofland (1995) Tabulating common and unique types of questions What are the topics types? What is this thing? What type of some already identified unit or aspect is it? What are the topics frequencies? What are the topics magnitudes? What are the topics structures? What are the topics processes? What are the topics causes? What are the topics consequences? What is human agency? What is the situation being dealt with? What strategies are being employed in dealing with that situation?

(a) Typologies of questions Johnston, 2003 Teaching and Learning, University College Dublin Education and psychology, University of Utah (2004) Education, Coe (2004) Community Nursing, Meadows (2003) Statistical methods, Morgan and Griego Public health, Petticrew and Roberts (2005) Campbell Collaboration: education, crime and justice, and social welfare, Cottingham et al (2006) Sociology, Lofland & Lofland (1995) Methods of inquiry syllabus, Rutgers University Diffusion, dissemination,& sustainability of innovations in health services delivery and organisation, Greenhalgh, Robert & Bate (2004) News research (journalism), University of Canberra (2004) Tabulating common and unique types of questions

(a) Analysis of existing question typologies What is X? (and Y etc)? How many? How much? How similar or different are X and Y (and z etc)? How do X and Y fit together structurally? How do X and Y fit together procedurally? How do X and Y influence each other? What are peoples situations and strategies? What do we know? And, how do we know it?

(b) Analysis of social science questions 70 articles spanning 16 social science categories used to: 1)Collect sample of social science questions 2)Develop typology of research questions

(b) Source of social science questions ESRC Social Science Category Number of Journal Articles Area Studies and Development Studies 5 Economic and Social History 5 Economics 5 Education 5 Human Geography 5 Interdisciplinary studies of Science, Technology and Innovation 4 Linguistics 4 Management and Business Studies 4 Planning, Environmental Studies & Housing Studies 4 Political Science, International Studies and International Relations 4 Psychology & Cognitive Science 4 Social Anthropology 4 Social Policy, Social Work and Health Studies 4 Socio-legal Studies and Criminology 5 Sociology 4 Statistics, Research Methods & Computing 4

(b) Social Science Questions RESEARCH ITEMS: e.g. variables, concepts Single itemMultiple itemsRelated items RESEARCH QUESTIONS What is it?How much? How many? How different?How related? PURPOSE: To GenerateExploreTest KNOWLEDGE

(c) Analysis of diverse reviews Purposive sample of diverse reviews –EPPI-reviews process, impact, agency and meaning –Dixon-Woods process, structure, impact, agency and meaning –WWW cause Compared questions and methods –Eligible studies, sources of studies, search methods

Question led research activity QuestionsResearch activities What is this/ are these? Describe: characterise, categorise, typologies How many/ much? Measure: frequency and magnitude How similar or different? Compare: descriptions and measures How do they fit together structurally? Analyse: structures How do they fit together procedurally? Analyse: processes How do they influence each other? Analyse: influences What are peoples situations and strategies? Analyse: agency and meaning What do we know? Synthesise: studies And how do we know it? Study: methodology

Questions, research activity and items QuestionsResearch activities What is this/ these items? Describe: characterise, categorise, typologies How many/ much? Measure: frequency and magnitude How similar or different? Compare: descriptions and measures How do they fit together structurally? Analyse: structures How do they fit together procedurally? Analyse: processes How do they influence each other? Analyse: influences What are peoples situations and strategies? Analyse: agency Research Items Single item Related Items Multiple Items What do we know? Synthesise: studies And how do we know it? Study: methodology

Questions, research activity and items QuestionsResearch activities What is this/ these items? Describe: characterise, categorise, typologies How many/ much? Measure: frequency and magnitude How similar or different? Compare: descriptions and measures How do they fit together structurally? Analyse: structures How do they fit together procedurally? Analyse: processes How do they influence each other? Analyse: influences What are peoples situations and strategies? Analyse: agency Research Items Single item Related Items Multiple Items What do we know? Synthesise: studies And how do we know it? Study: methodology

Questions, purpose, research activity and items QuestionsResearch activities What is this/ these items? Describe: characterise, categorise, typologies How many/ much? Measure: frequency and magnitude How similar or different? Compare: descriptions and measures How do they fit together structurally? Analyse: structures How do they fit together procedurally? Analyse: processes How do they influence each other? Analyse: influences What are peoples situations and strategies? Analyse: agency Research Items Single item Related Items Multiple Items What do we know? Synthesise: studies And how do we know it? Study: methodology Purpose: To Test Explore Generate Theory

Implications for searching Questions Process/ structure Influence: Cause Influence: Impact Agency

Implications for searching Questions Eligible studies Study sourcesSearch method Process/ structure Influence: Cause Influence: Impact Agency

Implications for searching Questions Eligible studies Study sourcesSearch method Process/ structure Influence: Cause Influence: Impact Agency Qualitative design Academic literatureElectronic, contacts & snowballing

Implications for searching Questions Eligible studies Study sourcesSearch method Process/ structure Influence: Cause Influence: Impact Quantitative design Academic literatureExhaustive: Electronic, contacts & snowballing Agency Qualitative designAcademic literatureElectronic, contacts & snowballing

Implications for searching Questions Eligible studies Study sourcesSearch method Process/ structure Influence: Cause Quantitative design Academic literatureElectronic, contacts & snowballing Influence: Impact Quantitative design Academic literatureExhaustive: Electronic, contacts & snowballing Agency Qualitative designAcademic literatureElectronic, contacts & snowballing

Implications for searching Questions Eligible studies Study sourcesSearch method Process/ structure Mixed designPublic services, academic literature Contacts, electronic & snowballing Influence: Cause Quantitative design Academic literatureElectronic, contacts & snowballing Influence: Impact Quantitative design Academic literatureExhaustive: Electronic, contacts & snowballing Agency Qualitative designAcademic literatureElectronic, contacts & snowballing

Implications for searching Questions Eligible studies Study sourcesSearch method Process/ structure Mixed designPublic services, academic literature Contacts, electronic & snowballing Influence: Cause Quantitative design Academic literatureElectronic, contacts & snowballing Influence: Impact Quantitative design Academic literatureExhaustive: Electronic, contacts & snowballing Agency Qualitative designAcademic literatureElectronic, contacts & snowballing

Conclusions Reviews addressing different questions (impact, cause, process, agency) have sought different types of studies Reviews testing theory have employed exhaustive searches to avoid a biased answer Some reviews generating or exploring theory also use exhaustive searching Does exhaustive searching benefit reviews generating or exploring theory?