“Opening the Doors of Policy-Making: Central Asia and South Caucasus” (UDF- GLO ) Skills Development Training for CSOs Istanbul, June 2-3, 2011 In-depth session: Research Design and Methods for Policy Analysts This presentation has been produced with the assistance of the United Nations Democracy Fund. The content of this presentation is the sole responsibility of Maria Golubeva and does not necessarily reflect the views of the United Nations, the United Nations Democracy Fund or its Advisory Board

From research question to research design: 1. The Purpose Do we want to describe how policy transfer and borrowing, policy formulation, policy implementation happens? Do we want to identify dependent and independent variables? (E.g. more social workers – less child labour) Do we want to compare policy or practice in several countries and see which is more effective/ equitable/ economically efficient? Do we want to evaluate the design or the implementation or the effect of a policy? Do we want to establish the direct and indirect costs and benefits of a policy? Do we want to assess the positions of stakeholders regarding a policy and identify stakeholders that are likely to support or oppose it?

Which purpose(s) does your policy project have? How does the research question you have chosen match these purposes? Which methods would you include in your research design?

Quantitative research ‘Empty yourselves of all emotions’

Quantitative research: surveys How do we choose the questions? Who gets to answer the questions? Who gets to enter and process the data?

Examples (undermining the illusion of objectivity) Deviant behaviour vs contribution to the economy Privatisation agenda in education Quality of sampling/ access to target groups

Qualitative research What can we actually find out with qualitative methods of social research? Regarding policy problems we want to identify (and their context in social reality)? Regarding existing policy programmes and legislation? Regarding stakeholders’ positions and interests?

From research question to research design: 2. Mix and Match E.g.: Do we need a qualitative study (document analysis, interviews, focus groups) before a survey? What do we hope to get out of it? Do we need in-depth qualitative studies as well as survey results? (e.g. case studies to illustrate how things happen) Do we need to double-check information obtained through structured interviews by doing desk research (analysis of policy documents and legal documents)?

From research question to research design: 3. How Participatory/ Advocacy oriented? Do we want the research design to affect the participants in a certain way (e.g. raise their awareness of the policy problem) or do we want the participants to affect the research design to reflect their needs or views? Do we see the research stage (data collection) as a phase of advocacy?

Group exercise Group 1: You are conducting research on the (assumed) linkages between drug trafficking and corruption in local councils in Kyrghizstan. How can you use the research procedures to raise awareness of the problem? Group 2: You are trying to find out whether teacher education in Armenia supports the goals defined by the national curriculum. How can you use the research procedures to help the teachers’ views have more impact on curriculum policies in Armenia?

What can we find out from a policy document? Assuming that it contains: Policy programme description and (possibly) budget

Absolutely essential components of a policy document Objectives Situation analysis Planned measures (Possibly) outputs (Possibly) outcomes

Some ‘softer’ suggestions Type(s) of policy discourse Sources of policy borrowing & transfer Type of policy culture

Formulating policy research conclusions Of course my research results are (reasonably) objective and obtained through ethical procedures, but... What do I want to change by raising this issue (desired impact)? What are the policy goals recognised as legitimate by my target audience(s)? What is the policy language they use? How do I bring the three together?