1. Research Agenda Workshop
[Insert relevant information. Consider:
Alliance or group name
Facilitator’s name(s)
Date
Location]
[Customize this slide with information relevant to your workshop. Have this slide up on the screen as participants enter the room]

2. Welcome and purpose Workshop objectives:
Engage in a collaborative process
Identify alliance research priorities
Develop a set of research questions
Develop a coherent research agenda for the next 3–5 years
Welcome to the Research Agenda Workshop. We look forward to a productive day. Our purpose for coming together today is to engage in a collaborative process where we identify research priorities for our alliance/group and develop a set of research questions aligned to these priorities. Together these will comprise a coherent research agenda that will guide our work in the next 3–5 years.

3. Session goals Morning: Afternoon: Review different types of research
Identify and prioritize research topics
Generate related research questions
Afternoon:
Refine and prioritize research questions to form initial research agenda
Our work today will focus on three things:
Reviewing different types of research, types of evidence, and types of research agendas
Identifying and prioritizing possible research topics
Generating, refining, and prioritizing research questions

4. Agenda Introduce participants
Identify possible priority research topics
Investigate types of research and levels of evidence
Explore research questions and research agendas
Prioritize research topics
Generate research questions
Share, refine, and prioritize research questions
Put it all together into a research agenda
[Facilitator reviews agenda for the workshop]

5. Introductions Name Where you work Your role
Let’s go around the room and introduce ourselves. Please share your:
Name
Where you work, and
Your role

6. Alliance goal
[Insert your alliance or group’s goal. For example: The USVI College and Career Readiness Research Alliance will support the Virgin Islands Department of Education in their efforts to prevent and reduce the number of students dropping out of schools by providing applied research and analytic technical support on how best to use available data to both establish robust early warning systems and identify interventions to help improve outcomes for students at risk.]
Let’s start by reviewing our Alliance Goal/Mission/Purpose [facilitator reads goal]

7. Alliance goal and research topics
Record in your workbook on page 5:
Topics that fit under your alliance’s goal
If you are unsure of a topic, include it in the “possible” column
Focus on topics that are most important to you.
You will have five minutes to think and write by yourself.
Our first task today is identify some possible research topics that are relevant to our alliance’s goal. We are going to first work individually and then in small groups.
First, turn to page 6 of your workbook; working on your own, take five minutes to brainstorm topics that are potentially relevant to the alliance goal. You will see the worksheet on page 6 is divided into two columns; use the left column to list topics that you think fit under the alliance’s goal; use the right column to list topics that you think might fit under the alliance goal.
Don’t worry right now about the “grain” size of your topics/ideas—we will revise and narrow the topics later.

8. Small-group discussions
Break up into small groups
On large Post-It notes, write down the topics your group would like considered as priority topics
One topic per Post-It note
Now in small groups, we will share and discuss the topics we came up with individually, thinking about such things as relevance and importance. Again, you don’t need to worry at this point about the grain size of the topic; it’s okay if some are narrow and some are broad.
Group members should write down topics on the large Post-it notes. These will be shared with the whole group, which will then group the topics.

9. Grouping our topics Affinity grouping process.
What topics generated in the groups go together?
Review the topic groupings and name them.
Items grouped under each topic name are now considered subtopics.
[Bring the small groups back together, have participants stick the Post-It notes on the wall, and in whole group review the topics. This is not a discussion but simply a review of what’s written on the post it notes.]
Together, we will group our topics using an affinity grouping process; everyone will have a say. Now that we’ve had a chance to review the topics listed, does anyone have ideas about which topics are “like” or kind of go together? Do any seem to be the same thing? Can someone help me by moving the Post-it notes around per the discussion?
Let’s see how many topic groups we have. Do we have any that don’t seem to go with anything else? That’s okay, they can be a group by themselves.
Before we continue, let’s review our grouping and see if we need to make more changes to it.
[When satisfied, we can now name our topic groupings]
Now that we have grouped our topics, let’s “name” the groups –this will be the “topic” and the items listed in the group will be the “subtopics.”

10. Types of research Typology used by the REL program at IES includes:
What’s Happening
Descriptive
What’s Known
Making Connections
Correlational
Making an Impact
Impact/causal
For the next little while, we will step back from the particular topics of interest to our alliance and agenda, and focus on research in general, as well as research questions and agendas in general. Don’t worry, we will come back to prioritize our topics in just a little bit.
First, here’s a brief overview of different types of research. There are many ways of organizing types of research. Here we are using the REL language and typology for research and research products. One type of research is not necessarily better than the other, but different types of research allow us to answer different types of questions and with more or less certainty.
We think it’s important to review information on different types of research because it may inform the types of questions we ask or possibly the sequence of our work. Note that more information than we will cover is in your workbook.

11. What’s Happening
Descriptive studies look at what is happening: trends; baselines; and experiences of individuals, groups, or programs
Methods include secondary data analysis and document and records review
The studies use descriptive statistics: averages, frequencies, and percentages
First is Descriptive Research. The REL program describes descriptive research by the product category of What’s Happening.
This type of research examines baselines, trends, and may look at experiences or characteristics of individuals, groups or programs. This type of research DOES NOT inform cause and effect but it can be a useful first step in a larger research agenda or set of questions, and it can inform hypotheses that can be tested using more rigorous methods.
Descriptive studies can use primary or newly collected data, or secondary data analysis from existing data sets, and even document review.

12. What’s Known
Another type of descriptive study focuses on reviews of the literature and other previous research. Examples include:
Literature reviews
Meta-analyses
What Works Clearinghouse reviews
The What Works Clearinghouse is an IES-funded initiative that provides reviews of existing education research and evaluates the credibility and reliability of the research evidence.
Another type of descriptive research focuses on reviews of literature and reviews of other research, known as a meta-analysis.
The REL program classifies research conducted in this category as “What’s Known.”
The What Works Clearinghouse, another program of IES, provides these types of reviews. Established in 2002, the WWC is a resource for informed education decision-making. The WWC provides reviews of existing education research and identifies studies that provide credible and reliable evidence of the effectiveness of a given practice, program, or policy (referred to as “interventions”). Summary information and reports, including REL reports, are available on the WWC website.
With over 700 publications available and more than 6,000 reviewed studies in the online searchable database, the WWC aims to inform researchers, educators, and policymakers as they work toward improving education for students.

13. Making Connections
Correlational studies look at the relationships among two or more variables or characteristics but do not imply that one causes another.
Correlational research tests whether relationships between variables are “statistically significant,” meaning they are not likely due to chance
Methods include analysis of existing data from administrative or other state, district, or school datasets.
Another type of research is correlational research. It’s about connections, but not causality. This type of research explores the relationships and trends between two or more variables. For example, a correlational study may look at how students’ math course-taking patterns vary by race and gender. While the results may show that course-taking patterns differ by race and gender, they cannot prove what causes the differences.
Correlational research can use primary or secondary data analysis. In education research, we frequently see secondary data analysis of school, district, or state data.
Statistical techniques test for relationships between two or more variables or differences between two or more variables. These techniques include: difference in means testing, Analysis of Variance (ANOVA), and regression techniques.

14. Making an Impact
Causal/impact studies usually examine questions about impact; they can build on descriptive studies
Look for “opportunistic” possibilities
An impact study:
Examines the effectiveness of a particular policy, program, or practice
Can be small or large
Can be used in a formative way (for example, a short-term pilot study) or a summative way
The fourth REL product category is called “Making an Impact. ” These types of products are reserved for the most rigorous types of research, which allow researchers to explore questions of cause and effect, impact, and effectiveness.
Many educational practices—such as the piloting of new curriculum or program—provide opportunities for this type of research.
Impact studies examine the effective of educational policies, programs, or practices. They can be small-scale or large-scale, and can be used for formative or summative purposes.

15. Making an Impact
Only certain types of designs can assert causal claims or claims about program impact
REL impact studies must meet What Works Clearinghouse standards
Randomized controlled trials (RCTs) = the “gold standard”
Other designs, including quasi-experimental and matched comparisons, may not yield unbiased estimates of impact
Causal or impact research designs require the use random assignment to treatment and control groups; this allows researchers to be more confident that any resulting differences between groups are not due to other factors. One generally embarks on this type of rigorous research only after extensive descriptive research.
On page 11 in your workbook, you can find a table titled The Continuum of Rigor in Impact Evaluation Designs. This is excerpted from a guide on evaluating educational reforms and provides information on and examples of the different types of impact study designs, including their level of rigor.

16. Investigating research examples
Review the research summary you read during prework (5 minutes)
Divide into small groups by summary and discuss (15 minutes)
What the research questions were
What data sources were used
What the research design was
What else there is to learn
What you would need to answer the remaining questions (data, RCT, etc.)?
Whole-group discussion (10 minutes)
Now that we’ve had a primer on types of research and designs, let’s dig a little deeper into some research examples we read as part of preparation for the workshop.
First, you will individually review the research summary that you read during pre-work. This will take about 5 minutes.
Then, we will divide the large group into small groups according to the study summary you read and have you discuss the studies.
Finally, we will have a brief whole group discussion about what you noticed about the research designs, questions, data and such from the studies.
You may want to use the Investigating Research Examples organizer on page 3 of your workbook to assist in organizing your thoughts about the research study.
[After first 5 minutes, divide up into groups, for another 15 minutes]
[When whole group comes back together prompt discussion by asking such things as – what was your study about? What did you note about the research questions or design? What were some of the strengths and weaknesses of the approach?]

17. Break
15 minutes

18. Researchable questions
What is a researchable question?
Reasonable
Appropriate
Answerable
Specific
Where do researchable questions come from?
Questions, concerns, and values of stakeholders
Important issues in the field or research literature
Professional standards or guidelines
Views and knowledge of experts
One’s own views and judgment
Now that we’ve reviewed some research study summaries, we have a sense of what research questions are. Earlier today, we identified some possible research topics; a research topic is a general statement about the research interest (e.g., the impact of early education on later achievement).
A research question, by contrast, is more specific and gives us an idea about the method or approach used and what exactly was studied. Descriptive and correlational questions tend to be “How” or “What is” questions, whereas impact or causal questions tend to ask “Does…” For example, “Does participating in this program change behavior or increase knowledge?”
Some guidelines about research questions include:
They are reasonable – That is, exploring the research question is do-able, given time and budget
They are appropriate – The research question or questions fits with the program or issue being studied; in this case, they questions fit with the alliance goals or purpose
They are answerable – This means appropriate data can be gathered or found that will actually answer the question or questions.
They are specific – This means they include clearly defined and measurable indicators of success or the desired outcome.
Research questions are more difficult to come up with than topics. And you might not immediately know if your questions are answerable or measureable. Sometimes, we need to do preliminary work or investigating to determine if a question is researchable.
Research questions come from many places–
Issues or questions of stakeholders or the community
Previous research or the literature
Views of experts
Professional standards and guidelines
Your own experiences and views

19. Research agendas What is a research agenda?
Identifies research priorities and questions for group
Is coherent
Leads to rigorous and relevant research that is actionable
May include current or future project
What does a research agenda look like?
Linear
Topical
For our purposes here, we define a research agenda as 2 to 4 focused research topics each with a set of 3 to 5 coherent research questions that will help to achieve the alliance’s goal. A research agenda also can include a set of possible studies or projects that align with the research questions.
Essentially, a research agenda identifies the research priorities for the group and can be specific enough to include questions and possible studies that provide a direction or path for the group’s work.
There is no right way to organize an agenda. Research agendas can be linear or topical. We include a few examples on the next slides and in your workbook.

20. Example: Research agenda that Is less coherent
Topic: College readiness
Research questions:
What are the high school dropout rates for particular subgroups of students?
Does coaching high school English teachers impact their students’ performance?
Studies:
Descriptive study of high school dropout rates for key subgroups
Descriptive study of policies for assigning students to college math versus general math
RCT of the impact of classroom coaching for new high school English teachers in five large urban districts
This first example is of a LESS coherent research agenda. The topic is College Readiness, but as you can see the identified studies are only tangentially related and don’t really inform each other in any way.

21. Example: Coherent, linear research agenda
Topic: High school graduation
Research questions:
What are the high school dropout rates for key subgroups of students?
What are the effects of alternative high school programs on students’ degree completion and graduation?
Studies:
Descriptive study of high school dropout rates for key subgroups
Descriptive study of re-enrollment rates in traditional high schools and education trajectories of re-enrollees
RCT of the impact of alternative programs for degree completion on high school graduation
This is an example of a coherent linear agenda. This agenda is linear because the studies build on one another, with one being completed before the next is undertaken.

22. Example: Coherent, topical research agenda
Topic: Mathematics learning
Research questions:
What is the impact of grade 8 students’ access to algebra I on their math achievement?
How do students with disabilities perform in math?
Studies:
RCT of the impact of student access to algebra I in grade 8
Descriptive study of math education practices for students with disabilities
Descriptive study of math performance patterns for students with disabilities
Research agendas also can be coherent and arranged more topically rather than sequentially or linear, as this example indicates.

23. Prioritizing research topics
Narrow the list of research topics to 24 priority topics for our alliance, using a modified Focus Four:
Brainstorm
Clarify
Advocate
Canvass
Now that we’ve reviewed different types of research, and learned about research questions and agendas, we are going to return to our alliance focus and the possible research topics that we identified earlier this morning. Specifically, we are going to narrow the list of topics to identify two to four PRIORITY topic areas for our alliance. We will get to these priority topics using a modified Focus Four process. The Focus Four includes four steps:
Brainstorm the topics; we have already done this
Clarify – clarify the topics by asking questions, reviewing our affinity groupings, and making any necessary changes
Advocating – participants have an opportunity to advocate, in a brief positive way, for a topic. We are not arguing against topics!
Canvass the group to see where interest lies.
Let’s begin with clarifying since we have already brainstormed our topics.
[Facilitator works through clarifying and advocating with the participants. When it comes time to canvass there are different ways to do it.
The topics should be posted around the group or listed on chart paper. Give each participant three ‘sticky’ dots to use to indicate their preferences. Ask participants to place sticky dots next to their topic or topics they believe should be the alliance’s research priority. Note that they can spread their dots among 2 or 3 topics or even use all three on one topic.]
[After canvassing, take a few minutes to review the results and identify the top-rated topics and the subtopics (if any) that fall under the topic. These will be the topics for which the group will generate research questions.]

24. Example: Gender in Schools Research Alliance
Alliance goal: Provide research that informs and promotes gender equity in schools
Topic
Subtopic
Questions
Timing
Girls in STEM
Tracking achievement differences
Encouraging interest and enrollment in STEM
Persistence in STEM majors
Attainment for boys
Structured Inequality
Now that you have identified your priority research topics, we will move to generating research questions for the topics. We have already reviewed the characteristics of good research questions, but we also want to quickly review how one can move from topics to subtopics to research questions.
This slide shows an example of a research alliance focused on gender in schools. Some of the priority topics this group identified include Girls in STEM, Attainment for Boys, and Structured Inequality. These topic areas are pretty broad and could lend themselves to many different foci and research paths. So we’ve identified subtopics. The subtopic examples provided here fall under the Girls in STEM topic. 24

25. Moving from subtopics to questions
This slide shows specific questions that were developed for the Girls in STEM subtopics. You can see that the subtopics each have multiple questions and different types of questions, with different levels of specificity.

26. Moving from subtopics to questions
A fully specified research question
Subtopic
Questions
Timing
Tracking achievement differences
What are gender differences in STEM achievement in K12 and how have they changed over time?
Do gender differences in STEM achievement vary among districts and schools?
Encouraging interest and enrollment in STEM
What can teachers do?
Are single-sex schools and classrooms better?
What programs help promote STEM for girls?
Persistence in STEM majors
A general question that can be used to develop a set of more specific research questions
Here we see the same subtopics and questions. While all are “research questions,” they represent different types of research questions. Some are fully specified research questions and some are more general.
We don’t expect the questions generated today to be necessarily fully specified. General research questions will be sufficient for us to work with to develop a set of fully specified questions. As we move to generating questions for our research topics, we don’t want you to get hung up on developing fully specified questions; if you come up with some great, but if not, the general research questions will be helpful in later development of more specific questions.

27. Generating research questions
In pairs, work for 30 minutes to generate research questions related to identified priority research topic(s).
Think about:
What are some possible research questions for these topics and subtopics?
What data would be needed to research these questions?
Now we will dive into our priority topics and begin to generate research questions. We will work in pairs to generate research questions for one of the priority research topics or the subtopics within a topic. You may want to use the organizer on page 19 of your workbook to take notes.
In pairs, discuss potential research questions that the topics and subtopics point to. Use the chart paper to keep track of the questions you develop. We will work in pairs for 30 minutes. Save some time at the end to list the questions you generated on newsprint. We will be sharing the newsprint in the next segment.

28. Lunch break
45 minutes
[Before going off for lunch have pairs post their newsprint of questions around the room.]

29. Sharing questions Post newsprint with questions from each group
Review the posted newsprint and consider:
What would you change or edit?
What would you delete?
What would you add?
Briefly discuss with a partner
The morning was fast paced and intense but we’ve done great work. Let’s take 10 minutes to walk around the room looking at the questions posted on the newsprint that each pair generated. You might want to take notepaper and a pen with you. As you walk around think about the questions and consider, what changes might you make to questions? Are there any that you don’t think are necessary? Anything missing?
After we review the questions on newsprint, we’ll discuss our observations briefly with a partner, and then we’ll have opportunities to revise the questions .

30. Refining questions Whole group (small groups if whole group is large):
Edit, add, delete, and combine questions as necessary
What kinds of studies would result from the questions (descriptive, correlational, impact evaluation)?
In pairs, discuss:
What is important to you?
What research topic and questions can provide a meaningful research agenda?
Together we will review and refine the questions so that we have a good list from which we can identify our research questions and create a research agenda.
For 15 minutes discuss:
What did you think of the questions?
What suggestions do you have for editing the questions?
Are there any that are essentially the same and can be grouped together?
Are there any that should be added? Deleted?
[Facilitator or note-taker edits the questions, so that at the end of this segment you have a list of questions to use for canvassing.]
For 5 minutes, turn to a partner and discuss what questions are important to the alliance’s research agenda and why.

31. Prioritizing research questions
Prioritize the research questions in each priority topic area
Focus Four process
Brainstorm
Clarify
Advocate
Canvass
Our list contains more questions than our alliance could realistically focus on so we will prioritize our research questions to form a research agenda. We will run through this process separately for each of the priority research topics for which we generated questions.
We will again use a modified Focus Four approach.
We have already brainstormed and clarified.
Let’s take a few minutes for the advocating step. Let’s hear from you about questions that you think should be part of the research agenda.
Let’s now canvass the group to identify the 3 to 5 questions for that topic area.

32. Preliminary research agenda
Review identified priority topics and questions
Any surprises?
How many top-rated questions should be on the agenda? Consider:
Alliance goal
Types of research
Sequence
Short- and long-term nature of questions
Let’s review the result of our canvassing.
[Read out the questions in order of identified priority.]
Any surprises?
Lets think about how many questions we should include on our agenda per topic. Thoughts?
Do we have a good mix of study types? What is the best sequence for this work?
Let’s look now at the Research Agenda Template found on page 23 of the Workbook.

33. Research agenda template
Turn to research agenda template on page 19
Use template to organize research agenda
Document and summarize the main research topics and subtopics identified, along with research questions
Let’s use the template to summarize our work today.
We can fill in the main topics and the subtopics that we prioritized.
Then we can list the top 3–4 questions for each area that we just prioritized.
[If there is not time to complete this template together at the workshop, the facilitator should show the template, and note he/she will complete it based on the workshop notes and will share it with participants in the near future.]

34. Next steps Summary document from workshop
Iterative review and refinement of agenda
Agenda as a “living document”
Collaborative work to:
Identify appropriate existing data or data collection for studies
Identify existing research relevant to agenda priorities and questions
Develop study designs
Conduct research or identify research support
We’ve done a lot today. We have identified our priority research topics and the main questions we have for each of these topics; together these comprise a coherent research agenda that will guide our work. However, we should not see the agenda as static. We will want to revisit it over time, especially as new research and information becomes available.
Now comes the hard but also fun part: developing a study or identifying research supports to carry out the studies indicated in our agenda.

35. Thank you For more information contact:
[insert contact information or customize with any special closing information]