1. ies.ed.gov Connecting Research, Policy and Practice 84.305L: Low-Cost, Short-Duration Evaluation of Education Interventions 84.324L: Low-Cost, Short-Duration Evaluation of Special Education Interventions Phill Gagne Allen Ruby National Center for Education Research Kimberley Sprague National Center for Special Education Research

2. ies.ed.gov Overview Overview of IES and its mission Requirements Specifics – Purpose – The project narrative Significance Partnership Research Plan Personnel Resources Other important sections of the application Preparing and submitting an application 2

3. ies.ed.gov Legislative Mission of IES Describe the condition and progress of education in the United States Identify education practices that improve academic achievement and access to education opportunities Evaluate the effectiveness of Federal and other education programs 3

4. ies.ed.gov Organizational Structure of IES 4 National Board for Education Sciences Standards & Review Office Office of the Director National Center for Education Evaluation National Center for Education Statistics National Center for Education Research National Center for Special Education Research

5. ies.ed.gov IES Grant Programs: Research Objectives Develop or identify education interventions (i.e., practices, programs, policies, and approaches) – that enhance academic achievement – that can be widely deployed Identify what does not work and thereby encourage innovation and further research Understand the processes that underlie the effectiveness of education interventions and the variation in their effectiveness 5

6. ies.ed.gov Partnerships & IES Priorities IES seeks to... Encourage education researchers to develop partnerships with stakeholder groups to advance relevance of research and usability of its findings for day- to-day work of education practitioners and policymakers Increase capacity of education policymakers and practitioners to use knowledge generated from high quality data analysis, research, and evaluation through wide variety of communication and outreach strategies (See http://ies.ed.gov/director/board/priorities.asp)http://ies.ed.gov/director/board/priorities.asp 6

7. ies.ed.gov Short Description Carried out by Partnerships – New or established – Minimum: research institution and a state or local education agency Purpose – Carry out rigorous evaluations of education interventions implemented by state or local education agencies High importance to the education agency Use secondary data (e.g., administrative data) Low-cost: maximum grant of $250,000 Short-duration: 2 years

8. ies.ed.gov Impetus for Low-Cost Grant Program Take advantage of opportunities to use administrative data to do evaluations Provide useful information to education agencies in a more timely manner than traditional evaluations Create additional opportunities for research institutions and education agencies to work together Identify the strengths, weaknesses, and applicability of this type of evaluation

9. ies.ed.gov General Requirements Focus on student education outcomes – 84:305L: For students from prekindergarten through postsecondary and adult education – 84.324L: For infants/toddlers through students in grade 12 with or at-risk for disability Research occurs in an authentic education setting Evaluate education interventions using secondary data Partnership between research institutions and state and local education agencies Disseminate findings in ways useful to agency decision-making

10. ies.ed.gov Student Population 84.305L: Students from prekindergarten through postsecondary and adult education 84.324L: Students from infants/toddlers through grade 12 with or at-risk for disability – A student with a disability is defined in Public Law 108-446, the Individuals with Disabilities Education Improvement Act of 2004 (IDEA) – Additional requirements for identifying students at risk for developing a disability – see http://ies.ed.gov/ncser/definition.asphttp://ies.ed.gov/ncser/definition.asp 10

11. ies.ed.gov Focus on Student Education Outcomes Research must address education outcomes of students. For both 305L and 324L these include – Academic outcomes – Social and behavioral competencies that support student success in school For 324L, these also include – Developmental, functional, and transitional outcomes for students with or at-risk for disability 11

12. ies.ed.gov Student Outcomes Age/GradeOutcome Infants/Toddlers 324LDevelopmental outcomes pertaining to cognitive, communicative, linguistic, social, emotional, adaptive, functional or physical development. Prekindergarten 305L & 324L 324L School readiness (e.g., pre-reading, language, vocabulary, early math and science knowledge, social and behavioral competencies) Developmental outcomes 12

13. ies.ed.gov Student Outcomes GradeOutcome K - 12 305L & 324L 324L Learning and achievement in reading, writing, mathematics, and science; Progress through the education system (e.g., course and grade completion or retention, high school graduation, and dropout); Social and behavioral competencies important to academic and post-academic success. Functional outcomes that improve educational results; Transitions to employment, independent living, and postsecondary education. 13

14. ies.ed.gov Additional 305L Student Outcomes GradeOutcome Postsecondary (Grades 13 – 16) (baccalaureate and sub-baccalaureate) Access to, persistence in, progress through, and completion of postsecondary education; for students in developmental programs, additional outcomes include achievement in reading, writing, English language proficiency, and mathematics; success in gateway math and science courses, introductory English composition Adult Education (Adult Basic Education, Adult Secondary Education, Adult ESL, and HS equivalency preparation) Student achievement in reading, writing, English language proficiency, and mathematics; access to, persistence in, progress through, and completion of adult education programs 14

15. ies.ed.gov Education Interventions The wide range of education curricula, instructional approaches, professional development, technology, and practices, programs, and policies that are implemented at the child/student, classroom, school, district, state, or federal level to improve student education outcomes The intervention is of high importance to the SEA or LEA The implementation of the intervention is managed or overseen by the SEA or LEA (not just allowing a researcher or organization to implement the intervention) Implementation of the intervention must occur in Year 1 of the project The Intervention is expected to produce meaningful improvement in student education outcomes within a short period (e.g., within a quarter, semester, or year) At a minimum, the administrative data (or other secondary source) contains student education outcomes (primary data collection not supported by grant)

16. ies.ed.gov Applications must be from a Partnership Applications must include at least one Principal Investigator (PI) from a research institution and at least one PI from a U.S. state or local education agency – PI from research institution: Must have the ability and capacity to conduct scientifically valid research and expertise in the education issue to be addressed – PI from state or local education agency: Must have decision-making authority for the issue within his or her agency 16

17. ies.ed.gov Partnership Partnership may be new or existing – Expected to continue through the project and, perhaps, afterward Research institution has a broad definition – Ability and capacity to conduct scientifically valid research 17

18. ies.ed.gov Partnership: SEAs State education agencies – Examples: education agencies, departments, boards, commissions – Oversee early learning, elementary, secondary, postsecondary and/or adult education – For 305L, oversee infant and child care, and/or early intervention services – Also includes education agencies in District of Columbia, the Commonwealth of Puerto Rico, and each of the outlying areas

19. ies.ed.gov Partnership: LEAs Local education agencies which are primarily public school districts County or city agencies that have primary responsibility for infant and child care (324L), early intervention services (324L), or prekindergarten Community college districts Tribal education agencies State and city postsecondary systems – Individual postsecondary institutions cannot serve as a partner – A postsecondary system that applies as an education agency partner cannot also serve as the research institution partner in the same project Adult education providers (defined under WIOA) can serve as the partner when there is no state or local education agency for adult education 19

20. ies.ed.gov Additional Partners Partnerships may include more than one state or local education agency if they share similarities and interests Intermediary/service districts that provide services to multiple districts but do not have decision-making authority over implementing programs and policies, Non-education state and local agencies may be partners as long as an education agency is a partner Partnerships may include more than one research institution if they have shared interests and make unique contributions Partnerships may include other non-research organizations (e.g., issue-oriented or stakeholder groups) that will contribute to the partnership and its work 20

21. ies.ed.gov Dissemination/Products Projects are to aid education agencies in decision-making Required dissemination – Oral briefing on results to education agency – Written brief, written for non-technical audience, made available free to public Recommended dissemination – Partner presentations to academic and practitioner audiences – Partner publications in academic and practitioner journals – Toolkit or guide for other education agencies on how to conduct a similar study

22. ies.ed.gov Check the Fit of Your Research and Low-Cost, Short-Duration Evaluation Grant Program If you are not looking at student outcomes, then IES is not the appropriate funding agency If you need time and effort to build a partnership and prepare for an evaluation, consider: – Researcher-Practitioner Partnerships under 84.305H If the intervention you want to evaluate – Is not implemented by a state or local education agency, – Cannot be evaluated using secondary data, – Will not be implemented in Year 1 of the project, or – Is not expected to improve student outcomes within a short period (e.g., a quarter – a year). – Then consider the: Education Research Grants Program (84.305A) or Special Education Research Grants Program (84.324A)

23. ies.ed.gov Low-Cost Evaluation: Purpose Promote joint evaluation research by research institutions and state and local education agencies – On an education intervention identified as having great importance by the education agency – That includes practitioner input into the research – That will provide timely rigorous evidence for the agency’s decision-making regarding the intervention – And the results of which will be broadly disseminated in ways easily accessible to researchers, practitioners, and the public 23

24. ies.ed.gov What should the partnerships do during the grant? Identify a specific education intervention Implemented by an SEA or LEA in Year 1 of project Of high priority to that agency Intended to improve student education outcomes within a year Carry out an evaluation of that intervention – Using a RCT or an RDD design (or a SCD under 324L) – Using secondary data – Estimate overall impacts – If data available: Estimate subgroup impacts for important subgroups Examine other moderators and mediators of interest, fidelity of implementation, and comparison group practice 24

25. ies.ed.gov Expected Products of the Grant Causal evidence of the impact of a clearly specified intervention implemented by an SEA or LEA – Overall impacts – Impacts for available subgroups of interest Advice for the SEA or LEA – Continuing and/or expanding the use of the intervention – Further research needs, e.g., Evaluation, e.g., variation in impacts, moderation and mediation, generalizability, replication Development, e.g., modifications to the intervention or its implementation 25

26. ies.ed.gov The Project Narrative (Maximum of 15 Pages) Significance Partnership Research Plan Personnel Resources 26

27. ies.ed.gov Significance The education intervention to be evaluated – The education problem/issue the intervention is to address within the SEA/LEA Relevance to other SEAs or LEAs (secondary importance) – Components of the intervention – Rationale for why the intervention can improve student outcomes within a short period (e.g., 1 quarter - 1 year) May include theory of change Difference from status quo Related findings from previous studies and how this study will improve upon past work

28. ies.ed.gov Significance The implementation of the intervention – Who will implement it and how will it be implemented The education agency will implement or will oversee implementation – Adequate funding available for implementation – Implementation during Year 1 of the project at a level expected to impact student outcomes Sources of secondary data to be used in the evaluation – How these data are collected – How these data will be obtained by researchers by the 1 st quarter of Year 2 of the project

29. ies.ed.gov The Project Narrative (Maximum of 15 Pages) Significance Partnership Research Plan Personnel Resources 29

30. ies.ed.gov Partnership Describe the partners – The research institution and the education agency Offices or divisions within the agency whose cooperation is necessary – Any other members of the partnership – Common interest in and benefit from this evaluation – The process through which the partners determined the specific intervention to evaluate – Data sharing agreement – the strategy to obtain the secondary data and provide it for analysis by the 1 st quarter of the second year

31. ies.ed.gov The Project Narrative (Maximum of 15 Pages) Significance Partnership Research Plan Personnel Resources 31

32. ies.ed.gov Research Plan State research questions and hypotheses Describe sample and setting – Define population and how your sample and sampling procedures will allow inferences to the population – Exclusion and inclusion rules and their justification – Strategies used to increase participation and reduce attrition – Describe the setting and its implications for the generalizability of your study 32

33. ies.ed.gov Research Plan: Design Discuss how design will support causal inferences and identify potential threats to internal validity Discuss how degree of equivalence at baseline will be determined Discuss possibility of bias from overall and differential attrition 305L: Required use of RCT or RDD – Potential to meet WWC evidence standards without reservations 324L: Required use of RCT, RDD, or Single-Case Experimental Design 33

34. ies.ed.gov Research Plan: Design Randomized Controlled Trial (RCT) Note unit of randomization and justify choice Describe process for random assignment and maintaining its integrity Different Approaches to RCTs - Potential Issues – Entire population-mandatory: Treatment fidelity – Volunteers: Comparison group status – Lotteries: Attrition of non-accepted parties – Staggered roll out: Little time for true comparison – Variations of program/policy: Issue of overall significance 34

35. ies.ed.gov Research Plan: Design Regression Discontinuity Design (RDD) Appropriateness of assignment variable Show true discontinuity Discuss possibility of manipulation of design variable and analyses to determine such manipulation Sensitivity analyses to assess influence of key procedural or analytic decisions on results 35

36. ies.ed.gov Research Plan: Design Single-Case Experimental Design (324L Only) Justify the use of a single-case experimental design as opposed to an RCT or RDD (e.g., a focus on students with a low-incidence disability) Describe the repeated, systematic measurement of a dependent variable before, during, and after the active manipulation of an independent variable (i.e., intervention) Include outcome measures that are not strictly aligned with the intervention Describe any quantitative analytic techniques, in addition to visual analysis, for analyzing the resulting data (e.g., between-case effect size calculations) 36

37. ies.ed.gov Research Plan: Statistical Power Detailed description of power analysis – Justify method used to calculate power – Justify parameters used and assumptions made Provide power for main analyses and important subgroup analyses Along with identifying minimum detectable effect for your analysis, justify its – Reasonableness – Practical meaning Reviewers should be able to check power calculations 37

38. ies.ed.gov Research Plan: Outcome Measures Student education outcome measures relevant to states, districts, and schools – Found in administrative data or other secondary data Discuss reliability, validity, and appropriateness Must be collected during Year 1 of project – Additional data from previous years of intervention’s implementation may also be used if appropriate to the evaluation design Clearly link measures to rationale for the intervention 38

39. ies.ed.gov Research Plan: Optional Measures If available in secondary data, describe measures of – Intermediate outcomes – Moderators (subgroups expected) – Mediators (intermediate outcomes) – Fidelity of implementation – Comparison group practice

40. ies.ed.gov Research Plan: Analysis Detail impact analyses – Make clear how analyses directly answer your research questions – Show that analyses are based on the design – Address clustering of students in classrooms in schools – Address missing data – If multiple datasets are to be linked, detail how this will be done Describe any other analyses to be done (e.g., subgroups, other moderators, mediators, and fidelity of implementation) 40

41. ies.ed.gov The Project Narrative (15 Page Maximum) Significance Partnership Research Plan Personnel Resources 41

42. ies.ed.gov Personnel Identify all key personnel on the project team – The PI from the research institution who has previous experience carrying out the proposed evaluation design (RCT, RDD, or SCD) – The PI from the education agency who makes program decisions – Other key personnel Roles and responsibilities on the project – Each individual’s roles and responsibilities on the project – Their qualifications (i.e., expertise and experience) for their role – Their % FTE on the project – Past success at working in similar partnerships PI qualifications for managing a grant of this type Ensure objectivity of evaluation 42

43. ies.ed.gov Resources: To Conduct the Project Describe the institutional resources of all the institutions involved in the partnership and how these resources will contribute to building the partnership and to the research – Institutional capacity to manage the grant – Resources available at the partner institutions to support the project – Plans to acquire any major resources not yet in hand (e.g., secondary data) – Joint Letter of Agreement by partners (Appendix D) – Letter of Agreement to provide administrative data (App. D) 43

44. ies.ed.gov Resources: Dissemination of Results Results expected to be useful to the SEA/LEA partner and, perhaps, other SEA/LEAs – Findings of both beneficial impacts or no impacts Describe your capacity and resources to disseminate findings – Required dissemination through an oral briefing for the agency and a written brief freely available to the public – Dissemination to other audiences (e.g., researchers, policymakers, practitioners, students and their families, public) 44

45. ies.ed.gov Other Important Sections of the Application Appendix A Appendix B Appendix C Appendix D Budget & Budget Narrative 45

46. ies.ed.gov Appendix A Page Limit: 3 If you are resubmitting an application, use up to 3 pages to discuss how you responded to reviewer comments 46

47. ies.ed.gov Appendix B (Optional) Page Limit: 5 Figures, charts, or tables that supplement the project narrative Timelines for the project (very useful) Examples of instruments used in the collection of the administrative or other secondary sources of data Do NOT include narrative text

48. ies.ed.gov Appendix C (Optional) Page Limit: 5 Examples of materials used in the intervention: – curriculum materials – computer screen shots – training documents – assessment items – other materials Do NOT include narrative text 48

49. ies.ed.gov Appendix D (Required) No Page Limit Required Letters of Agreement – Joint Letter from the research institution and the SEA/LEA Document participation and cooperation in the partnership Set out each’s roles and responsibilities under the project – Letter from the office in charge of the agency’s data Project will have access to data required in time to do analysis Optional Letters of Agreement – Separate Letters from other organizations taking part – Letters from any consultants and schools taking part 49

50. ies.ed.gov Budget & Budget Narrative Maximum project length is 2 years Maximum award is $250,000 – Funds must be used for evaluation only (e.g., cannot be used for implementation of the intervention or primary data collection) – Award size depends on project scope Include a detailed budget form (SF 424) AND a budget narrative that links the activities, personnel, etc. from the Project Narrative to the funds requested 50

51. ies.ed.gov Preparing Your Application Important dates Information sources – Read the RFA – Talk with a program officer Review process

52. ies.ed.gov Application Deadline Letter of Intent Due Date Application Package Posted Start Dates August 4, 2016 4:30:00 PM DC Time May 19, 2016 July 1 to September 1, 2017 Important Dates & Deadlines 52

53. ies.ed.gov Information Sources Request for Applications – http://ies.ed.gov/funding/ http://ies.ed.gov/funding/ Abstracts of Projects – http://ies.ed.gov/funding/grantsearch/index.asp http://ies.ed.gov/funding/grantsearch/index.asp Application Package – www.grants.gov www.grants.gov Program Officers – Kimberley.Sprague@ed.gov 84.324L Kimberley.Sprague@ed.gov – Phill.Gagne@ed.gov 84.305L Phill.Gagne@ed.gov – Allen.Ruby@ed.gov 84.305L Allen.Ruby@ed.gov 53

54. ies.ed.gov Peer Review (Standards & Review Office) Compliance screening for format requirements Responsiveness screening for program requirements Assignment to review panel – 2 to 3 reviewers (substantive and methodological) – The most competitive proposals are reviewed by full panel Many panelists will be generalists to your topic Panels contain experts in relevant methodologies – Panel provides an overall score plus specific scores on Significance, Partnership, Research Plan, Personnel, and Resources 54

55. ies.ed.gov Notification All applicants will receive e-mail notification that the following information is available via the Applicant Notification System (ANS): Status of award Reviewer summary statement If you are not granted an award the first time, consider resubmitting and talking with your Program Officer 55

56. ies.ed.gov For More Information http://ies.ed.gov/funding Kimberley Sprague National Center for Special Education Research Kimberley.Sprague@ed.gov Phill Gagne Allen Ruby National Center for Education Research Phill.Gagne@ed.gov Allen.Ruby@ed.gov 56