1. ies.ed.gov Connecting Research, Policy and Practice 84.305L: Low-Cost, Short-Duration Evaluation of Education Interventions 84.324L: Low-Cost, Short-Duration Evaluation of Special Education Interventions (FY 2016 Grant Competition) Robert Ochsendorf, Ph.D. National Center for Special Education Research Phill Gagne, Ph.D. Allen Ruby, Ph.D. National Center for Education Research

2. ies.ed.gov Overview Overview of IES and its mission General Requirements Purpose and Products Project Narrative – Significance – Partnership – Research Plan – Personnel – Resources Other important sections of the application Preparing an application 2

3. ies.ed.gov Legislative Mission of IES Describe the condition and progress of education in the United States Identify education practices that improve academic achievement and access to education opportunities Evaluate the effectiveness of Federal and other education programs 3

4. ies.ed.gov Organizational Structure of IES 4 National Board for Education Sciences Standards & Review Office Office of the Director National Center for Education Evaluation National Center for Education Statistics National Center for Education Research National Center for Special Education Research

5. ies.ed.gov IES Grant Programs: Research Objectives Develop or identify education interventions (i.e., practices, programs, policies, and approaches) – that enhance academic achievement – that can be widely deployed Identify what does not work and thereby encourage innovation and further research Understand the processes that underlie the effectiveness of education interventions and the variation in their effectiveness 5

6. ies.ed.gov Partnerships & IES Priorities IES seeks to... Encourage education researchers to develop partnerships with stakeholder groups to advance relevance of research and usability of its findings for day- to-day work of education practitioners and policymakers Increase capacity of education policymakers and practitioners to use knowledge generated from high quality data analysis, research, and evaluation through wide variety of communication and outreach strategies (See http://ies.ed.gov/director/board/priorities.asp)http://ies.ed.gov/director/board/priorities.asp 6

7. ies.ed.gov Short Description of Low-Cost Evaluation Projects Carried out by Partnerships – New or established – Minimum: research institution and a state or local education agency Purpose – Carry out rigorous evaluations of education interventions implemented by state or local education agencies High importance to the education agency Use secondary data (e.g., administrative data) Low-cost: maximum grant of $250,000 Short-duration: 2 years

8. ies.ed.gov Impetus for Low-Cost Grant Program Take advantage of opportunities to use administrative data to do evaluations Provide useful information to education agencies in a more timely manner than traditional evaluations Create additional opportunities for research institutions and education agencies to work together Identify the strengths, weaknesses, and applicability of this type of evaluation

9. ies.ed.gov General Requirements Focus on student education outcomes – 84:305L: For students from prekindergarten through postsecondary and adult education – 84.324L: For students from prekindergarten through grade 12 with or at-risk for disability Research occurs in an authentic education setting Evaluate education interventions using secondary data Partnership between research institutions and state and local education agencies Disseminate findings in ways useful to agency decision-making

10. ies.ed.gov Student Population 84:305L: Students from prekindergarten through postsecondary and adult education 84.324L: Students from prekindergarten through grade 12 with or at-risk for disability – A student with a disability is defined in Public Law 108-446, the Individuals with Disabilities Education Improvement Act of 2004 (IDEA) – Additional requirements for identifying students at risk for developing a disability – see http://ies.ed.gov/ncser/definition.asphttp://ies.ed.gov/ncser/definition.asp 10

11. ies.ed.gov Focus on Student Education Outcomes Research must address education outcomes of students. For both 305L and 324L these include – Academic outcomes – Social and behavioral competencies that support student success in school For 324L, these also include – Functional and transitional outcomes for students with or at-risk for disability 11

12. ies.ed.gov Student Outcomes GradeOutcome Prekindergarten 305L & 324L 324L School readiness (e.g., pre-reading, language, vocabulary, early math and science knowledge, social and behavioral competencies) Developmental outcomes K - 12 305L & 324L 324L Learning and achievement in reading, writing, mathematics, and science; Progress through the education system (e.g., course and grade completion or retention, high school graduation, and dropout); Social and behavioral competencies important to academic and post-academic success. Functional outcomes that improve educational results; Transitions to employment, independent living, and postsecondary education. 12

13. ies.ed.gov Additional 305L Student Outcomes GradeOutcome Postsecondary (Grades 13 – 16) Access to, persistence in, progress through, and completion of postsecondary education; for students in developmental programs, additional outcomes include achievement in reading, writing, English language proficiency, and mathematics; success in gateway math and science courses, introductory English composition Adult Education (Adult Basic Education, Adult Secondary Education, Adult ESL, and HS equivalency preparation) Student achievement in reading, writing, English language proficiency, and mathematics; access to, persistence in, progress through, and completion of adult education programs 13

14. ies.ed.gov Education Interventions The wide range of education curricula, instructional approaches, professional development, technology, and practices, programs, and policies that are implemented at the student, classroom, school, district, state, or federal level to improve student education outcomes High importance to state or local education agency Expected to produce meaningful improvement in student education outcomes within a short period (e.g., a semester or year) Administrative data (or other source) focused on intervention and outcomes (primary data collection not supported by grant).

15. ies.ed.gov Applications Must be from a Partnership Applications must include at least one Principal Investigator (PI) from a research institution and at least one PI from a U.S. state or local education agency – PI from research institution: Must have the ability and capacity to conduct scientifically valid research and expertise in the education issue to be addressed – PI from State or local education agency: Must have decision-making authority for the intervention within his or her agency 15

16. ies.ed.gov Partnership Partnership may be new or existing Research institution has a broad definition – Ability and capacity to conduct scientifically valid research – Not restricted to types of institutions 16

17. ies.ed.gov Partnership: SEAs State education agencies – Examples: education agencies, departments, boards, commissions – Oversee early learning, elementary, secondary, and/or postsecondary and adult education – Also includes education agencies in District of Columbia, the Commonwealth of Puerto Rico, and each of the outlying areas.

18. ies.ed.gov Partnership: LEAs Local education agencies are primarily public school districts Community college districts Tribal education agencies State and city postsecondary systems – If there is a state or city higher education agency that oversees the postsecondary system, include them as an agency partner – If there is no state or city education agency that oversees the postsecondary system, the system can apply as the sole agency partner – A postsecondary system that applies as an education agency partner cannot also serve as the research institution partner in the same project 18

19. ies.ed.gov Additional Partners Partnerships may include more than one State or local education agency if they share similarities and interests Non-education state and local agencies may be partners as long as an education agency is a partner Partnerships may include more than one research institution if they have shared interests and will make unique contributions Partnerships may include other non-research organizations (e.g., issue-oriented or stakeholder groups, non-public organizations that oversee or manage schools) that will contribute to the partnership and its work 19

20. ies.ed.gov Dissemination/Products Projects are to aid education agencies in decision-making Required dissemination – Oral briefing on results to education agency – Written brief, written for non-technical audience, made available free to public Recommended dissemination – Partner presentations to academic and practitioner audiences – Partner publications in academic and practitioner journals – Toolkit or guide for other education agencies on how to conduct a similar study

21. ies.ed.gov Check the Fit of Your Research and Low-Cost, Short-Duration Evaluation Grant Program If you are not looking at student outcomes, then IES is not the appropriate funding agency If you need time and effort to build a partnership and prepare for an evaluation, consider: – Researcher-Practitioner Partnerships under 84.305H If the intervention you want to evaluate – Is not implemented by a state or local education agency, – Cannot be evaluated using secondary data, or – Is not expected to improve student outcomes within a short period. – Then consider the: Education Research Grants Program (84.305A) or Special Education Research Grants Program (84.324A)

22. ies.ed.gov Low-Cost Evaluation: Purpose Promote joint evaluation research by research institutions and state and local education agencies – On an education intervention identified as having great importance by the education agency – That includes practitioner input into the research – That will provide timely rigorous evidence for the agency’s decision-making regarding the intervention – And the results of which will be broadly disseminated in ways easily accessible to researchers, practitioners, and the public 22

23. ies.ed.gov What should the partnerships do during the grant? Identify an education intervention Implemented by an SEA or LEA Of high priority to that agency Intended to improve student education outcomes Carry out an evaluation of that intervention – Using a RCT or an RDD design – Using secondary data – Obtain overall impacts – Obtain subgroup impacts for subgroups identified in the secondary data and of interest to the education agency and/or literature – Examine other moderators and mediators of interest, fidelity of implementation, and comparison group practice if such data are available 23

24. ies.ed.gov Expected Products of the Grant Causal evidence of the impact of a clearly specified intervention implemented by an SEA or LEA – Overall impacts – Impacts for available subgroups of interest Advice for the SEA or LEA – Continuing and/or expanding the use of the intervention – Further research needs, e.g., Evaluation, e.g., variation in impacts, moderation and mediation, generalizability, replication Implementation 24

25. ies.ed.gov The Project Narrative Page Limit: 15 Significance Partnership Research Plan Personnel Resources 25

26. ies.ed.gov Significance The education intervention to be evaluated – The education problem/issue the intervention is to address within the SEA/LEA Relevance to other SEAs or LEAs (secondary importance) – Components of the intervention – Rationale for why the intervention can improve student outcomes within a short period (e.g., 1 semester, 1 year) May include theory of change Difference from status quo Related findings from previous studies and how this study will improve upon past work

27. ies.ed.gov Significance The implementation of the intervention – Who will implement it and how will it be implemented – That education agency will implement or will oversee implementation – Adequate funding available for implementation – Implementation during Year 1 of the project at a level expected to impact student outcomes Sources of secondary data to be used in the evaluation – How these data are collected – How these data will be obtained by researchers by 1 st quarter of Year 2 of project

28. ies.ed.gov The Project Narrative Page Limit: 15 Significance Partnership Research Plan Personnel Resources 28

29. ies.ed.gov Partnership Describe the partners – The research institution and the education agency Offices or divisions within agency whose cooperation is necessary – Any other members of the partnership – Common interest in and benefit from this evaluation – The process through which the partners determined the specific intervention to evaluate – Data sharing agreement – the strategy to obtain the secondary data and provide it for analysis by the 1 st quarter of the second year

30. ies.ed.gov The Project Narrative Page Limit: 15 Significance Partnership Research Plan Personnel Resources 30

31. ies.ed.gov Research Plan State research questions/hypotheses Describe sample and setting – Define population and how your sample and sampling procedures will allow inferences to the population – Exclusion and inclusion rules and their justification – How sample size set to address expected attrition – Strategies used to increase participation and reduce attrition – Describe the setting and its implications for the generalizability of your study 31

32. ies.ed.gov Research Plan: Design Discuss how design will support causal inferences and identify potential threats to internal validity Discuss how degree of equivalence at baseline will be determined Discuss possibility of bias from overall and differential attrition Required use of RCT or RDD – Meet WWC evidence standards without reservations 32

33. ies.ed.gov Research Plan: Design Randomized Controlled Trial (RCT) Note unit of randomization and justify choice Describe process for random assignment and maintaining its integrity Different Approaches to RCTs - Potential Issues – Entire population: Treatment fidelity – Volunteers: Comparison group status – Lotteries: Attrition of non-accepted parties – Staggered roll out: Little time for true comparison – Variations of program/policy: Issue of overall significance 33

34. ies.ed.gov Research Plan: Design Regression Discontinuity Design (RDD) Appropriateness of assignment variable Show true discontinuity Discuss possibility of manipulation of design variable and analyses to determine such manipulation Sensitivity analyses to assess influence of key procedural or analytic decisions on results 34

35. ies.ed.gov Research Plan: Statistical Power Detailed description of power analysis – Justify method used to calculate power – Justify parameters used and assumptions made Provide power for main analyses and important subgroup analyses Along with identifying minimum detectable effect for your analysis, justify its – Reasonableness – Practical meaning Reviewers should be able to check power calculations 35

36. ies.ed.gov Research Plan: Outcome Measures Student education outcome measures relevant to states, districts, and schools – Found in administrative data or other secondary data Discuss reliability, validity, and appropriateness Must be collected during Year 1 of project – Additional data from previous years of intervention’s implementation may also be used if appropriate to the evaluation design Clearly link measures to rationale for the intervention 36

37. ies.ed.gov Research Plan: Optional Measures If available in secondary data, describe measures of – Moderators (subgroups expected) – Mediators (intermediate outcomes) – Fidelity of implementation – Comparison group practice

38. ies.ed.gov Research Plan: Analysis Detail impact analyses – Make clear how analyses directly answer your research questions – Show that analyses are based on the design – Address clustering of students in classrooms in schools – Address missing data – If multiple datasets are to be linked, detail how this will be done Describe any other analyses to be done (e.g., subgroups, other moderators, mediators, and fidelity of implementation) 38

39. ies.ed.gov The Project Narrative Page Limit: 15 Significance Partnership Research Plan Personnel Resources 39

40. ies.ed.gov Personnel Identify all key personnel on the project team – The PI from the research institution who has previous experience carrying out the proposed evaluation design (RCT or RDD) – The PI from the education agency who makes program decisions – Other key personnel Roles and responsibilities on the project – Each individual’s roles and responsibilities on the project – Their qualifications (i.e., expertise and experience) for their role – Their % FTE on the project – Past success at working in similar partnerships PI qualifications for managing a grant of this type Ensure objectivity of evaluation 40

41. ies.ed.gov Resources: To Conduct the Project Describe the institutional resources of all the institutions involved in the partnership and how these resources will contribute to building the partnership and to the research – Institutional capacity to manage the grant – Resources available at the partner institutions to support the project – Plans to acquire any major resources not yet in hand (e.g., secondary data) – Joint Letter of Agreement by partners (Appendix D) – Letter of Agreement to provide administrative data (App. D) 41

42. ies.ed.gov Resources: Dissemination of Results Results expected to be useful to the SEA/LEA partner and, perhaps, other SEA/LEAs – Findings of both beneficial impacts or no impacts Describe your capacity and resources to disseminate findings – Required dissemination through an oral briefing for the agency and a written brief freely available to the public – Dissemination to other audiences (e.g., researchers, policymakers, practitioners, students and their families, public) 42

43. ies.ed.gov Other Important Sections of the Application Appendix A Appendix B Appendix C Appendix D Budget & Budget Narrative 43

44. ies.ed.gov Appendix A (Required for Resubmissions) Page Limit: 3 If you are resubmitting an application, use up to 3 pages to discuss how you responded to reviewer comments 44

45. ies.ed.gov Appendix B (Optional) Page Limit: 5 Figures, charts, or tables that supplement the project narrative Timelines for the project (very useful) Examples of instruments used in the collection of the administrative or other secondary sources of data Do NOT include narrative text

46. ies.ed.gov Appendix C (Optional) Page Limit: 5 Examples of materials used in the intervention: – curriculum materials – computer screen shots – training documents – assessment items – other materials Do NOT include narrative text 46

47. ies.ed.gov Appendix D (Required) No Page Limit Required Letters of Agreement – Joint Letter from the research institution and the SEA/LEA Document participation and cooperation in the partnership Set out each’s roles and responsibilities under the project – Letter from the office in charge of the agency’s data Project will have access to data required in time to do analysis Optional Letters of Agreement – Separate Letters from other organizations taking part – Letters from any consultants and schools taking part 47

48. ies.ed.gov Budget & Budget Narrative Maximum project length is 2 years Maximum award is $250,000 – Funds must be used for evaluation only (e.g., cannot be used for implementation of the intervention or primary data collection) – Award size depends on project scope Include a detailed budget form (SF 424) AND a budget narrative that links the activities, personnel, etc. from the Project Narrative to the funds requested 48

49. ies.ed.gov Preparing Your Application Important dates Information sources – Read the RFA – Talk with a program officer Review process

50. ies.ed.gov Application Deadline Letter of Intent Due Date Application Package Posted Start Dates January 12, 2016 4:30:00 PM DC Time November 12, 2015 July 1 – September 1, 2016 Important Dates & Deadlines 50

51. ies.ed.gov Information Sources Request for Applications – http://ies.ed.gov/funding/ http://ies.ed.gov/funding/ Abstracts of Projects – http://ies.ed.gov/funding/grantsearch/index.asp http://ies.ed.gov/funding/grantsearch/index.asp Application Package – www.grants.gov www.grants.gov Program Officers – Robert.Ochsendorf@ed.gov 84.324L Robert.Ochsendorf@ed.gov – Phill.Gagne@ed.gov 84.305L Phill.Gagne@ed.gov – Allen.Ruby@ed.gov 84.305L Allen.Ruby@ed.gov 51

52. ies.ed.gov Peer Review (Standards & Review Office) Compliance screening for format requirements Responsiveness screening for program requirements Assignment to review panel – 2 to 3 reviewers (substantive and methodological) – The most competitive proposals are reviewed by full panel Many panelists will be generalists to your topic Panels contain experts in relevant methodologies – Panel provides an overall score plus specific scores on Significance, Partnership, Research Plan, Personnel, and Resources 52

53. ies.ed.gov Notification All applicants will receive e-mail notification that the following information is available via the Applicant Notification System (ANS): Status of award Reviewer summary statement If you are not granted an award the first time, consider resubmitting and talking with the Program Officer 53

54. ies.ed.gov For More Information http://ies.ed.gov/funding Robert.Ochsendorf@ed.govRobert.Ochsendorf@ed.gov 84.324L Phill.Gagne@ed.govPhill.Gagne@ed.gov 84.305L Allen.Ruby@ed.govAllen.Ruby@ed.gov 84.305L 54