1. OFFICE OF ENGLISH LANGUAGE ACQUISITION NATIONAL PROFESSIONAL DEVELOPMENT PROGRAM (NPD) NPD Grant Competition Webinar 3: Evidence and Evaluation January 20, 2016 Note: These slides are intended as guidance only. Please refer to the official documents published in the Federal Register.

2. ATTENTION!  To hear the presentation you must call into the conference line: –Dial: 1-800-857-4871 –Code: 7662107  You cannot hear or fully participate in the webinar unless you call the conference line  During the presentation all phone lines will be placed on mute to avoid feedback during the presentation. Webinar #3 – Evidence and Evaluation Dial In: 1-800-857-4871 Audio Code: 7662107 2

3. DISCLAIMER  This webinar training series serves to provide technical assistance to applicants in preparing their applications and to respond to applicant questions. The purpose of the webinars is to review the competition priorities, requirements, and selection criteria with potential applicants, as well as for Department staff to answer questions about the Notice Inviting Applications (NIA) for this competition.  Please refer to the NIA published in the Federal Register for additional information.  Application instructions can be found in the Application package located in grants.gov (search CFDA 84.365) or the NPD applicant information website http://www2.ed.gov/programs/nfdp/applicant.html http://www2.ed.gov/programs/nfdp/applicant.html  Do not rely solely on the information shared in the webinars.  The NPD staff is unable to address applicant specific questions at any time during the competition. Webinar #3 – Evidence and Evaluation Dial In: 1-800-857-4871 Audio Code: 7662107 3

4. WEBINARS  There are a total of three pre-application webinars. –This is the third and final pre-application webinar.  Please use the “chat box” located on the right side of the screen to submit questions. If time permits, your questions will be answered during the presentation and we will respond to others at the end of the presentation or via FAQs on the NPD website.  All webinars will be recorded. A link to the recording along with the PowerPoint slides will be posted at: http://www2.ed.gov/programs/nfdp/applicant.html. http://www2.ed.gov/programs/nfdp/applicant.html Webinar #3 – Evidence and Evaluation Dial In: 1-800-857-4871 Audio Code: 7662107 4

5. WEBINAR SERIES o Webinar 1: NPD Competition Overview  December 16, 2015 o Webinar 2: GPRA & Selection Criteria  January 13, 2016 o Webinar 3: Competition Priorities & Evidence  January 20, 20162pm Updates and links to webinars located on the NPD applicant information site http://www2.ed.gov/programs/nfdp/applicant.htmlhttp://www2.ed.gov/programs/nfdp/applicant.html NOTE: A series of post award webinars will to be offered to all new grantees following the conclusion of the competition. Webinar #3 – Evidence and Evaluation Dial In: 1-800-857-4871 Audio Code: 7662107 5

6. AGENDA  NPD Program Priorities –Absolute –Competitive –Invitational  NPD Program Evidence Criteria –“Entry” and “Exit” Evidence  Competitive Preference Priority 1 – Moderate Evidence of Effectiveness  Project Evaluation –WWC with reservations  Resources and Reminders Webinar #3 – Evidence and Evaluation Dial In: 1-800-857-4871 Audio Code: 7662107 6

7. PRIORITIES  Absolute Priority: Providing Professional Development to Improve Instruction for English Learners  Competitive Preference Priority (CPP) 1: Moderate Evidence of Effectiveness (0 or 5 points)  Competitive Preference Priority (CPP) 2: Improving Parent, Family, and Community Engagement (0 to 5 points)  Invitational Priority 1: Dual Language Approaches  Invitational Priority 2: Supporting the Early Learning Workforce to serve ELs Webinar #3 – Evidence and Evaluation Dial In: 1-800-857-4871 Audio Code: 7662107

8. ABSOLUTE PRIORITY  Absolute Priority: For FY 2016 and any subsequent year in which we make awards from the list of unfunded applicants from this competition, this priority is an absolute priority. Under 34 CFR 75.105(c)(3) we consider only applications that meet this priority. Under this priority we provide funding to projects that provide professional development activities that will improve classroom instruction for ELs and assist educational personnel working with ELs to meet high professional standards, including standards for certification and licensure as teachers who work in language instruction educational programs or serve ELs. Webinar #3 – Evidence and Evaluation Dial In: 1-800-857-4871 Audio Code: 7662107 8

9. COMPETITIVE PREFERENCE PRIORITY 1 MODERATE EVIDENCE OF EFFECTIVENESS  Applicants seeking to address Competitive Preference Priority (CPP) 1 should identify up to two citations for studies that meet the definition of Moderate Evidence of Effectiveness.  Cited studies may include those already listed in the Department’s What Works Clearinghouse (WWC) Reviewed Studies Database (see http://ies.ed.gov/ncee/wwc/ReviewedStudies.aspx).http://ies.ed.gov/ncee/wwc/ReviewedStudies.aspx  Cited studies must be included in both the application narrative section of the application and the application abstract. NOTE: The Department will review the studies cited by the applicants to determine if they meet the requirements for, Competitive Preference Priority 1, Moderate Evidence of Effectiveness. 9 Webinar #3 – Evidence and Evaluation Dial In: 1-800-857-4871 Audio Code: 7662107

10. COMPETITIVE PREFERENCE PRIORITY 2 IMPROVING PARENT, FAMILY AND COMMUNITY ENGAGEMENT  Up to an additional 5 points may be awarded to applicants who address Competitive Preference Priority 2.  This would be for projects that are designed to improve student outcomes through one or more of the following activities: Webinar #3 – Evidence and Evaluation Dial In: 1-800-857-4871 Audio Code: 7662107 10

11. COMPETITIVE PREFERENCE PRIORITY 2 IMPROVING PARENT, FAMILY AND COMMUNITY ENGAGEMENT CPP 2: (a)Developing and implementing Systemic Initiatives to improve Parent and Family Engagement by expanding and enhancing the skills, strategies, and knowledge (including techniques or use of technological tools needed to effectively communicate, advocate, support, and make informed decisions about the student’s education) of parents and families. (b)Providing professional development that enhances the skills and competencies of school or program leaders, principals, teachers, practitioners, or other administrative and support staff to build meaningful relationships with students’ parents or families through Systemic Initiatives that may also support students’ learning at home. (c)Implementing initiatives that improve Community Engagement, the relationships between parents or families and school or program staff by cultivating Sustained Partnerships. Webinar #3 – Evidence and Evaluation Dial In: 1-800-857-4871 Audio Code: 7662107 11

12. INVITATIONAL PRIORITY 1 DUAL LANGUAGE APPROACHES  We encourage applicants to propose projects to improve educator preparation and professional learning for dual language implementation models to support effective instruction for ELs. In particular, we encourage such approaches to take into account the unique needs of recently arrived limited English proficient students, immigrant children and youth, and Native American students, who are members of federally recognized Indian tribes. 12

13. INVITATIONAL PRIORITY 2 SUPPORTING THE EARLY LEARNING WORKFORCE TO SERVE ELS  We encourage applicants to propose projects that improve the quality and effectiveness of the early learning workforce, including administrators, so that they have the necessary knowledge, skills, and abilities to improve ELs’ cognitive, health, social-emotional, and dual language development. Early learning programs are designed to improve early learning and development outcomes across one or more of the Essential Domains of School Readiness for children from birth through third grade (or for any age group within this range). Further, we encourage applicants to include in such projects these foundational professional learning domains for educators at all levels of teaching including secondary preparation. 13

14. AGENDA  NPD Program Priorities –Absolute –Competitive –Invitational  NPD Program Evidence Criteria –“Entry” and “Exit” Criteria  Competitive Preference Priority 1 – Moderate Evidence of Effectiveness  Project Evaluation –WWC with reservations  Resources and Reminders Webinar #3 – Evidence and Evaluation Dial In: 1-800-857-4871 Audio Code: 7662107 14

15. DISTINCTION BETWEEN ENTRY AND EXIT EVIDENCE CRITERIA  Encourage the use of evidence to increase the effectiveness of projects funded by the National Professional Development program (“entry” criteria) –Competitive Preference Priority 1: Moderate Evidence of Effectiveness  Develop evidence base of effective professional development practices for educators of ELs (“exit” criteria) –Project Evaluation (Section D of selection criteria) 15

16. AGENDA  NPD Program Priorities –Absolute –Competitive –Invitational  NPD Program Evidence Criteria –“Entry” and “Exit” Evidence  Competitive Preference Priority 1 – Moderate Evidence of Effectiveness  Project Evaluation –WWC with reservations  Resources and Reminders 16

17. MODERATE EVIDENCE OF EFFECTIVENESS: “ENTRY” EVIDENCE Characteristics of the Cited Prior Evidence Option 1 Option 2 Note: Greyed-out cells indicate criteria on which the standards are silent. *See What Works Clearinghouse (WWC) Procedures and Standards Handbook, which can currently be found at the following link: http://ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19. ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19 17

18. CPP 1: MODERATE EVIDENCE OF EFFECTIVENESS  To receive points for Competitive Preference Priority (CPP) 1, an application for an NPD grant must be supported by moderate evidence of effectiveness  An applicant should identify up to two study citations to be reviewed against WWC Evidence Standards for the purpose of meeting the Competitive Preference Priority  An applicant should clearly identify these citations in both the application narrative and the application abstract. The Department will not review a study citation that an applicant fails to clearly identify for the evidence review. 18

19. HOW TO IDENTIFY RELEVANT STUDIES OPTION 1 - USING THE IES PRACTICE GUIDE  Applicants can reference studies in the IES Practice Guide, Teaching Academic and Literacy to English Learners in Elementary and Middle School http://ies.ed.gov/ncee/wwc/PracticeGuide.aspx?sid=19 http://ies.ed.gov/ncee/wwc/PracticeGuide.aspx?sid=19 19

20. HOW TO IDENTIFY RELEVANT STUDIES USING THE IES PRACTICE GUIDE (CONTINUED)  Using the Practice Guide to identify studies that will meet the moderate evidence priority is a multi-step process.  First, go to Appendix D (pgs. 85-102) of the guide for a listing of all studies that provide evidence for the recommendations in the guide. 20

21. HOW TO IDENTIFY RELEVANT STUDIES USING THE IES PRACTICE GUIDE (CONTINUED)  After you have identified a study, you will need to locate it in the WWC Database of reviewed studies to see the study’s What Works Clearinghouse rating: http://ies.ed.gov/ncee/wwc/ReviewedStudies.aspxhttp://ies.ed.gov/ncee/wwc/ReviewedStudies.aspx  Note: This database will provide you with the study rating, but you will need to examine the practice guide tables in Appendix D and/or the original study to determine whether the study meets the other moderate evidence requirements. 21

22. HOW TO IDENTIFY RELEVANT STUDIES OPTION 2 – PROPOSE ANOTHER STUDY  Applicants can also propose to use studies that are not included in the Teaching Academic and Literacy to English Learners in Elementary and Middle School Practice Guide, provided that studies meet WWC Standards 22

23. 23 OVERVIEW: WHAT KINDS OF STUDIES QUALIFY UNDER MODERATE EVIDENCE? STUDIES NEED TO ADDRESS EFFECTIVENESS OF AN INTERVENTION AND INCLUDE TECHNICAL DETAIL Publicly-available original research studies Peer-reviewed or not Full written technical document (report, article), final version of the results Studies by developers of their own product or independent evaluators IN Newspaper articles Literature reviews, briefs, summaries Secondary research (re- analysis) Implementation studies Correlational studies (unless EOP) Descriptive studies of baselines and trend lines OUT

24. PROCESS FOR ASSESSING RESEARCH EVIDENCE  The WWC reviews studies using a relevant topic-specific protocol where possible. –Review Protocols guide all WWC reviews http://ies.ed.gov/ncee/wwc/Publications_Reviews.aspx?f=All %20Publication%20and%20Product%20Types,5;#pubsearch http://ies.ed.gov/ncee/wwc/Publications_Reviews.aspx?f=All %20Publication%20and%20Product%20Types,5;#pubsearch –If a relevant topic-specific protocol does not exist, the WWC will use the Single Study Review protocol to guide the review: http://ies.ed.gov/ncee/wwc/documentsum.aspx?sid=234 http://ies.ed.gov/ncee/wwc/documentsum.aspx?sid=234 –For more information on the WWC standards and review process, please see the “Tools and Resources” section of the WWC’s website: http://ies.ed.gov/ncee/wwc/tools.aspxhttp://ies.ed.gov/ncee/wwc/tools.aspx 24

25. RESPONSIBILITY FOR THE “ENTRY” EVIDENCE REVIEWS  The Institute of Education Sciences (IES) conducts the review of “entry” evidence for CPP 1. –IES sends evidence citations to What Works Clearinghouse (WWC) contractors to conduct reviews.  IES provides the WWC reviews, as well as information about the intervention, sample, and setting for each study reviewed to OELA.  OELA makes a determination about the relevance of the intervention, sample, and setting of the study compared to what the applicant has proposed. 25

26. EVIDENCE CITATION SUBMISSIONS  ED will limit reviews to studies explicitly cited in the application narrative and application abstract for Competitive Preference Priority 1  Applicants must ensure evidence is available to ED from publicly available sources and provide links or other guidance indicating where it is available. –Or an applicant may include copies of evidence in Section 6: Other Forms.  Include only citations that are relevant to the intervention being proposed.  For evidence that will go through WWC review, include only citations that are primary analyses of the effect of the intervention being proposed. 26

27. AGENDA  NPD Program Priorities –Absolute –Competitive –Invitational  NPD Program Evidence Criteria –“Entry” and “Exit” Evidence  Competitive Preference Priority 1 – Moderate Evidence of Effectiveness  Project Evaluation –WWC with reservations  Resources and Reminders 27

28. SELECTION CRITERIA QUALITY OF THE PROJECT EVALUATION (UP TO 20 POINTS) 28  The Secretary considers the quality of the evaluation to be conducted of the proposed project. In determining the quality of the evaluation, the Secretary considers the following factors:  (1) The extent to which the methods of evaluation are thorough, feasible, and appropriate to the goals, objectives, and outcomes of the proposed project.  (2) The extent to which the methods of evaluation will, if well implemented, produce evidence about the project’s effectiveness that would meet the What Works Clearinghouse Evidence Standards with reservations.  (3) The extent to which the methods of evaluation will provide performance feedback and permit periodic assessment of progress toward achieving intended outcomes. Webinar #2 – GPRA & Selection Criteria Dial In: 1-800-369-3355 Audio Code: 1101616

29. PROJECT EVALUATION  Plan for design and execution of the evaluation must be able to meet WWC standards with reservations  Implications – must have: –Comparison or control group, strategy for matching/assigning –Statistical model for analyzing data 29

30. HOW DOES THE WWC ASSESS RESEARCH EVIDENCE?  Type of design: Does the study design allow us to draw causal conclusions?  Strength of data: Does the study focus on relevant outcomes and measure them appropriately?  Adequacy of statistical procedures: Are the data analyzed properly? 30

31. WWC RATINGS AND STUDY DESIGNS 31 Meets WWC evidence standards without reservations Meets WWC evidence standards with reservations Does not Meet WWC standards Randomized controlled trials (RCTs) with little sample attrition Non-randomized studies with comparison groups similar at baseline Comparison groups are not similar at baseline

32. THE WWC REQUIRES DISTINCT GROUPS  At least two groups (one intervention and one comparison)  Not a group compared to itself  Not sequential/historical cohorts (3 rd graders in 2015 vs 3 rd graders in 2014)  Clear definition of intervention and eligibility criteria – what services and who can get them?  Intervention group 3 rd Grade EL Students Distinct groups Comparison group 3 rd Grade EL Students Created randomlyorNon-randomly  Identify eligible sample  Convenience sample  Lottery or flip of coin  Matching  Prospective  Retrospective 32

33. CAUTION 1 RCTS WITH HIGH SAMPLE ATTRITION MUST DEMONSTRATE BASELINE EQUIVALENCE  Randomly assigning study participants to intervention and comparison groups creates groups with similar characteristics at the start of the study.  If there is attrition, members of the intervention and comparison groups may not have had similar characteristics at baseline, preventing us from being able to attribute any differences in outcomes solely to the intervention 33

34. STANDARDS ACCOUNT FOR OVERALL AND DIFFERENTIAL ATTRITION RATES 34

35. CAUTION 2 ALL QEDS MUST DEMONSTRATE BASELINE EQUIVALENCE BETWEEN THE TREATMENT AND CONTROL GROUPS 35

36. BASELINE EQUIVALENCE STANDARD EQUIVALENCE MUST BE DEMONSTRATED ON THE ANALYTIC SAMPLE 36

37. AGENDA  NPD Program Priorities –Absolute –Competitive –Invitational  NPD Program Evidence Criteria –“Entry” and “Exit” Evidence  Competitive Preference Priority 1 – Moderate Evidence of Effectiveness  Program Evaluation –WWC with reservations  Resources and Reminders 37

38. RESOURCES  Additional competition resources, including links to this pre-application webinar series can be found at: http://www2.ed.gov/programs/nfdp/applicant.html.http://www2.ed.gov/programs/nfdp/applicant.html  We also invite applicants to view two Webinar recordings that were hosted by the Institute of Education Sciences (IES). –The first Webinar addresses strategies for designing and executing well- designed quasi-experimental design studies. This webinar is available at: http://ies.ed.gov/ncee/wwc/Multimedia.aspx?sid=23. http://ies.ed.gov/ncee/wwc/Multimedia.aspx?sid=23 –The second Webinar focuses on more rigorous evaluation designees, including strategies for designing and executing randomized controlled trials. This Webinar is available at: http://ies.ed.gov/ncee/wwc/Multimedia.aspx?sid=18.http://ies.ed.gov/ncee/wwc/Multimedia.aspx?sid=18  Regional Educational Laboratory (REL) Southeast is conducting a webinar series designed for professional development program evaluators. –Participants will develop a working knowledge of the critical components of professional development program evaluation. –The entire webinar series will be recorded and available for future viewing on the IES YouTube Channel. Register at rel-se.fsu.edu/events Webinar #3 – Evidence and Evaluation Dial In: 1-800-857-4871 Audio Code: 7662107 38

39. Dates to Remember December 11, 2015 - application period opens Pre-Application webinars December 16, 2015 – Webinar #1 January 13, 2016 – Webinar #2 January 20, 2016 – Webinar #3 February 19, 2016 - application period closes July, 2016 - awards announced* Grant period begins after awards are announced * tentative 39 Webinar #3 – Evidence and Evaluation Dial In: 1-800-857-4871 Audio Code: 7662107

40. APPLICATIONS DUE FEBRUARY 19, 2016 IMPORTANT DATE

41. Questions?

42. THANK YOU!