1. The State of High School Equivalent Program (HEP) Evaluation U.S. Department of Education Office of Migrant Education Webinar August 22, 2013 2:00 pm – 3:30 pm EDT

2. Overview I. Background and Purpose II. OME Review of Evaluation Reports III. Rubric for Review IV. Evaluation Reports Reviewed V. Observations VI. Notable Evaluation Reports VII. Initial Grantee Feedback VIII. Today’s Feedback IX. Next Steps 2

3. I.Background and Purpose 3

4. Compliance Improvement 4

5. Compliance: EDGAR EDGAR -- Education Department General Administrative Regulations 34 CFR §75.590 requires a grantee to evaluate annually: – The recipient’s progress in achieving the objectives in its approved application; – The effectiveness of the project in meeting the purposes of the program; and – The effect of the project on served participants 5

6. Compliance: HEP Application Evaluation Plans Each approved and funded grant application included an evaluation plan within the Narrative and Quality of Project Evaluation (Section 7) “…In determining the quality of the evaluation the Secretary considers the following factors: – (i.) The extent to which the methods of evaluation are appropriate to the context within which the project operates. (3 points possible) – (ii.) The extent to which the methods of evaluation provide for examining the effectiveness of project implementation and strategies. (3 points possible) – (iii.) The extent to which the methods of evaluation will provide performance feedback and permit periodic assessment of progress toward achieving intended outcomes. (4 points possible). 6

7. Compliance: HEP Application Section 7: Note Quality of the Project Evaluation: Important note about the project evaluation: A strong evaluation plan should be included in the application narrative and should be used, as appropriate, to shape the development of the project from the beginning of the grant period. The plan should include benchmarks to monitor progress toward specific project objectives and also outcome measures to assess the impact on teaching and learning or other important outcomes for project participants. More specifically, the plan should identify the individual and/or organization that have agreed to serve as evaluator for the project and describe the qualifications of that evaluator. (continued) 7

8. Compliance: Application Section 7: Note, continued The plan should describe the evaluation design indicating: (1) What types of data will be collected (2) When various types of data will be collected (3) What methods will be used (4) What instruments will be developed and when (5) How the data will be analyzed (6) When reports of results and outcomes will be available (7) How the applicant will use the information collected through the evaluation to monitor progress of the funded project and to provide accountability information both about success at the initial site and effective strategies for replication in other settings. Applicants are encouraged to devote an appropriate level of resources to project evaluation 8

9. Compliance: APR: Evaluation References Annual Performance Report (APR) – requirement for grantees Includes evaluation material: – APR Section D Project Goals and Objectives – Section 2) Explanation of Progress (Includes Qualitative Data and Data Collection Information) maximum 2500 words 1. For each project objective and associated performance measures, indicate what data (quantitative and/or qualitative) were collected and when they were collected, the evaluation methods that were used, and how the data were analyzed. Clearly identify and explain any deviations from your approved evaluation plan, including changes in design or methodology, or the individual or organization conducting the evaluation…. 4. Indicate how you used your data and information from your evaluation to monitor the progress of your grant, and if needed, to make improvements to your original project plan (e.g., project activities and milestones) which are consistent with your approved objectives and scope of work. 9

10. Improvement: Continuous Improvement Cycle Guide Incor- porate 10

11. II.OME Review of Evaluation Reports 11

12. II. OME Review of Evaluation Reports OME initiated a review of evaluation reports received from HEP and CAMP grantees: Requested current grantees submit evaluation reports by April 30, 2013 25 HEP and CAMP evaluation reports were submitted to OME Evaluation reports addressed periods of review ranging from 2009 through 2012 12

13. HEP Grantee Institutions and Agencies CURRENT GRANTEE INSTITUTIONS AND AGENCIES Type of Grantee Institution of Higher Education Non-IHE AgenciesCombinationTotal HEP389047 13

14. III.Rubric for Review 14

15. III. Rubric for Review Rubric for review of evaluation reports: Developed to permit consistent review of all submitted evaluation documents Identified core information of particular interest to OME, based on approved applications Scored on a general scale: – “0” – Does Not Meet Expectations – “1” – Needs Improvement – “2” – Meets Expectations – “3” – Exceeds Expectations 15

16. Rubric for Report Review, continued Scoring of components 1 – 5 was influenced by degree evaluation documents provided qualitative and/or quantitative data Maximum score possible: 108 Average HEP evaluation report score: 40.6 16

17. Rubric: Outline of Components Rubric incorporated 5 components: Component 1 – GPRA Results Component 2 – Fidelity of Implementation to Design Component 3 – Effectiveness of Project Design Component 4 – Collaborative Agreements Component 5 – Recommendations 17

18. Rubric: Component 1 – GPRA 1 and GPRA 2 Evaluation report scored in Component 1 on: – 1.a.b. Presentation of GPRA 1 and GPRA 2 results and related progress to meeting performance requirements – 1.c. Presentation of performance measures included in grantee’s approved application and target v. actual results – 1.d. Information on data validation for GPRA 1 and GPRA 2 results Maximum points available – 12 points 18

19. Rubric: Component 2 – Fidelity of Implementation to Design Rubric Component 2 scored evaluation report’s qualitative or quantitative data for grantee level of fidelity to implementation of: – 2.a. Instructional Services Design – 2.b. Support Services Design – 2.c. Placement Services Design – 2.d. Management Plan Design – 2.e. Recruitment Plan Design Maximum points available – 15 points 19

20. Rubric: Component 3 – Effectiveness of Project Design Component 3 details areas in Component 2 Evaluation reports scored for information on specific topics including: – Review of eligibility screening tools – Use of student/staff surveys – Staff qualifications – Student stipends – Planned against actual costs – Student demographic information Maximum points available – 57 points 20

21. Rubric: Component 3.a. – Effectiveness of Project Design, continued 3. Effectiveness of Project Design – 3.a. Instructional Services Design 3.a.1. The report includes data on the number of instruction hours provided and schedule/availability of instruction hours to meet individual needs. 3.a.2. The report includes data on the type and usage of eligibility screening tools. 3.a.3. The report includes data on how instructional requirements of individual students are assessed and met. 21

22. Rubric: Component 3.b.-3.d. – Support, Placement, Management – 3.b. Support Service Design – 3.c. Placement Services Design for HEP projects – 3.d. Management Plan Design. Analyzes the effectiveness of the grantee’s management plan design 3.d.1. Teaching and administrative staff structure 3.d.2. Senior Project administrative staff 3.d.3. Qualifications of project instructors 3.d.4 Professional Staff development 3.d.5 Student Records 3.d.6 Approved Procedures Policy Manual 22

23. Rubric: Component 3.d.-3.e. – Management, Recruitment 3.d.7. Financial/cost sharing issues with host institutions 3.d.8. Student stipends 3.d.9. Planned and Actual costs – 3.e. Recruitment Plan Design Report analyzes the effectiveness of the grantee’s recruitment plan design 3.e.1. Demographic information 3.e.2. Training for recruiters 23

24. Rubric: Component 4 – Collaborative Agreements In Component 4, evaluation reports scored based on information and analysis that addressed: – 4.a. Existing collaborative agreements with educational institutions or service providers – 4.b. Planned collaborative agreements – 4.c. Relationships with host institutions, including unresolved facilities, equipment/computers, HR and related issues that affect project performance Maximum points available – 9 points 24

25. Rubric: Component 5 – Recommendations In Component 5, evaluations scored on recommendations for key areas that were linked to information included in the report: – 5.a. Instructional Services – 5.b. Support Services – 5.c. Placement Services – 5.d. Management Plan – 5.e. Recruitment Plan Maximum points available – 15 points 25

26. IV.Evaluation Reports Reviewed 26

27. HEP Evaluation Reports 15 evaluation reports submitted from 14 grantees Reports submitted from 10 states: Arizona, Arkansas, California, Kansas, Maine, New Jersey, Oklahoma, Pennsylvania, Texas, and Washington TYPE OF GRANTEE INSTITUTIONS Type of Grantee Institution of Higher Education Non-IHE AgenciesCombination HEP10*40 * One grantee submitted documents for two different periods of performance that were scored as separate evaluations 27

28. Type of HEP Evaluations Submitted Wide range of evaluation reports Several reports focused specifically on: – Questionnaires – Student surveys – Site visit reports – Focus group results Some reports were detailed, 15-30 page formative or summative evaluations with numerous appendices Each document submitted to OME was reviewed through the rubric 28

29. Type of HEP Evaluations Submitted TYPE OF HEP EVALUATION DOCUMENTS SUBMITTED (as defined by evaluator and/or noted in document) Type of Grantee Number of Distinct Grantees that Submitted Evaluations Number of Documents Submitted FormativeSummative Combination Formative Summative HEP1421121 TYPE, continued (as defined by evaluator and/or noted on document) Student/Staff Surveys QuestionnairesFocus Groups Other (including site visits, annual, etc. 04211 29

30. V.Observations 30

31. General Observations Grantees determined the format for evaluation reports All grantees may not have submitted all evaluation reports on file in response to OME request Although approved applications define grantee evaluation processes, a number of submitted reports did not reference, or reflect fully, the processes indicated in the funded application Few reports addressed implementation of a planned budget 31

32. Observations: Component 1 – GPRA 1 and GPRA 2 All HEP formative or summative evaluations provided Component 1 – GPRA 1 and GPRA 2 data Five HEP evaluation documents presented general information on validation of data for GRPA results 32

33. Observations: Component 2 – Fidelity of Implementation to Design Formative and summative evaluations presented data on progress in meeting GPRA 1 and 2 and measurable performance objectives Few evaluations presented data on grantee fidelity to implementation of design for instructional, support and placement services and management and recruitment plans Expanded information might have been provided through: – Documentation of services – Student and staff survey results – Tutoring and counseling logs – Class attendance sheets 33

34. Observations: Component 3 – Effectiveness of Project Design Instructional Services – HEP evaluations provided minimal data on instructional services offered Support Services – References in several evaluations were limited to a simple listing of the support services – One evaluation included detailed information regarding delayed stipend payments, related staff interviews, and a proposed interim solutions 34

35. Observations: Component 3 – Effectiveness of Project Design, continued Management Plan – Limited discussion was included in reports about: Management plan Senior management Administrative staff Security and confidentiality measures for student records Fiscal management 35

36. Observations: Component 3 – Effectiveness of Project Design, continued Recruitment Plan – To evaluate the effectiveness of the recruitment plan design, few reports provided: Demographic data Selection and non-selection rates Screening tools References to trainings offered to recruiters 36

37. Observations: Component 4 – Collaborative Agreements Collaborative agreements key to helping grantees leverage Federal, State and local resources Ineffective execution of agreements can be a detriment to grantee success Few evaluations included discussion about the relationships or plans to expand collaborative partners 37

38. Observations: Component 5 – Recommendations Many evaluation documents did not provide extensive recommendations Recommendations often did not link directly to discussion within the report Five HEP evaluations made no recommendations Texas A&M International University evaluation provided comprehensive recommendations that were directly related to issues in the report 38

39. VI.Notable Evaluation Reports 39

40. Notable Evaluation Reports: Validation, Surveys Reports notable for data validation, survey results and collaborative agreements Data Validation: – 2011-2012 Heritage University HEP Survey Results: – 2011 Texas State Technical College, Harlingen HEP Collaborative Agreements: – 2011 Texas State Technical College, Harlingen HEP 40

41. Notable Evaluation Reports: Overall Notable Overall Evaluations: 2012 Hartnell Community College District, Hartnell College HEP 2009-2010 Texas A&M International University (TAMIU) HEP 2011 Texas State Technical College, Harlingen HEP 41

42. VII.Initial Grantee Feedback (HEP-CAMP ADM Session) 42

43. Initial Grantee Feedback Group 1: Effectiveness of Project Design - Umbrella – requires support from the following designs Management plan Recruitment plan Support services Placement services – Instructional services design as the “handle” that supports the umbrella. 43

44. Initial Grantee Feedback Group 2: Rubric = Trash Receptacle Repurpose the rubric by simplifying it Most importantly, emphasize Fidelity to your grant application and objectives No. Served is important, fidelity to recruitment plan GPRA 1 and GPRA 2, important to meet/exceed 44

45. Initial Grantee Feedback Group 3: Successful Evaluation is Dependent Upon Data Validation Data on Performance Measures (GPRAs) Graduate HEP Postsecondary, upgraded employment, military 45

46. Initial Grantee Feedback Group 4: HEP-CAMP Success = School – Takes nourishment and collaboration with Agencies K-12 MEP Private Sector 46

47. VIII. Today’s Feedback 47

48. Today’s Feedback Please consider the rubric, and provide feedback on the rubric. What component(s) would you keep? Why? What component(s) would you eliminate? Why? What component(s) would you add? Why? 48

49. IX. Next Steps – Provide Continued Feedback 49

50. Recommendations for Grantee Evaluations In the future, OME may develop guidance/outline for evaluation reports Further grantee input is requested by e-mail to: Edward.Monaghan@ed.gov Resources may include: – HEP-CAMP Toolkit http://www.hepcamptoolkit.org/hep/index.php/overseei ng-your-grant/evaluation/ http://www.hepcamptoolkit.org/hep/index.php/overseei ng-your-grant/evaluation/ – MEP Evaluation Toolkit http://results.ed.gov/sites/results.ed.gov/files/pe- toolkit.pdf http://results.ed.gov/sites/results.ed.gov/files/pe- toolkit.pdf 50

51. Rubric Components 51

52. OME HEP/CAMP RUBRIC for EVALUATION REPORTS Grantee Name: HEP or CAMP: Evaluator Name/s: Type of Evaluator:Internal; Third Party Type of Report: Formative, Summative, Combination, Focus Group, Site Evaluation Form, Survey 52

53. GPRA Results Component 1 GPRA Results Evaluation Component Does Not Meet Expectations Needs Improvement Meets Expectations Exceeds Expectations Points 1. GPRA Results0123 1.a. GPRA 1 results The report does not include GPRA 1 results. The report includes minimal information on GPRA 1 results. The report presents GPRA 1 results and the grantee’s attainment level in relation to GPRA 1 target. The report presents GPRA 1 results and a detailed description of the grantee’s progress to meet performance requirements. x 1.b. GPRA 2 results The report does not include GPRA 2 results. The report includes minimal information on GPRA 2 results. The report presents GPRA 2 results and the grantee’s attainment level in relation to GPRA 1 target. The report presents GPRA 2 results and a detailed description of the grantee’s progress to meet performance requirements. x 53

54. GPRA Results, continued Component 1 GPRA Results Evaluation Component Does Not Meet Expectations Needs Improvement Meets ExpectationsExceeds Expectations Points 1. GPRA Results0123 1.c. The report presents data on the project specific performance measures included in the grantee’s application, including actual results compared with targeted results. The report presents no information on other performance measures. The report only provides a listing of other performance measures. The report includes data on most project specific performance measures included in the grantee’s application The report presents specific performance measures included in the grantee’s application, including actual results compared with targeted results. x 1.d. Data validation for GPRA 1 and GPRA 2 data. The reports presents no information on validation of data. The report provides minimal reference to validation of data. The report discusses data validation processes for the project. The report discusses in detail the grantee’s processes for validating the project data. x 1. Sub-Total0 54

55. Fidelity of Implementation to Design Component 2 Fidelity of Implementation to Design Instructional Services Evaluation Component Does Not Meet Expectations Needs Improvement Meets Expectations Exceeds Expectations Points 2. Fidelity of Implementation to Design 0123 2.a. Fidelity to instructional services design. The report presents qualitative and/or quantitative data on the grantee's level of fidelity of implementation to the instructional services design. The report presents no qualitative and/or quantitative data on the grantee’s level of fidelity of implementation to the instructional services design. The report presents minimal qualitative and/or quantitative data on the grantee’s level of fidelity of implementation to the instructional services design. The report presents qualitative and/or quantitative data on the grantee’s level of fidelity of implementation to the instructional services design. The report presents qualitative and/or quantitative data on the grantee’s level of fidelity of implementation to key elements in the instructional services design. x 55

56. Fidelity of Implementation to Design, continued Component 2 Fidelity to Implementation of Design Support Services Evaluation Component Does Not Meet Expectations Needs Improvement Meets Expectations Exceeds Expectations Points 2. Fidelity of Implementation to Design 0123 2.b. Fidelity to support services design. The report presents qualitative and/or quantitative data on the grantee's level of fidelity of implementation to the support services design. The report presents no qualitative or quantitative data on the grantee’s level of fidelity of implementation to the support services design. The report presents minimal qualitative or quantitative data on the grantee’s level of fidelity of implementation to the support services design. The report presents qualitative or quantitative data on the grantee’s level of fidelity of implementation to the support services design. The report presents qualitative and quantitative data on the grantee’s level of fidelity of implementation to the support services design. x 56

57. Fidelity of Implementation to Design, continued Component 2 Fidelity of Implementation to Design Placement Services Evaluation Component Does Not Meet Expectations Needs Improvement Meets Expectations Exceeds Expectations Points 2. Fidelity of Implementation to Design 0123 2.c. Fidelity to the placement services design. The report presents qualitative and/or quantitative data on the grantee's level of implementation to the placement services design. The report presents no qualitative or quantitative data on the grantee’s level of fidelity of implementation to the placement services design. The report presents minimal qualitative or quantitative data on the grantee’s level of fidelity of implementation to the placement services design. The report presents qualitative or quantitative data on the grantee’s level of fidelity of implementation to the placement services design. The report presents qualitative and quantitative data on the grantee’s level of fidelity of implementation to the placement services design. x 57

58. Fidelity of Implementation to Design, continued Component 2 Fidelity of Implementation to Design Management Plan Evaluation Component Does Not Meet Expectations Needs Improvement Meets Expectations Exceeds Expectations Points 2. Fidelity of Implementation to Design 0123 2.d. Fidelity to management plan design. The report presents qualitative and/or quantitative data on the grantee's level of fidelity of implementation to the management plan design. The report presents no qualitative or quantitative data on the grantee’s level of fidelity of implementation to the management plan design. The report presents minimal qualitative or quantitative data on the grantee’s level of fidelity of implementation to the management plan design. The report presents qualitative or quantitative data on the grantee’s level of fidelity of implementation to the management plan design. The report presents qualitative and quantitative data on the grantee’s level of fidelity of implementation to the management plan design. x 58

59. Fidelity of Implementation to Design, continued Component 2 Fidelity to Implementation of Design Recruitment Plan Evaluation Component Does Not Meet Expectations Needs Improvement Meets Expectations Exceeds Expectations Points 2. Fidelity of Implementation to Design 0123 2.e. Fidelity to the recruitment plan design. The report presents qualitative and/or quantitative data on the grantee's level of fidelity of implementation to the recruitment plan design. The report presents no qualitative or quantitative data on the grantee’s level of fidelity of implementation to the recruitment plan design. The report presents minimal qualitative or quantitative data on the grantee’s level of fidelity of implementation to the recruitment plan design. The report presents qualitative or quantitative data on the grantee’s level of fidelity of implementation to the recruitment plan design. The report presents qualitative and quantitative data on the grantee’s level of fidelity of implementation to the recruitment plan design. x 2. Sub-Total0 59

60. Effectiveness of Project Design Component 3 Effectiveness of Project Design Instructional Services Evaluation Component Does Not Meet Expectations Needs Improvement Meets Expectations Exceeds Expectations Points 3. Effectiveness of Project Design 0123 3.a. Instructional services design. The report analyzes the effectiveness of the grantee's instructional services design. Report does not include a quantitative analysis of the impact of key instructional services upon GPRA results. Report includes a quantitative analysis of the impact of some instructional services upon GPRA results. Report includes a quantitative analysis of the impact of key instructional services upon GPRA results. Report includes a quantitative analysis of the impact of all instructional services upon GPRA results. x 3.a.2. The report includes data on the type and usage of eligibility screening tools. Report does not include information on tools for eligibility screening. Report includes minimal data on the type and usage of eligibility screening tools and/or indicates an open enrollment process. Report includes general data on the type and usage of eligibility screening tools and/or links an open enrollment process to the grant application. Report specifies specific tools used for eligibility screening, procedures and timelines for screening, and the effect on performance. x 60

61. Effectiveness of Project Design, continued Component 3 Effectiveness of Project Design Instructional Services Evaluation Component Does Not Meet Expectations Needs Improvement Meets Expectations Exceeds Expectations Points 3. Effectiveness of Project Design 0123 3.a.3. The report includes data on how instructional requirements of individual students are assessed and met. Report does not include data on individual student instructional needs. Report presents minimal data on individual student instructional needs. Report presents general data on individual student instructional needs and how the needs are met. Report provides extensive data on assessment of individual student instructional needs and the resources dedicated to meet the identified needs. x 61

62. Effectiveness of Project Design, continued Component 3 Effectiveness of Project Design Support and Placement Services Evaluation Component Does Not Meet Expectations Needs Improvement Meets Expectations Exceeds Expectations Points 3. Effectiveness of Project Design 0123 3.b. Support services design. The report analyzes the effectiveness of the grantee's support services design. No student survey data included. Student survey data included, items align minimally with services, and survey data analyzed. Student survey data included, items align mostly with services, and survey data analyzed. Student survey data included, items align completely with services, and survey data is analyzed for effectiveness of the support services. x 3.c. Placement services design for HEP projects No student survey data included. Student survey data included, items align minimally with grantees placement services design plan. Student survey data included, items align partially with placement services design plan. Student survey data included, items align completely with placement services design plan. x 62

63. Effectiveness of Project Design, continued Component 3 Effectiveness of Project Design Management Plan Evaluation Component Does Not Meet Expectations Needs Improvement Meets Expectations Exceeds Expectations Points 3. Effectiveness of Project Design 0123 3.d. Management plan design. The report analyzes the effectiveness of the grantee's management plan design. No employee survey data included. Employee survey data included, items align minimally with management plan design and survey data analyzed. Employee survey data included, most items align with management plan design and survey data analyzed. Employee survey data included, items align completely with management plan design and survey data analyzed. x 3.d.1. Teaching and administrative staff structure. The report presents no breakdown data on staffing structure. The report presents minimal breakdown data on staffing structure. The report presents data on staffing structure. The report presents quantitative data on staffing structure, including full-time equivalents (FTEs) in each functional area. x 63

64. Effectiveness of Project Design, continued Component 3 Effectiveness of Project Design Management Plan Evaluation Component Does Not Meet Expectations Needs Improvement Meets Expectations Exceeds Expectations Points 3. Effectiveness of Project Design 0123 3.d.2. Senior project administrative staff. The report provides no description for senior administrative staff. The report provides minimal discussion about the senior administrative staff. The report identifies the senior administrative staff. The report identifies the senior administrative staff and their qualifications for project roles. x 3.d.3. Qualifications of project instructors. The report presents no data on qualifications of instructors. The report present minimal data on qualifications of instructors. The report presents qualification data for some instructors. The report presents data on the qualifications of all project instructors. x 3.d.4. Professional staff development The report presents no data on professional staff development. The report presents minimal data on professional staff development. The report provides data on some professional staff development. The report presents extensive data on professional staff development. x 64

65. Effectiveness of Project Design, continued Component 3 Effectiveness of Project Design Management Plan Evaluation Component Does Not Meet Expectations Needs Improvement Meets Expectations Exceeds Expectations Points 3. Effectiveness of Project Design 0123 3.d.5. Student records The report provides no qualitative or quantitative data on student records. The report provides minimal qualitative and/or quantitative date on student records. The report provides qualitative and/or quantitative data on student records examined. The report provides extensive quantitative and qualitative data on the completeness and condition of student records examined. x 3.d.6. Approved procedures policy manual The report provides no information about the presence of a procedures policy manual. The report indicates there is a procedures policy manual. The report indicates the presence of an current policy manual. The report evaluates the quality and utility of a current policy procedures manual. x 3.d.7. Financial/cost sharing issues with host institution The report presents no information on cost sharing issues with the host institution. The report presents brief discussion about cost sharing issues with the host institution. The report discusses financial and cost sharing issues with the host institution. The report provides discussion of cost sharing issues and how to resolve. x 65

66. Effectiveness of Project Design, continued Component 3 Effectiveness of Project Design Management Plan Evaluation Component Does Not Meet Expectations Needs ImprovementMeets ExpectationsExceeds Expectations Points 3. Effectiveness of Project Design 0123 3.d.8. Student stipends The report provided no data on stipends. The report provided minimal data on availability and plan for distribution of stipends. The report provided data on the type of available stipends and distribution. The report described in detail the stipends, distribution schedules and any related issues. x 3.d.9. Planned and actual cost The report presents no quantitative data on the planned and actual annual cost per student The report presents minimal quantitative data on the planned and actual annual cost per student. The report presents quantitative data on the planned and actual annual cost per student as well as detailed information about any carry-over funds. The report presents detailed quantitative data on the planned and actual annual cost per student and identified reasons for any variances as well as detailed information about any carry-over funds. x 66

67. Effectiveness of Project Design, continued Component 3 Effectiveness of Project Design Recruitment Plan Evaluation Component Does Not Meet Expectations Needs ImprovementMeets ExpectationsExceeds Expectations Points 3. Effectiveness of Project Design 0123 3.e. Recruitment plan design. The report analyzes the effectiveness of the grantee's recruitment plan design. The report presents no information on the effectiveness of the recruitment plan design to enroll students. The report provides minimal information about the effectiveness of the recruitment plan design to enroll students. The report presents some qualitative information on the effectiveness of the recruitment plan for students. The report presents detailed qualitative information on the effectiveness of the recruitment plan designed to enroll the planned number of students. x 3.e.1. Demographic information The report provides no demographic data on students recruited. The report provides limited demographic data on the students recruited to participate. The report provides demographic data, including age and gender of recruited students. The report provides detailed demographic data for recruited students by age and gender and corresponding success rates. x 67

68. Fidelity of Implementation to Design, continued Component 3 Effectiveness of Project Design Recruitment Plan Evaluation Component Does Not Meet Expectations Needs Improvement Meets Expectations Exceeds Expectations Points 3. Effectiveness of Project Design 0123 3.e.2. Training for Recruiters The report does not include any training program for project recruiters. The report includes limited data on electronic and/or in-person training programs for project recruiters. The report provides general information on electronic and in-person training opportunities for recruiters, topics, timelines, and completion rates. The report includes detailed information on electronic and in-person training opportunities for recruiters, topics including outreach/enrollment techniques, timelines, completion rates, and evaluation of contribution of trainings to recruiters' success rates. x 3. Sub-Total0 68

69. Collaborative Agreements Component 4 Collaborative Agreements Existing and Planned Evaluation Component Does Not Meet Expectations Needs Improvement Meets ExpectationsExceeds Expectations Points 4. Collaborative Agreements 0123 4.a. Existing collaborative agreements The report provides no listing or analysis of existing collaborative agreements with educational institutions or service providers. The report lists existing collaborative agreements with educational institutions or service providers. The report provides some analysis of existing collaborative agreements with educational institutions or service providers. The report provides in- depth analysis of existing collaborative agreements with educational institutions or service providers. x 4.b. Planned collaborative agreements The report provides no analysis of planned collaborative agreements with educational institutions or service providers. The report provides names of planned collaborative agreements with educational institutions or service providers. The report provides analysis of some planned collaborative agreements with educational institutions or service providers. The report provides analysis of and reasons for additional planned collaborative agreements with educational institutions or service providers. x 69

70. Collaborative Agreements, continued Component 4 Collaborative Agreements Relationships with Host Institutions Evaluation Component Does Not Meet Expectations Needs Improvement Meets Expectations Exceeds Expectations Points 4. Collaborative Agreements 0123 4.c Relationships with host institutions, including unresolved facilities, equipment/computers, HR and related issues. The report presents no qualitative information on issues with host institutions that affect project performance The report presents minimal facilities, equipment/comput ers, HR and related issues with host institutions but does not discuss impact on program performance. The report presents any facilities, equipment/comput ers, HR and related issues with host institutions and discusses impact on program performance. The report presents facilities, equipment/computers, HR and related issues with host institutions that impact program performance and how they can be resolved. x 4. Sub-Total0 70

71. Component Recommendations Component 5 Recommendations Instructional and Support Services Evaluation Component Does Not Meet Expectations Needs Improvement Meets ExpectationsExceeds Expectations Points 5. Component Recommendations 0123 5.a. Instructional services The report presents no recommendations to improve GPRA 1 and 2 results. The report presents minimal recommendations to improve GPRA 1 and 2 results. The report provides some recommendations to improve GPRA 1 and/or 2 results that are linked to report information. The report provides extensive recommendations to improve GPRA 1 and 2 results that are linked to report information. x 5.b. Support Services The report presents no recommendations for support services. The report presents minimal recommendations for improving support services. The report provides some recommendations for improving support services that are linked to report information. The report provides extensive recommendations on improving support services that are linked to report information. x 71

72. Component Recommendations, continued Component 5 Recommendations Placement Services and Management Plan Evaluation Component Does Not Meet Expectations Needs Improvement Meets ExpectationsExceeds Expectations Points 5. Component Recommendations 0123 5.c. Placement services The report presents no recommendations for placement services. The report presents minimal recommendations on improving placement services. The report provides some recommendations for improving placement services that are linked to report information. The report provides extensive recommendations on improving placement services that are linked to report information. x 5.d. Management plan The report presents no recommendations for the management plan. The report presents minimal recommendations on improving the management plan. The report provides some recommendations for improving the management plan that are linked to report information. The report provides extensive recommendations on improving the management plan that are linked to report information. x 72

73. Collaborative Agreements, continued Component 5 Recommendations Recruitment Plan Evaluation Component Does Not Meet Expectations Needs Improvement Meets Expectations Exceeds Expectations Points 5. Component Recommendations 0123 5.e. Recruitment plan The report presents no recommendations for strengthening the recruitment plan. The report presents minimal recommendations on strengthening the recruitment plan for improved results. The report provides some recommendations for strengthening the recruitment plan results that are linked to report information. The report provides extensive recommendations on strengthening the recruitment plan to improve results that are linked to report information. x 5. Sub-Total0 TOTAL Rubric Score 73