2014 AmeriCorps External Reviewer Training Assessing Need, Theory of Change, and Logic Model of Tier 1 Education Applications 2014 AmeriCorps External Reviewer Training

Topics Purpose of Logic Model Assessing Need, Theory of Change, and Logic Model

Preview of Assessment Questions (True/False) The Logic Model is a visual representation of the applicant’s Theory of Change External Reviewers should assess whether the outcomes in the Logic Model are measurable (Multiple Choice) Data about the number of economically disadvantaged beneficiaries in a community would most likely describe: (Practice Your Understanding) Complete portion of Review Assessment Form

Logic Model What is a Logic Model? A visual summary or snapshot of your program that communicates how your program works, the resources you have to operate your program, the activities you plan, and the outcomes you hope to achieve A well-specified conceptual framework that identifies key components of the proposed process, product, strategy, or practice and describes the relationships among the key components and outcomes, theoretically and operationally (Notice of Funding Opportunity (NOFO) p. 18) The purpose of a Logic Model is to describe how a program will create change.

Certain resources are needed to operate your program Logic Model How a Logic Model Works Figure 1: How to Read a Logic Model If these benefits to participants are achieved, then certain changes in organizations, communities, or systems might be expected to occur If you accomplish your planned activities, then you will hopefully deliver the amount of product and/or service that you intended If you accomplish your planned activities to the extent you intended, then your participants will benefit in certain ways If you have access to them, then you can use them to accomplish your planned activities Certain resources are needed to operate your program Resources/ Inputs Activities Outputs Outcomes/ Impacts Long-Term Goal j k l m n Your Planned Work Your Intended Results

Applicant Logic Models Inputs – Resources, including AmeriCorps members Activities (Interventions) Outputs (Beneficiaries, other products of program services) Outcomes Short Term (Knowledge, Skills, Attitudes, Opinions) Medium Term (Behavior) Long-Term (Condition)

Logic Model Key Points The Logic Model and narrative work together to depict the program’s Theory of Change All the information in one doesn’t have to be in the other, as long as the two are aligned The Logic Model is simply a visual representation of the Theory of Change and program design Applicants do not measure all outcomes in Theory of Change; they continue to measure outcomes that make sense for their program; External reviewers do not assess applicants’ performance measures and so should not be concerned with whether outcomes are measurable CNCS provided a template for applicants’ Logic Models

Theory of Change Elements Assessment: Theory of Change Theory of Change Elements Community Problem/Need Specific Intervention Intended Outcome Statistics documenting the need Evidence Guides choice of intervention Supports cause-effect relationship

How Do The NOFO Pieces Fit? Assessment How Do The NOFO Pieces Fit? Applicant’s Overall Theory of Change Need narrative Theory of Change narrative Logic Model

The Selection Criteria & Standards Assessment The Selection Criteria & Standards Four Key Themes Completeness (Logic Model) Alignment/Logic Quality of Theory of Change Level/Quality of Evidence

Review Rubric Most items scored on a five-point scale Assessment Review Rubric Most items scored on a five-point scale Some items scored on a two-point scale (Satisfactory/Unsatisfactory) Refer to the Review Rubric in the handbook to read the descriptions for each Rating

Completeness (Logic Model) Assessment Completeness (Logic Model) Number of locations and sites where members are providing services Number of members that will be delivering the intervention Core activities that define the intervention or program model Duration of the intervention Dosage of the intervention Target population for the intervention

How Assessed Series of check boxes on the review form Assessment How Assessed Series of check boxes on the review form Reviewers check the box if the item is present in the Logic Model

Eligible Intervention(s) in Logic Model or Narrative Assessment Eligible Intervention(s) in Logic Model or Narrative Reviewers will also indicate whether or not the applicant’s Logic Model and/or Narrative incorporates at least one of the following six required member interventions for Tier 1 Education applicants (listed on this slide and the next slide): 1) Providing ongoing mechanisms for family and community engagement 2) Establishing a school culture and environment that improves school safety, attendance, and discipline and addresses other non-academic factors that impact student achievement, such as students’ social, emotional, and health needs 3) Accelerating students’ acquisition of reading and mathematics knowledge and skills

Eligible Intervention(s) in Logic Model or Narrative Assessment Eligible Intervention(s) in Logic Model or Narrative 4) Increasing graduation rates through strategies such as early warning systems, credit-recovery programs, and re-engagement strategies. 5) Increasing college enrollment rates through college preparation counseling assistance to include completing the Free Application for Federal Student Aid (FAFSA) and college applications, and educating students and their families on financial literacy for college 6) Supporting school implementation of increased learning time

Alignment with Logic Model Assessment: Theory of Change Alignment with Logic Model The Theory of Change is described consistently in the Logic Model and application narrative The inputs, activities and outputs are logically aligned The Theory of Change and Logic Model cover comprehensively the applicant’s entire program

Quality of Theory of Change Assessment: Theory of Change Quality of Theory of Change Problem The applicant cites specific, relevant data to document the needs that the AmeriCorps members will be addressing and these needs were identified with the involvement of eligible partner school(s) and LEA leadership The need in community and in the school(s) where the proposed AmeriCorps members will serve is prevalent and severe Intervention The potential contribution of AmeriCorps members is an effective way to address the needs identified by eligible school and LEA leadership The project effectively addresses multiple student needs Outcomes The outcomes identified in the Logic Model are logically aligned with the problem/need and intervention

Assessing Alignment and Quality Assessment: Theory of Change Assessing Alignment and Quality Alignment and Quality are assessed using a five-point scale: Greatly exceeds the standard Exceeds the standard Meets the standard Partially meets the standard Does not meet the standard at all Refer to the Review Rubric to read the descriptions for each Rating.

Need vs. Evidence Applicants are asked to provide data to demonstrate: Assessment: Need Need vs. Evidence Applicants are asked to provide data to demonstrate: Prevalence and severity of need versus the level and quality of evidence Specific, relevant data to document the need Evidence to support the choice of intervention

Community and School Problem/Need Assessment: Need Community and School Problem/Need Data documenting problem/need should demonstrate the prevalence and severity of the problem: Who and how many are directly affected? How severe is the problem?

Examples Graduation rate Assessment: Need Examples Graduation rate Percent of students eligible for free/reduced lunch Student academic achievement data Data about the school culture or environment Data about family engagement Student attendance data

Assessment: Need Data Should Be From a reputable source such as a government agency, foundation, university or research organization Recent Local

Assessment: Need Common Issues State or national data is provided instead of local data describing the need in the specific communities where the service will occur Statistics are unrelated to the problem/need Statistics cited are not recent Data sources are not identified or are not reputable

Next Steps To check for understanding and verify that you have completed this orientation session, please reference the remaining slides for the two-part Assessment of this Orientation.

Practice Your Understanding Assessment: Practice Practice Your Understanding Answer the Assessment Questions at the following link: https://www.surveymonkey.com/s/logicmodels_assessment Read and assess the norming/sample application and complete the Rating sections for the Problem/Need, Theory of Change, and Logic Model sections of the review form. Do not include significant strengths, weaknesses, or other narrative sections. You will receive a review form completed by a CNCS staff member. Compare your assessments to those in the key. This will complete your verification requirement for this training module.

Assessment: Practice Practice SELF-ASSESSMENT When you receive the key for the practice activity from CNCS, compare your Ratings to those of the CNCS standard example. If your Rating was more than one Rating higher or lower than the example provided by CNCS, your assessment may not be normed closely enough to CNCS’s expectations for applicants