Beginning with the End in Mind (aka the latest and greatest on AmeriCorps performance measurement) 2011 NY Project Director Training – New Rochelle October.

Slides:



Advertisements
Similar presentations
Numbers Treasure Hunt Following each question, click on the answer. If correct, the next page will load with a graphic first – these can be used to check.
Advertisements

Jack Jedwab Association for Canadian Studies September 27 th, 2008 Canadian Post Olympic Survey.
Symantec 2010 Windows 7 Migration EMEA Results. Methodology Applied Research performed survey 1,360 enterprises worldwide SMBs and enterprises Cross-industry.
1 A B C
Designing Effective Action for Change
AP STUDY SESSION 2.
1
Evidence: What It Is And Where To Find It Evidence: What It Is and Where to Find It How evidence helps you select an effective intervention.
Evidence 2/8/2014 Evidence 1 Evidence What is it? Where to find it?
Overview of Performance Measurement. Learning Objectives By the end of the module, you will be able to: Describe what performance measurement is, and.
Performance Measurement and Evaluation 2/8/2014 Performance Measurement and Evaluation 1 Performance Measurement and Evaluation What are they? How are.
High Quality Performance Measures What is Performance Measurement? What makes a high quality performance measure? Copyright © 2012 by JBS International,
An overview of the National Performance Measures 118 NOVEMBER 2011 THE CORPORATION FOR NATIONAL & COMMUNITY SERVICE.
An overview of the National Performance Measures 118 NOVEMBER 2011 THE CORPORATION FOR NATIONAL & COMMUNITY SERVICE.
Theory of Change 2/8/2014 Theory of Change 1 Theory of Change What is it? How to use it?
Serve DC Training 9/28/11. Theories of Change and Logic Models Evidence Performance Measurement 101 Reviewing Performance Measures eGrants Tips Serve.
Performance Measurement Overview 2/8/2014 Performance Measurement Overview 1 What is it? Why do it?
Alignment and Quality 2/8/2014 Alignment and Quality 1 What is it? What do I need to look for?
1 Creating Strong Reports New Mexico AmeriCorps April 20, 2006 Sue Hyatt, Project STAR Coach.
ASTM Member Website Tools Jeff Adkins Diane Trinsey 1 September 2012 Officers Training Workshop.
David Burdett May 11, 2004 Package Binding for WS CDL.
NTDB ® Annual Report 2009 © American College of Surgeons All Rights Reserved Worldwide Percent of Hospitals Submitting Data to NTDB by State and.
Teaching and Learning Science & Assessment Informational Webinars Presenter: Linda Cabe Smith, Science Assessment Specialist Ellen Ebert, Science Director,Teaching.
Implementing RTI Using Title I, Title III, and CEIS Funds Key Issues for Decision-makers U.S. Department of Education 1.
1 DPAS II Process and Procedures for Teachers Developed by: Delaware Department of Education.
Create an Application Title 1Y - Youth Chapter 5.
CALENDAR.
1 Screening & Eligibility Jeopardy Title V/PHC.
1 Click here to End Presentation Software: Installation and Updates Internet Download CD release NACIS Updates.
The 5S numbers game..
A Fractional Order (Proportional and Derivative) Motion Controller Design for A Class of Second-order Systems Center for Self-Organizing Intelligent.
Chapter 7: Steady-State Errors 1 ©2000, John Wiley & Sons, Inc. Nise/Control Systems Engineering, 3/e Chapter 7 Steady-State Errors.
The SCPS Professional Growth System
Break Time Remaining 10:00.
Turing Machines.
Table 12.1: Cash Flows to a Cash and Carry Trading Strategy.
PP Test Review Sections 6-1 to 6-6
1 The Royal Doulton Company The Royal Doulton Company is an English company producing tableware and collectables, dating to Operating originally.
Exarte Bezoek aan de Mediacampus Bachelor in de grafische en digitale media April 2014.
TESOL International Convention Presentation- ESL Instruction: Developing Your Skills to Become a Master Conductor by Beth Clifton Crumpler by.
Copyright © 2012, Elsevier Inc. All rights Reserved. 1 Chapter 7 Modeling Structure with Blocks.
1 TV Viewing Trends Rivière-du-Loup EM - Diary Updated Spring 2014.
Adding Up In Chunks.
FAFSA on the Web Preview Presentation December 2013.
SLP – Endless Possibilities What can SLP do for your school? Everything you need to know about SLP – past, present and future.
MaK_Full ahead loaded 1 Alarm Page Directory (F11)
Facebook Pages 101: Your Organization’s Foothold on the Social Web A Volunteer Leader Webinar Sponsored by CACO December 1, 2010 Andrew Gossen, Senior.
Artificial Intelligence
Evidence: What It Is And Where To Find It Evidence: What It Is and Where to Find It How evidence helps you select an effective intervention Copyright ©
Before Between After.
7/16/08 1 New Mexico’s Indicator-based Information System for Public Health Data (NM-IBIS) Community Health Assessment Training July 16, 2008.
Subtraction: Adding UP
: 3 00.
5 minutes.
1 hi at no doifpi me be go we of at be do go hi if me no of pi we Inorder Traversal Inorder traversal. n Visit the left subtree. n Visit the node. n Visit.
Types of selection structures
Speak Up for Safety Dr. Susan Strauss Harassment & Bullying Consultant November 9, 2012.
1 Phase III: Planning Action Developing Improvement Plans.
Converting a Fraction to %
Numerical Analysis 1 EE, NCKU Tien-Hao Chang (Darby Chang)
Clock will move after 1 minute
Physics for Scientists & Engineers, 3rd Edition
Select a time to count down from the clock above
Copyright Tim Morris/St Stephen's School
1.step PMIT start + initial project data input Concept Concept.
Presented to: By: Date: Federal Aviation Administration FAA Safety Team FAASafety.gov AMT Awards Program Sun ‘n Fun Bryan Neville, FAASTeam April 21, 2009.
Performance Measures AmeriCorps Project Director Training Saratoga, NY October 7 th – 9 th, 2013.
Performance Measurement Overview 9/18/2015 Performance Measurement Overview 1 What is it? Why do it?
Presentation transcript:

Beginning with the End in Mind (aka the latest and greatest on AmeriCorps performance measurement) 2011 NY Project Director Training – New Rochelle October 26, 2011 Susan Hyatt – JBS 1

Session Agenda Why Measurement Matters CNCS Requirements for Performance Measurement Theory of Change for AmeriCorps Program Interventions Evidence-basis for Interventions The Must Haves - Performance Measurement and Evaluation (similarities and differences) How to Review PMs for Alignment 2

3

AmeriCorps GARP Tiers Tier 1: Programs that select Priority Measures in Education, Veterans and Military Families, and Disaster Services Tier 2: Programs that select Priority Measures in Economic Opportunity, Environmental Stewardship, Healthy Futures and Capacity Building. Tier 3: Programs that select Pilot Measures. Tier 4: Programs in the Focus Areas with self-­nominated measures. Tier 5: Programs outside the Focus Areas with self-­ nominated measures. 4

Why Measurement Matters Touch Points During Year Strategy Development Strategic planning of program Program Implementation Annual program year launch Site monitoring and data validation Site T/TA provision Continuous improvement Using Results/Data and Reporting Quarterly/mid-year/end of year reporting Stakeholder reporting/promotion - Telling the Story Review of annual achievements/performance outcomes Celebrate success!! 5

What is a Theory of Change? A theory of change looks at cause and effect relationships and identifies specific interventions to achieve the desired result If the INTERVENTION (X) is delivered at a certain dosage, then the expected OUTCOME (Y) will happen. X Y 6

Theory of Change: Components PROBLEM: The identified community need INTERVENTION: The National Service participant (and community volunteer) activities delivered (what is done, with whom, and at what dosage) OUTCOME: The change that happens because of the intervention EVIDENCE: Why you believe a certain set of actions (the intervention) will lead to the intended outcome 7

An Example From Everyday Life I have strep throat (PROBLEM). I want to kill the germs (GOAL) If I take antibiotics (INTERVENTION)… Which antibiotics fight strep the best (Evidence) then I will get better (OUTCOME). Antibiotics I get better. X Y 8

Is This Always True? If I take penicillin, I will get better. If I take a different antibiotic, will I get better? Some interventions (antibiotics) work better than others. Some dont work at all. 9

How Do I Know? How do I know which antibiotic is best? I look at the evidence. There is research that shows which antibiotic is likely to get the best result. I consider constraints that may preclude the ideal intervention. (Penicillin may be too expensive.) If I cant have the most promising intervention, I need to understand the tradeoffs. 10

Reflection: Theory of Change What is your programs theory of change? 11

What Evidence Do We Need? Two types of evidence are required: 1.Data that documents the community need; and 2.Data that documents why you think your education- related intervention (using National Service participants and community volunteers) will achieve the intended outcome. 12

What is Evidence? Data that demonstrates that the proposed intervention is likely to solve the identified problem. For example: Evidence says that x hours of tutoring leads to academic outcomes … so … the chosen intervention features x hours of tutoring a 3rd grader so that the 3rd grader will meet grade level standards. 13

Evidence Basis for An Intervention It is NOT enough to just say that the intervention is likely to be successful. The evidence basis for an intervention may include: o Past performance measurement data; o Results from a program evaluation; o Research studies that document the outcomes of similar programs; and o Evaluations that document outcomes of similar programs. 14

Evidence Basis for An Intervention 1. Past performance measurement data: What does your past PM data tell you? Do you have multiple years of data you can aggregate? Are you getting the most mileage from how you present your past PM data? 15

Evidence Basis for An Intervention 2. Results from a program evaluation: Have you done an impact evaluation (as opposed to a process evaluation)? Were the results positive? Does it clearly show that your intervention is what created/caused the change? 16

Evidence Basis for An Intervention 3. Research studies that document the outcomes of similar programs; and 4. Evaluations that document outcomes of similar programs. 17

Where to Look for Evidence? A Scavenger Hunt…???!! 18

19

NWREL is a well known research organization working with education programs Has been involved with National Service and CNCS It is an evaluation Gives tools for action 20

Google Tip - Did You Know? 21

Google Scholar Option 22

23

Advanced Scholar Research 24

What to Look For? University or research organizations ( National or local) Names of Known Professionals/Thought Leaders Similar sounding programs/descriptions Meta-articles that review multiple studies 25

26

27

28

29

What Did We Learn? Intervention specifics including dosage required for desired outcomes Are our output and outcome targets reasonable (given our population and dosage)? Do we need to tweak our intervention to make it more effective in creating our desired outcomes? Do we need to find go back to the drawing board and retool our intervention? 30

Measurement Must Haves 1.Performance Measurement Counts numbers served Captures near term changes in beneficiaries on an annual basis 2.Evaluation – 2 types (Impact) Captures longer term changes that occur as a direct result of your program intervention (Process) Gives insight into aspects of implementation 31

Differences Performance Measurement Impact Evaluation Tracks outputs and outcomes on a regular, ongoing basis Does not show causality Seeks to show causality or prove theory of change Longer term focus Uses the most rigorous form of evaluation that is right for the program (often quasi- experimental design) 32

Example: Minnesota Reading Corps Performance Measurement: Individual benchmark assessments on Dynamic Indicators of Basic Early Literacy Skills (DIBELS) three times/year State Reading Exam --Number of students who graduate from the Minnesota Reading Corps who pass state reading exam 33

Example: Minnesota Reading Corps Impact Evaluation: Matched sample research project in Minneapolis School District Reading Corps pre-school participants scored significantly higher in phonemic awareness, alphabetic principle, and total literacy than children in matched comparison group entering kindergarten 34

EXAMPLE: Minnesota Reading Corps A strategic initiative of ServeMinnesota 35

3 rd Grade: A Critical Turning Point PreK – 3 rd grade Learn to Read 4 th grade – 12 th Read to Learn 36

Vision: All Minnesota children will become proficient readers by the end of 3 rd grade. 37

38

Supporting Age 3 to Grade 3 Students K-3 Settings: Member does one-on- one, 20-minute tutoring sessions throughout the day Literacy support provided during: One-on-one tutoring PreK Settings: Member placed in a PreK classroom to support students throughout the day Literacy support provided during: Whole classroom Small group time One-on-one tutoring 39

K-3 Program Model Step 1: Conduct benchmark assessments in fall Step 2: Select children to receive tutoring Step 3: Select intervention for each student Step 4: Begin tutoring! Step 5: Monitor the progress of each student weekly Step 6: Conduct benchmark assessments in winter & spring 40

Our model is based on … Research 41

Assessment Instruction Tier 1: 75-80% Universal / Core Tier 2: 15-20% Strategic / Supplemental Tier 3: 5-10% Intensive Sweet Spot 42

Our model is … Laser- Focused & Scripted 43

44

Our model uses … Data (constantly) 45

46

47

Our model doesnt just train, we … Coach 48

A model of support to ensure student success Master Coach Program Staff Internal Coach Reading Corps Tutor Students 49

Need Intervention/Strategy Etc. I. Between Narrative and PMs (Theory of Change) II. Between Results III. Within Results Three Types of Alignment 50

Example for Reviewing the Three Types of Alignment – Output: Number of students that completed participation in CNCS-supported K-12 education programs (ED2) – Outcome: Number of students that improved their school attendance over the course of the CNCS- supported programs involvement with the student (ED6) 51

Alignment (I) – Between Narrative and PMs – Need: Documented high rates of chronic absence from school (that can lead to other problems...) – Intervention/Strategy (ToC): Members provide individual and group mentoring to students with documented attendance problems. Mentoring focuses on promoting re-engagement with school. – Output: Students participate in mentoring – Outcome: Students improve school attendance 52

Alignment (II) – Between Results – Activity: Members mentor students with documented attendance problems. – Output: Students with documented high rates of chronic absence from school participate in mentoring. – Outcome: Students improve school attendance 53

Alignment (III) – Within Results – Result (outcome): Students improve school attendance – Indicator: Number of students with improved school attendance at end of school year – Target: 300 of 400 (75%) participating students – Instrument/Method: Log of Student Attendance and Absences (compared to baseline data from school) 54

Practice, Part 1: Review Sample Application Narrative for PM-related Information 1. On your own, read application narrative Use PM Assessment Checklist and Considerations, Definitions and Tips to identify issues and concerns for clarification. What do you think the programs PMs are? 2. In small groups, discuss PM Checklist responses. Identify issues and areas where you agree and differ. Can you come to consensus on ratings and questions? 55

Assessing Quality and Rigor: Look for evidence that: – Beneficiaries are selected according to need-based criteria (e.g. instructions for national PMs) – Intervention is supported by evidence. Consider if evidence is preliminary, moderate or strong. – Transparent reporting; clear how results are obtained – Instrument are valid and reliable – Data collection is rigorous 56

Heads Up: CNCS Verification and Validation 57

Its not just the tool you use….are your data collection methods rigorous? Look for: – Feasibility – Prior successful use (or pilot tested) – Data sources and respondents clearly identified – Training of data collectors – Prevention of output duplication 58

Practice, Part 2: Review Sample PMs 1.On your own, read PMs in application and use checklist and definitions to see whats there and whats missing. Identify clarification questions. 2.In small groups, discuss your PM checklist responses and questions; areas where you agree and differ. Step back and consider application review process and your organizations practices 59

Resources 2012 AmeriCorps National PMs Instructions – (nationalservice.gov) Resource Center – Most Recent PM Materials: me – AC Pilot Measures: -performance-measures/home 60