1. Evaluation Tools & Approaches for Engaging Stakeholders: The AREA Approach Presentation at the Canadian Evaluation Society Dr. Mariam Manley, Principal Evaluator Mid-Continent Research for Education & Learning Mmanley@mcrel.org June 1, 2003

2. What is the Advanced Reading Enhancement Approach (AREA)? AREA is a statewide program designed to help ensure that, by the third grade, all South Dakota students are reading and writing at grade level. The AREA project trains and mentors teachers over an entire school year and consists of the following elements:  45 hours of direct instruction to teachers  18-22 hours of administrator training  Ongoing mentoring & coaching  Provision of resources & materials  Student Assessment

3. Stakeholder Involvement … Do I REALLY want to do this?! External, third party Objectivity Validity Reliability Time Consensus (yikes!) Friend/Foe Useful Meaningful Ownership Cost Effective Educational

4. Methods of Stakeholder Involvement  Logic Model and evaluation design  Instrument development  Data collection  Design & management of data collection system  Feedback mechanisms  Reporting

5. DEVELOPMENT OF A PROGRAM LOGIC MODEL Resources/Inputs  Activities  Short-term  Intermediate  Long-term Impact External conditions influencing success  Understanding & “Big Picture”  Identification of outcomes  Links between activities and outcomes  Communication  Formative  Identify evaluation questions and sources of existing information  Reporting

6. What types of information were collected as part of the AREA evaluation?  Documentation and ongoing monitoring of program implementation  Pre and post teacher & administrator surveys  Classroom observations  Student assessment data  Site visits & interviews

7. INSTRUCTIONS FOR USE What is the sequence of events in using these instruments? 1. Arrange the observation session(s) with the teacher 2. Ask a pre-observation question (2 minutes or so) 3. Observe the instructional area to be focused upon 4. Conduct a post-observation interview (approximately 15 minutes) 5. Complete the observation form 6. Enter the rating form into the web-based data collection system