No Child Left Behind Sub-grant Application Project Planning & Evaluation John Cradler Ruthmary Cradler Educational Support Systems

Evaluation and NCLB Grants Central to the design of the project Document project implementation Document student outcomes Document change in teaching Inform future project improvements Increase future funding possibilities 3

Lack of evidence about outcomes and impact of projects Lack of evidence relating project outcomes to project activities Over emphasis on subjective and anecdotal results Typical evaluation issues 4

Project Evaluation Process Activities JASONDJFMAMJ JASONDJFMAMJ Formative Evaluation Analysis Project Activities JASONDJFMAMJ Project Implementation Summative Evaluation Recommended Project Change 5

Project Planning and Evaluation I. System Conditions III. Instructional Consequences II. Instructional Interventions IV. Learning Outcomes Instructional Change Technology Integration School Conditions District Conditions NCLB Intervention Develop Implement Evaluate Student Change Academic Behavioral Tech-use

I. Evaluation Readiness A. Evaluation Pre-planning 1. Project faculty & administrators understand the necessity to evaluate 2. Sufficient resources have been made available for evaluation 3. Evaluation planning is part of project planning 4. NCLB Evaluation Guidelines inform the development of the evaluation plan 6

I. Evaluation Readiness B. Project Readiness 1. Context for the project is described 2. Teacher technology standards are considered 3. Faculty needs & resources are identified 4. Population(s) served are defined 5. Objectives are clear & related to needs 7

I. Evaluation Readiness B. Project Readiness (cont.) 6. Activities are linked to objectives 7. Activities are scheduled 8. Project products, services, etc. are defined 9. Project personnel are identified 10.Training & resources are described 8

I. Evaluation Readiness C. Develop Evaluation Questions 1. Questions address each project objective 2.Questions can be answered by the evaluation strategies and assessments 3.Questions address extent of project implementation (inputs) 4. Questions address outcomes (outputs) resulting from project interventions (inputs 9

II. Input Evaluation Questions A. Implementation Conditions 1. Faculty & stakeholders support project 2. Project is coordinated with existing programs 3. Resources to implement are available 4. Adequate time for project activities is provided 5. Partner support & resources are evidenced The extent to which... 11

II. Input Evaluation Questions B. Implementation Progress 1.Activities were implemented as planned 2.Professional development was implemented as planned 3. State curriculum standards were applied 4.Faculty implemented project specific activities The extent to which... 12

II. Input Evaluation Questions B. Implementation Progress (cont.) 5.Mid-course adjustments were made as needed 6.Faculty observed engaging in project activities 7.Resources & practices with greatest influence were identified 8.Activities unanticipated or unique to particular projects were implemented The extent to which... 13

III. Output Evaluation Questions A. Instructional Change 1.Faculty integrate technology into curriculum 2.Technology resources changed instruction 3. Technology resources supported the use of state curriculum standards 3. Unanticipated faculty changes occurred The extent to which... 14

III. Output Evaluation Questions B. Student Change 1. Increases in student performance were documented 2.Student use of technology increased learning opportunities 3. Student motivation was increased as a result of the project 4. Unanticipated student change was observed The extent to which... 15

IV. Evaluation & Analysis Strategies B. Data Analysis & Reporting 1.Implementation conditions are related to level of implementation 2.Interventions are related to outcomes 3.Barriers and facilitating factors are discussed 4.Ongoing & end-project recommendations are made 5.Reports are prepared for stakeholders 6.Analysis determines promising practices for future implementation 18

Causal Map for Overall Project Evaluation GEPRA Indicators Technology Standards (four pillars/ISTE) Federal Evaluation Guidelines Goals & Objectives Report Findings to USDOE & Congress Analysis of overall impact Recommend Project Revisions Overall Progress Assessment Faculty Development Project Development & Implementation Student Change Instructional Change Outcome Evaluation Individual Project Evaluations Individual Project Evaluations Individual Project Evaluations Individual Project Plans & Causal Maps Individual Project Plans & Causal Maps Individual Project Evaluations Individual Project Plans & Causal Maps Aggregated Project Evaluation Individual Project Plans & Causal Maps 21

Advanced technology delivering traditional instruction Computer assisted instruction supplementing traditional instruction Technology as a supplement to systemic education reform initiatives Passive learningEngaged learning Low technology High technology Advanced technology integral to systemic education reform initiatives Authentic Performance & project-based research & evaluation Of highly infused technology Weak & Limited research With low technology & skill-based learning Educational Support Systems, 1998 Technology Integration Analysis Technology Driven & Low Education Integration Research Authentic Performance & project-based research & evaluation with very limited technology infusion Analysis of technology in teaching and learning in relation to instructional approach 22