INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology Using Evidence for Educational Technology Success FETC 2005, Orlando, January 27, 2005 Talbot Bielefeldt and Don Knezek International Society for Technology in Education

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology What is Ed Tech Success? Predicated on essential conditions Requires evaluation at each stage

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology NETS Essential Conditions Vision with support and proactive leadership from the education system Educators skilled in the use of technology for learning Content standards and curriculum resources Student-centered approaches to learning Assessment of the effectiveness of technology for learning Access to contemporary technologies, software, and telecommunications networks Technical assistance for maintaining and using technology resources Community partners who provide expertise, support, and real-life interactions Ongoing financial support for sustained technology use Policies and standards supporting new learning environments

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology Entering the Evaluation Model Predicated on essential conditions Requires evaluation at each stage

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology Integration Questions What learning activities do teachers design? What instructional strategies do they use? What technologies do they use? How often do they use these approaches? Are there constraints, such as tech support or professional development? What distinguishes these approaches from alternatives?

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology Integration Data Lesson plans. Classroom observations. Surveys and interviews. Training records and observations. Technology inventories and service records. Content standards and curriculum guides.

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology Intermediate Outcomes Predicated on essential conditions Requires evaluation at each stage

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology Student Experience Questions To what extent are students engaged in the integrated learning activities? Are technology-integrated learning activities accessible to all? What other learning activities complement or compete with the technology-based approaches?

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology Student Experience Data Classroom observations of technology-integrated and comparison classrooms. Surveys and interviews of teachers and students. Technology inventories.

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology Student Learning Outcomes Predicated on essential conditions Requires evaluation at each stage

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology Behavior/Achievement Questions Do valid and reliable measures of behavior and achievement exist? Do students with technology-integrated learning experiences behave or perform differently than other students? What other explanations of differences exist, besides the technology-based learning?

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology Behavior/Achievement Data Lesson plans and lesson observations Standards and curriculum guides Assessment instruments, local and standardized Research designs that establish causality

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology Started with a grant from the Bill & Melinda Gates Foundation. A partnership of ISTE and Educational Support Systems. A review process + online documents.

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology CARET Process Identify critical questions and survey the literature. Apply the CARET criteria for identifying the most credible and relevant studies.

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology CARET Criteria A-B A. The study addresses important educational issues 1. At least one of the critical questions listed in Section II is clearly addressed 2. The results and recommendations from the study can help inform planning decisions 3. The results are explained in a way that answers an important educational question B. Objectives, interventions, and evaluation questions 1. The objectives for the study are clearly stated and related to documented needs 2. The objectives address local, state and/or national curriculum and/or performance standards 3. The interventions and related activities for all participants are linked to the project objectives 4. The study defines specific products, services, policies, and/or practices to be studied. 5. An explicit description of the technology applications being studied is provided 6. Evaluation and/or research questions or hypothesis link to the objectives of the study

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology CARET Criteria C-D C. Context for the study 1. Level of district and school support for the project are addressed 2. Population demographics related to students, community, other participants are described 3. The geographical setting and location(s) in terms of rural, suburban, urban are described 4. Extent of access to resources relevant to implementation of the intervention are considered 5. Community support for the project is addressed as appropriate 6. Extent of teacher professional development and ongoing support is defined 7. Level of necessary technical/administrative/funding support is addressed D. Evaluation and Research Design 1. A needs assessment guided the objectives of the study 2. Clear definition of problem and/or evaluation or research questions is present 3. The study builds upon related prior research and evaluation studies 4. The study is designed to directly answer research/evaluation questions presented 5. The study provides an experimental design to statistically test a hypothesis 6. Comparison groups to control for extraneous variables is incorporated in the study 7. The study provides a rationale for use of a particular research or evaluation approach 8. A description and analysis of the outcomes or impact on participants is provided 9. A formative assessment of the level of implementation of the intervention is present

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology CARET Criteria E-F E. Data Collection Strategies and Instrumentation 1. Sources of evidence clearly described 2. Instruments directly related the objectives and evaluation questions 3. Instrumentation has been validated and tested for reliability 4. Self-assessments completed by participants were used in this study 5. Participant observations guided by a set of criteria were incorporated 6. Interviews of participants were a part of the study 7. Existing participant data and information was used in the study 8. Criterion referenced assessments based on established standards were used 9. State and/or school district performance assessments were used 10. Standardized achievement tests were used 11. Quantitative records of participant behavior were included in the study F. Findings Adaptable to Multiple Contexts 1. The study documents similar findings in other contexts under specified conditions 2. The study documentation justifies dissemination and replication in other settings 3. The study suggests strategies for its replication by others 4. The cost of the program or intervention is justified by the probable outcomes 5. The study has a strong potential for replication in another setting

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology CARET Criteria G G. Reporting of Findings and Analysis 1. Recommendations are based on documented findings from the study 2. Contextual factors were considered in the analysis of findings 3. The extent of implementation of the intervention of project studied was considered 4. The study identifies research and evaluation questions for future studies 5. The study relates the specific interventions to the documented outcomes 6. The reporting suggests ways to replicate the interventions and findings 7. Reports on this study have previously been disseminated and/or published

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology CARET Process Identify critical questions and survey the literature. Apply the CARET criteria for identifying the most credible and relevant studies. Prepare answers to the questions based on the studies, emphasizing relevance to practicing educators.

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology

CARET Process Identify critical questions and survey the literature. Apply the CARET criteria for identifying the most credible and relevant studies. Prepare answers to the questions based on the studies, emphasizing relevance to practicing educators. Prepare formal reviews of key studies.

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology

Types of Studies in CARET 1.General Articles and Expert Positions (White papers, literature summaries) 2.Descriptive and Survey Studies (Surveys, case studies, meta-analyses) 3.Formal Evaluation Studies (Context-specific systematic program assessment) 4.Formal Research Studies (Experimental/quasi-experimental research)

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology

Other CARET Features Search site by keyword (e.g., author name) or reviews by characteristics Reading List of approx. 700 articles and books. Resources, including links to other sites, online journals, and CARET Frequently Asked Questions.

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology

A CARET Decision Map

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology Live link to CARET

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology For more information... Dr. Don Knezek Talbot Bielefeldt International Society for Technology in Education ( Center for Applied Research in Educational Technology ( National Educational Technology Standards (