1. Critical Issues in Evaluating Educational Technology Mark Hawkes Dakota State University Presentation at the American Evaluation Association Annual Conference, Nov. 2, 2000

2. School Expenditure on Technology 1997-98 1998-99 1999-2000 $5.2 Billion $6.0 Billion $7.2 Billion

3. The Secretarys Conference on Educational Technology Evaluating the Effectiveness of Technology The effectiveness of technology is embedded in the effectiveness of other school improvement efforts. Standardized test scores offer limited formative information with which to drive the development of a schools technology program.

4. Critical Issues… Current evaluation practices need broadening. Schools must document and report their evaluation findings in ways that satisfy diverse stakeholders need to know. Everyone must agree on a common language and standards of practice for measuring how schools achieve that end.

5. Critical Issues... The role of teachers is crucial in evaluating the effectiveness of technology in schools. Some existing policies need to be transformed to match the new needs of schools using technology.

6. Scope of Evaluation: status then Technology focus: Drill-and-practice software, integrated learning systems, videotaped lessons, computer-animated picture books, low-level problem-solving and simulation computer software Professional Development: Participation, topics, accessibility, quality Hardware capability: Processing speed, memory, application functionality Student products Rate of implementation

7. Scope of Evaluation: status now Technology: information-processing and productivity tool Professional Development: relationship to student performance with technology Breadth of technology vision Level and breadth of ownership Community involvement Propensity for new practice

8. Metrics: status then Standardized measures (IGAP, ITBS, CTBS) Student-to-computer ratio Survey: Self-report (competency, nature of use) Affect oriented Behavioral indicators Attendance/Graduation rates Principal referrals Post-secondary options Teacher discourse: Technology-centered

9. Metrics: status now Critical incident analysis Incidents of learning activities that reflect job- related skills and performances Token Economies Media Production Comparative science Nature of discourse, signs of: Teacher-centerednature of pedagogy Student centeredconstructivist pedagogy Classroom observation of innovative uses

10. Role of Teacher: status then Hands on time with the technology Trips to the lab Types of activities Tools used Report affective measures Student attitudes toward technology Document categories of outcomes (motivational, behavioral)

11. Role of Teacher: status now Teacher activity logs Journaling on how teachers implemented technology facilitated instructional strategies with corresponding outcomes Content area professionals in examining how technology is linked to state learning standards Examining relationship between instructional practices and teaching and learning roles

12. Technology Policy: status then Lab topology; limited access detached form the learning experience Access to technology resources (equity) Focus with use policy and security Acceptable Internet use Email accounts

13. Technology Policy: status now Aligning school support systems with technology outcomes Libraries as information centers Continuous financial support to sustain system components Exploring technology use opportunities that extends learning beyond the school day Extended lab hours Lap top check-out

14. Inputs Outputs Infrastructure Student motivation Wireless features Accessibility Restructured leadership Improved PD New roles for Improved use of support services services Individual Staff Renewed teacher development plans skill building Competitive Improved efficiency collaboration grants in technology use

15. Over the two years... Continued reliance on standardized measures Policy at all levels remain prohibitive in many cases Improved support for technology purchases and professional development Limited examination of systemic indicators: - Vision of technology use - Support for technology implementation - Community roles and involvement - Administrative effectiveness

16. Over the two years... Parsing the effects of technology from the numerous school improvement ventures in the school proves virtually impossible Growth needed in collaborative gap analysis and deployment of technology to address those gaps Efforts to evaluate look quasi- experimental, have more attributes of research than evaluation

17. Slides available at: www.homepages.dsu.edu/hawkesm/ Presentations.htm Paper available at: www.ed.gov/Technology/TechConf/1999/conf sum.html