1. Monitoring and Evaluation of ICT Use in Rural Schools Ok-choon Park Global Symposium on ICT in Education – Measuring Impact: Monitoring and Evaluation of ICT in Education Initiatives November 9 – 11, 2009 Seoul, Korea 1

2. Should monitoring and evaluation of ICT use in rural areas be different from those in other areas? Probably little for logistics Important difference should be the purpose of ICT use. Why ? 2 But not for basic procedures and methods Rural schools have different contextual characteristics and needs.

3. Geographically isolated. Smaller student populations, smaller class sizes, and more combined classes Fewer well qualified teachers Lower SES of parents Fewer opportunities for supplemental education services Fewer resources and supports inside and outside schools What are important characteristics of rural schools? Fewer Educational Opportunities 3 Lower Educational Quality

4. Fewer Educational Opportunities Fewer advanced courses; fewer enrichment classes; fewer remedial classes; fewer elective courses Fewer options for extra curriculum activities Fewer internship opportunities Fewer field trip options Fewer mentoring opportunities Fewer options for supplemental education Others 4

5. Lower Educational Quality Fewer opportunities Lower quality Less experienced and less qualified teachers Fewer resources Less stimulating interactions with peers Less parental assistance Others 5

6. ICT for Expanding Educational Opportunities Distance learning, virtual classrooms, and virtual schools Access to web-based resources On-line library On-line mentoring Virtual field trips Cooperative learning with students in other schools Others 6

7. On-line courses and supplemental instruction materials ICT-based learning tools and resources Virtual laboratory experiments, field trips, and others On-line teaching aids for teachers Data management tools for teachers Data-based teaching Administrative and management tools for teachers More teaching time On-line cooperative learning environments On-line interactions between teachers, students and parents Others Enhancing Educational Quality 7

8. Technology is a double-edged sword! Technology can expand educational opportunities and enhance educational quality for all children. But, technology creates digital gaps between rural and other poor schools and wealthy schools. Digital gaps widen: Closing the gaps is a challenging responsibility for policy makers! 8 Educational opportunity and quality gaps Learning achievement gaps Social and economic gaps

9. Initiative Examples of the US Government for Narrowing the Gaps Early initiatives for educational technology - PLATO system – hardware and software development - Computer Curriculum Corporation – Courseware development Star Schools – Distance learning E-Rate – ICT infrastructure building 21 st Community Learning Centers – After-school learning Many other grants for educational technology These and other government initiatives stimulated private investments in educational technology and its use at schools and homes! 9

10. Current Access to ICT in the USA Schools Near universal access to the Internet for all schools and classrooms One computer for every 3 or fewer students But, digital gaps and educational opportunity and quality Gaps are still there! Limited access to high-speed broadband internet Fewer computers for students and teachers Less technical support More students without computers or Internet connections at home Fewer resources Less parental support, and others 10

11. Laptop loan programs – Pennsylvania example - A laptop for every high school student - On-site technical support (1/2 time technician) - Teacher technical training – 30 hours Recent Trends for ICT Use in the USA 11 Virtual schools and classrooms - Increasing number of state-wide and independent virtual schools and classrooms - Widely different between states, districts, and schools: Course offerings and requirements Digital textbook programs

12. Needs of Systematic Monitoring and Evaluation for ICT Use How much do we know about current ICT access and use in schools and homes? Do we know what impacts ICT use has had for schools, teachers, and children? How do we know what supports schools, teachers and students need for effective use of ICT? How can we provide systematic guidance for teachers, parents and students to choose and use ICT programs and features? Do we have clear ideas how to improve ICT uses in schools and homes? How do we know which programs are good or bad? 12

13. Two Types of Evaluation for ICT Use Descriptive evaluation of ICT Use (e.g., NCES survey) - Information gathering for ICT access and use - Surveys, observations, interviews, etc. 13 Impact Evaluation of ITC Use (e.g., IES RCT studies) - Investigation of impacts of specific ICT features and programs on student learning - Experiments, quasi-experiments, regression discontinuity analysis, etc.

14. Access at schools, classrooms and homes for teachers and students - Internet access - Computer availability - Technical support Use of ICT features and programs - Instructional and instructional support use - Administrative and management use - Other uses – communication Extent of ICT use – frequencies and lengths Barriers for ICT use Descriptive Evaluation for ICT Access and Use 14

15. Longitudinal Database of Descriptive Evaluation Data Trend analysis of ICT access and use Correlational studies between ICT use and other variables - Teacher instructional methods - Student study patterns - Interaction patterns among teachers, students and parents - Student learning achievements - Student attitudes - Others 15

16. Need of scientific evidence-based information for the users and policy makers Limited value of past evaluation study results - Laboratory-based experiments - Meta-analysis results - Quasi-experiments and descriptive studies - Correlational, descriptive, and case studies Needs for Impact Evaluation of ITC Use 16

17. Selection of specific ICT use or programs for evaluation: Justifications - Theory-based - Demonstrated efficacy - Practical justification – wide use - Availability of implementation requirements and guidance Methodological Considerations for Impact Evaluation of ICT Use 17 Selection of evaluator - Objective third party: Firewall between developer/implementer and evaluator - Experience and technical competence

18. Selection of evaluation method - Randomized control study - Quasi-experiment - Regression discontinuity analysis 18 Selection of samples - Students - Teachers/classrooms - Schools

19. Determination of generalization level of the results - Fixed effect - Random effect - Determination of sample size: Power analysis 19 Selection of outcome measures - Terminal outcome measures – student achievements Tailored measures Standard assessments - Intermediate outcome measures Teacher knowledge Instructional methods Student attitudes Others

20. Selection of study scope - Data collection points - Number of cohorts - Selection of implementation feasibility data: surveys, observations, interviews, etc. 20 Development of analysis plan - Analysis models - Exploratory data analysis; Sensitivity analysis; subgroup analysis - Missing data management - Multiple comparison

21. Recruitment of samples Assignment of samples to treatment and control conditions 21 Implementation: as required by developer - Teacher training - Classroom implementation - Technical supports Data analysis and report generation - Confirmatory analysis – Summative report - Exploratory analysis – Formative report - Report with or without the results and interpretation of exploratory data