1. Evaluating Technologys Impact on Teaching and Learning Educational Technology Integration

2. Contact Information zjsun@sun-associates.com y978-453-3070 zwww.sun-associates.com/eval/samplewww.sun-associates.com/eval/sample yThis presentation is linked to that page

3. Where Do We Stand? zHot topics yYour District yState yNational zYour expectations for today

4. Workshop Goals zTo overview a scheme for creating a formative evaluation of technologys impact zTo review related issues and developments visa vis our experience in a particular school district (Fayette County, KY) zTo create an action plan for conducting a similar assessment in your own district

5. A Framework for Review

6. Why Evaluate? zTo realize your investment in technology yWhat sort of difference has all of this technology made? zTo continue your school or district technology planning effort yEvaluation is a key component of ongoing technology planning!

7. Evaluation Starts with Technology Goals zEvaluation should be rooted in a strong educational technology plan which... yIs more than an infrastructure plan yFocuses on technologys impact on teachers and students yHas clear goals and objectives for what you want to see happen

8. Skeletal Plan Format

9. Your Goals? zUsing the Plan Mapping Worksheet, map your plan in terms of vision, goals, and actions.

10. Goals Lead to Questions zWhat do you want to see happen? yThese are your goals zAchieving these goals requires a process yAre you performing the process steps? zAll of this can be measured through a formative evaluation

11. The Evaluation Process zCreate meaningful evaluation questions zDesign relevant indicators zOrganize those indicators in a performance rubric which can be used to measure progress and achievement zCollect data to score the rubrics zCreate a report which applies the evaluation in a formative manner

12. Take a break...

13. FCPS Example Questions zHow have students been impacted by technology integration? zAre teachers using technology in ways that match district goals for technology and use the potential for instructional technology? zHave we adequately allocated district technology resources?

14. Processes of Technology Evaluation zWhat are you looking for? yWays to make the implementation of technology better yTeaching and learning impacts this is not about counting machines!

15. Developing Indicators zWhat is it that you want to measure? yImpact on teaching and learning yChange yProgress towards desired outcomes zYou need indicators for progress to be measured

16. zIndicators should reflect your schools unique goals and aspirations yArise from your plan yRooted in your vision and goals yIndicators must be indicative of your unique environment...what constitutes success for you might not for someone else.

17. Indicators of Engaged Learning zEngaged learners are yResponsible for their own learning yStrategic problem-solvers yEnergized by learning yCollaborative learners zThese are all observable characteristics zThese indicators can help inform your student impact rubric

18. Levels of Teacher Appropriation

19. zPoint is... There are things we can look for in terms of student and teacher behavior. zThe goal in creating indicators is to link aspects of behavior to levels of achievement

20. Whats Realistic? zWhen developing questions and indicators, what makes sense visa vis the research on technologys potential impacts? zwww.sun-associates.com/eval/resources.htmlwww.sun-associates.com/eval/resources.html

21. What the Research Says zCBI generates measurable increases in mastery of core content yKuliks meta-analysis yFletchers evaluation of military training zPace, repetition, and standardization promote memorization of facts zKuliks work shows that CBI did virtually nothing in terms of changing student attitude toward subject matter

22. zProductivity tools -- particularly tools for writing -- encourage students to produce more and pay more attention to the quality of their work. yThis more free-form use of technology cuts directly to the heart of the connection between attitude toward learning and student achievement yStudents will do more of what they like to do

23. zTechnology improves the learning environment yMotivates students yMakes learning more exciting yMakes learning more relevant to real-world tasks zNone of these benefits come solely through technology!

24. zIt gets more gray when considering cognitive uses yCooperative and collaborative tasks yCommunication yProblem solving yUsing technology as a key element of a rich, supportive, educational environment

25. Student Performance in Rich Educational Environments zHigher rates of attendance and graduation (ETS report) zHigher qualification rates for honors programs zGreater development of critical thinking and problem-solving skills zGenerally higher scores on traditional assessments zLower numbers of disciplinary incidents

26. zThese benefits have been observed/documented specifically in those classrooms where technology has been a key intervention...but never the only intervention!

27. What Other Interventions? zMore teacher training yUsually in relation to using the technology tools as part of an overall reformed instructional environment zMore individualized, student-centered, learning yUsually as a result of problem-based, cooperative learning environments where technology is the catalyst for change

28. So, What Changes? zThe student outcomes from participation in a technology-enriched environment are indeed positive zMeasures are usually in terms of new things that can be done versus increases on traditional assessments zNevertheless, traditional assessments do show improvement...it just occurs over time and is part of a broader, system-wide, picture of change.

29. Try a Sample Indicator zUsing the Developing Indicators worksheet, lets develop a few sample indicators yWork in several groups ySelect someone to report out yTry to develop some consensus on indicators which would work across districts

30. FCPSs Indicators zUse of technology positively impacts and fosters the students motivation to engage in learning practices that lead to new ways of thinking, understanding, constructing knowledge, communicating results, and acquiring basic skills

31. zThe faculty and staff are proficient, knowledgeable, and current with available technology and translate that knowledge into relevant learning opportunities for students zTeachers create learning opportunities and physical environments that allow students to assume more independent roles in their own learning through their use of technology.

32. All teachers and learners throughout the district have sufficient access to technology-based productivity tools, on- line services, media-based instructional materials, primary sources of data, and adequate support for using these resources so as to enrich and extend their learning goals.

33. Rubrics www.fayette.k12.ky.us/central/edtech/suneval.asp

37. Rubric Tips zEven number of levels zCreate from ideal to lesser zYes, Yes But, No But, No zBe as descriptive as possible zDo not get excessively quantitative

38. Take a break...

39. FCPS Rubric Scores

40. Student Impact, 2 zTechnology is being used daily as a basic tool within a range of traditional learning activities zSome evidence that technology is supporting new ways of thinking and learning zMost student technology use is very teacher-directed and occurs in settings outside of the classroom

41. Teacher Fluency zTeacher Fluency = 2.5 yTeachers are reasonably fluent, but need to do more to connect technology to the core curriculum zUse of Technology with Students = 2.25 yTeachers direct students to use technology, but need to use technology to empower students as independent, engaged, learners

42. zTeacher Productivity = 4 yTeachers are very proficient in using technology as a tool for personal and professional productivity zAdministrator Support = 2 yAdministrators use technology, but need to advocate and support more integrated uses among their teaching staff

43. Infrastructure Allocation, 2.75 zThe basic infrastructure -- hardware, network, software -- is complete and adequate zIndividuals in Technology Support functions (schools and district) are doing an exemplary job and should receive much of the credit for the advances we have seen thus far zSchools need even more technology resource teacher support zEfforts must continue to map technology on to the core curriculum

44. Sum It All Up? zFayette County is doing very well! yBased on our research nationwide, FCPS is clearly above average in most areas of its technology implementation zTeachers are using technology in ways that are consistent with their overall instructional approach yDirected instruction yTechnology as a skill versus an integrated element of the learning process

45. zTechnology can do more, and teachers can do more yProfessional development and increased levels of resource teacher support will make this happen yThe full report contains a number of recommendations for professional development and policy designed to positively impact performance on the indicators

46. To Summarize... zStart with your plan zFrom your goals, develop indicators and a performance rubric zAccumulate data and/or develop your data collection tools yHow will you collect the data necessary to document your stage of fulfillment?

47. Data Collection zReview Existing Data yCurrent technology plan yCurriculum yDistrict/school improvement plans zwww.sun-associates.com/eval/samplewww.sun-associates.com/eval/sample zCreate a checklist for data collection

48. zCollect New Data yFocus Groups/Interviews xTeachers xParents xStudents xAdministrators xOther stakeholders yClassroom Observations

49. ySurveys xCreating good surveys what do you want to know length differentiation (teachers, staff, parents, community, etc..) quantitative data attitudinal data timing/response rates (getting returns!) xwww.sun-associates.com/eval/samples/samplesurv.htmlwww.sun-associates.com/eval/samples/samplesurv.html

50. zSolicitation of teacher/parent/student stories yThis is a way to gather truly qualitative data yWhat does the community say about the use and impact of technology?

51. zCompile the report zDetermine how to share the report ySchool committee presentation yPress releases yCommunity meetings

52. Conclusion zBuild evaluation into your technology planning effort zRemember, not all evaluation is quantitative zYou cannot evaluate what you are not looking for, so its important to zDevelop expectations of what constitutes good technology integration

53. Technology and Traditional Assessment Methods zDoes technology use result in higher test scores? zProbably, but there is not a direct link zTechnology alone is not helpful in improving student performance zA high degree of technology integration is more than likely to be an indicator itself of high performing schools and students

54. Is There a Connection? zHigh scores on technology integration rubrics generally equate to successful teaching and learning practices -- regardless of whether these practices have any specific relation to technology ye.g., Constructivism, Engaged Learning, and Student-Centered Classrooms

55. zTechnology supports these practices and these practices are shown to improve student learning zSchools where these practices are in place and technology is used appropriately are high performing schools zIts not the technology, its the teaching!

56. zWe measure how well the technology is used to support successful teaching practices. zOur formative evaluations highlight how to improve teaching, technology integration, and ultimately student success

57. Next Steps? zAwareness building yEstablish the need for evaluation yEstablish a process zDivision leadership yAs with technology implementation in general, this is very important yThe evaluation will only be as good as it is taken seriously

58. zEvaluation committee membership yStakeholders! zAre tech plans in shape? yIdeally, the evaluations will be based in sound visions and goals and will take into account existing data on completed actions yDo you divisions have this capacity?

59. zEvaluation work can be used to rejuvenate the local revision process, but this will take time and organization zTraining yIt takes about a day to train people in this approach/process yRegional workshops? yDistrict teams?

60. zStandardizing and cross-division review yPeople should share and review their questions, indicators, and rubrics before setting out to conduct an evaluation zThink about data collection yIntegrating a meaningful technology impact survey into existing survey efforts yIntegrating observation data into existing teacher/class observations

61. More Information zjsun@sun-associates.com y978-453-3070 zwww.sun-associates.com/eval/samplewww.sun-associates.com/eval/sample yThis presentation is linked to that page