1. Can an Online Professional Development Project Impact Teacher Knowledge & Student Achievement? Vicky Zygouris-Coe, Ph.D., Associate Professor & Principal Investigator, FOR-PD vzygouri@mail.ucf.edu Bonnie Swan, Ph.D., Program Evaluator bswan@mail.ucf.edu Catherine Glass, Director, FOR-PD cc@orion.itrc.ucf.edu Nancy Lewis, Ph.D. nlewis@mail.ucf.edu The 12 th Annual Sloan-C International Conference on Asynchronous Learning Networks Orlando, FL November 9, 2006

2. Teacher Quality Matters information environment  While teachers may come to the classroom fully competent to teach, ongoing changes in this new information environment require ongoing, effective professional development—a powerful cornerstone to reading achievement.  This need is even more pronounced in secondary schools.

3. Documenting Change Research Questions:  What impact can online professional development have on teachers’ knowledge of scientifically based reading research; and  What (if any) changes result in their teaching practice?

4. Background

5. MISSION  FOR-PD is a 14-week online course, that was designed to improve Florida K-12 students’ reading achievement.  It functions as a primary delivery mechanism to translate, empower, and support Florida teachers in using scientifically-based reading research and implementing innovative, creative, and effective strategies; and  Provides teachers with ongoing access to abundant, rich, and relevant reading resources.

6. Florida’s Plan for Developing & Supporting Teacher Expertise in Reading Florida Add-on Reading Endorsement Competency 1: Foundations in Language and Cognition Competency 2: Foundations of Research-Based Practices Competency 3: Foundations of Assessment Competency 4: Foundations of Differentiation Competency 5: Application of Differentiated Instruction Competency 6: Demonstration of Accomplishment (Practicum)

7. Background, cont’d University of Central Floridaand is  The project, which is housed at the University of Central Florida, was launched in January, 2003; and is Just Read, Florida! Just Read, Florida! Initiative.  Funded by the Florida DOE Just Read, Florida! Initiative.Just Read, Florida!  FOR-PD was developed collaboratively with –literacy and technology experts, –school districts, –professional organizations, and –teacher educators across the state of Florida.  The free online course is facilitated by reading specialists and other well-qualified educators.

8. U.S.Department of Education Secretary’s No Child Left Behind Leadership Summit  In a report presented at the U.S.Department of Education Secretary’s No Child Left Behind Leadership Summit FOR-PD was named as an “innovative” e-Learning program for teacher training. Kleiman, 2004, p. 6 Kleiman, G. L. (2004, July). Meeting the need for high quality teachers: E-Learning solutions. White paper written for the U.S. Dept. of Education Secretary’s No Child Left Behind Leadership Summit: Increasing Options through E Learning, Orlando, FL. Retrieved March 6, 2006 from http://www.ed.gov/about/offices/list/os/technology/plan/2004/site/documents/Kleiman-MeetingtheNeed.pdf Background, cont’d

9.  The project is a highly-collaborative endeavor. Currently FOR-PD works with sixty-seven Florida districts, seven universities, and recently the project has been extended through five Community College Educator Preparation Institutes (EPIs).Educator Preparation Institutes Background, cont’d

10. To date (4 yrs) FOR-PD has served almost 25,000 participants statewide. Large Scale

11. FOR-PD Course Content  Content is based on current scientifically-based research and is updated frequently  14 Lessons  Online Text  Online Support Materials & Resources  Lessons include online discussions in an interactive environment  Assessment tools include online quizzes and assignments including a Literacy Log which can document evidence of best practices  Face-to-face components are sometimes added on an as-needed basis

12.  How do we assess what teachers learn and do in a large- scale online course of professional development?

13. Evaluation Tools

14. FOR-PD Evaluation Tools TREKA  Pre- and post-test of reading knowledge (TREKA)  Discussion Boards –Facilitator –Participant  School-level student achievement  Analyze participant work samples  Data-base analysis  Quality Assurance Monitoring

15. Evaluation Tools, cont’d Focus Group and Telephone Interviews  Facilitator and participant focus groups and large group discussions conducted at annual meetings  District contact interviews  Content-area teacher interviews  Facilitator interviews

16.  Facilitator End-of-course survey Training course survey  Participant Non-completer survey Implementation survey End-of-course survey  Help desk  QAC Facilitator Survey Evaluation Tools, cont’d Web-based Surveys

17. Evaluation Tools, cont’d Work Sample Example: Literacy Log

18. Results of Analysis

19. What FOR-PD Facilitators Are Saying …  I fully believe FOR-PD has made me a more effective facilitator and my students [participants] more successful learners.

20. Facilitator Responses to Rating an Impact of FOR-PD on K-12 Student Reading Skills Note. Data were obtained from Phase IV Facilitator End-of-Course Survey where 75% of spring 2006 facilitators responded (n = 93). FOR-PD Facilitators Results from End-of-course Surveys

21. Results of Analysis Literacy Logs  Analysis of FOR-PD literacy logs have shown: –Teachers are provided with effective avenues toward becoming more reflective in their teaching practice; and –Providing graphic organizers enables teachers to organize their thoughts on a particular topic or lesson and generate ideas for applying their knowledge and strategies at the classroom and school levels.

22. Participant Rate the Impact of FOR-PD on Classroom Instruction Note. Data were obtained from Phase IV Classroom Implementation Survey where 31% of fall 2005 participants responded. Ninety-nine percent of respondents answered this item (n = 427). FOR-PD Teachers Results from Classroom Implementation Survey

23. Participant Ability to Use Reading Strategies Taught in FOR- PD for Classroom Instruction in School Note. Data were obtained from Phase IV Classroom Implementation Survey where 31% of fall 2005 participants 99% of respondents answered this item (n = 426). FOR-PD Teachers Results from Classroom Implementation Survey

24. Frequency of Participant Use of Reading Strategies Taught in FOR-PD for Classroom Instruction in School Note. Data were obtained from Phase IV Classroom Implementation Survey where 31% of fall 2005 participants responded. (n = 426). FOR-PD Teachers Results from Classroom Implementation Survey

25. FOR-PD Participants Results from End-of-course Surveys

26. Reading Knowledge Assessment: Results of TREKA Pre to Post, Spring 2006 Participants’ Pre and Posttest Mean Scores  Pretest means were similar from one group to another.  Substantial gains on reading knowledge after completing the FOR-PD course: (t(1491) = 43.36, p <.0005). –A large effect size: the eta squared value =.558. –The true mean difference (95% C.I.): 9.4 to 10.3.  Posttest means were similar from one group to another.

27. FOR-PD Teachers–Reading Strategies Results from Classroom Implementation Surveys

29.  Almost all participants (94%) who responded indicated that FOR-PD positively impacted their classroom instruction.  All but 2% of respondents indicated they regularly use FOR-PD strategies with K-12 students. Most (65%) use the strategies they learned 3-or-more times a week.  Most respondents (83%) indicate that they now implement instructional strategies they learned from FORPD before, during, and after reading related instruction on a regular basis.  Differentiated instruction techniques learned in FOR-PD are regularly used by 74% of respondents.  Many (77%) indicated their classroom environment was print-rich, and that they regularly made use a classroom library and word walls.  Sixty-nine percent now use screening diagnosis and progress monitoring to better meet the needs of their students. FOR-PD Teachers–Reading Strategies Results from Classroom Implementation Surveys

30. FOR-PD Teachers Results from End of Course Surveys

31. Focus Groups Participants  “I know that everything is data driven. They’re [my students] not taking the FCAT, but their reading level, and I’ve got all the data, has gone up on average of my eleven kids, two grade levels. It is working for my kids.”

32. Focus Groups Facilitators  “Even though I am not a reading teacher, I needed to have a base understanding of reading so that I could make sure I support the classroom teacher and get the appropriate accommodations for the students in my class. And it helped me do that – get a stronger base of my own knowledge.”

33. FOR-PD Teachers Results from Classroom Implementation Surveys Participants Report Improvement in Students’ Reading Performance *See next slide for explanation *

34. FOR-PD Teachers Results from Classroom Implementation Surveys Why Some Participants Do Not Believe FOR-PD has a Positive Impact on FCAT Scores FactorPercent They didn’t have students’ FCAT scores yet or it was too soon to tell37% Their students don't take the FCAT (other)24% That they were not a classroom teacher6% Their students don't take the FCAT (special ed/ESOL)8% They already had knowledge taught by FOR-PD6% Curriculum was mandated by school/district4% Students were too low at start or other factors were too strong8% Note. Data were obtained from Phase IV Classroom Implementation Survey (n = 430).

35.  Analysis of data from ongoing, rigorous, and well embedded evaluation show that almost all participants use scientifically-based reading strategies for reading instruction taught in FOR-PD; and believed the course helped them to help their students learn. Summary of Results

36. Research has shown that professional development that supports ongoing, collaborative learning is vital for improving student outcomes.

37. QUESTIONS

38. Thank you! Please visit the FOR-PD website: http://www.itrc.ucf.edu/forpd

39. References References  Kleiman, G. L. (2004, July). Meeting the need for high quality teachers: E-Learning solutions. White paper written for the U.S. Dept. of Education Secretary’s No Child Left Behind Leadership Summit: Increasing Options through E-Learning, Orlando, FL. Retrieved March 6, 2006 from http://www.ed.gov/about/offices/list/os/technology/plan/2004 /site/documents/Kleiman-MeetingtheNeed.pdf http://www.ed.gov/about/offices/list/os/technology/plan/2004 /site/documents/Kleiman-MeetingtheNeed.pdf  Swan, B., Huh, J., & Ramos, R. (August, 2006). Florida Online Reading Professional Development (FOR-PD) Phase IV Evaluation Results of Surveys. Orlando FL: University of Central Florida