1. Online Professional Development for Transition: Evaluating the Effectiveness Mary E. Morningstar, PhD – mmorningstar@ku.edu Dana L. Lattin, MSEd – dlattin@ku.edu Amy Gaumer Erickson, PhD – aerickson@ku.edu Ryan Kellems, MEd – rkellems@ku.edu October 18, 2007 DCDT International Conference – Orlando, FL

2. Late 1960’s Department of Special Education commitment to secondary special education and career development starts in late 1960s First, a little background…. Late 1960’s 1980’s 2000-2004 1970’s 2004-?1990’s 1970’s  Federal grant for secondary teacher education personnel preparation model, 1972-76  Federal RETOOL project for higher education faculty on career education for students with disabilities, 1975-79 1980’s  National study of high school programs, 1985-88  1985 Department of Defense Dependents’ Schools Summer Institute on career/ vocational  Summer transition workshops late 1980s  Secondary Special Education/ Transition personnel preparation grant, 1989-91 1990’s  Kansas Systems Change Project - Statewide Transition Training 1993-1998  Kansas Transition Network – contract with KSDE 1998-2000  KU Secondary/Transition Personnel Preparation Program 1994-1999  KU Secondary/Transition Outreach project 1997-2000. 2000-2004 Interprofessional/Transition Personnel Preparation, 2000-03 National Transition Outreach Project (National Transition Coalition), 2001-2004 National Online Specialization in Transition Project, 2001-2004 2004 - Present KU TransCert: Online Transition Graduate Certificate Program 2004-2008 Evaluating the Effectiveness of Online Professional Development for Transition 2004-2007 KU Secondary Transition Leadership Program 2005-2009

3. Continuing Ed. Online Classes Overview of Transition Transition Assessment Interagency Collaboration Employment Student & Family Involvement SEAs Online training Face to face State Needs Assess COPs

5. Online Resources Searchable Databases Models of Success Online Training User friendly and interactive information Making sense of complex information Applying information Connect & network Non-Instructor Led Online Professional Development

6. Online Training Modules Currently Available Working with Families Transition Assessment: The Big Picture Best Practices in Transition and IDEA 2004 Cultural Diversity and Transition Students with ED/BD and Employment In Design Phase Aligning Transition IEPs with Standards Self-Determination SBR & Transition Students with Significant Disabilities & PBS

7. Online Module Framework oLearning Objectives oCase-Based Examples & Activities oConceptual Models/Unifying Frameworks oImportant Point oNow It’s Your Turn oMore Information & My Library oInteractivity: Click & Compare; True/False; Games; Audio/video oSession Summary oPre/Post Test oLearning activities oReflective questions Impact on teaching Barriers to implementation Reflection of general learning content GeneralContent UserPortfolio

8. Learning Objectives

9. Case-based Examples & Activities

10. Conceptual Models & Unifying Frameworks

12. Important Point

13. Now It’s Your Turn

14. More Information & My Library

16. True/False or Myth/Reality

18. Click & Compare

20. Games

23. Audio/Video

24. Session Summary

26. Demographic Data of All Users 2405 practitioners –68% educators (sped, gen ed, transition, related services, administrators) ~ 50% serving secondary-aged students –18% College/university students Types of disability groups served –30% LD; 18% multiple groups; 5% MR; 4% ED/BD; 2% autism; 5% SMD; 25% Programmatic –57% regular schools; 7% special schools; 6.5% community agencies Level of Education –44% Masters/specialist; 44% bachelors; 4% doctorates; 3% associates March, 2007

27. Educators Certification Status –43% fully certified for current teaching assignment; 3% emergency certif.; 7% certif. in field other than teaching; 8% provisional; 17% NA Years Teaching –36% 10+; 8% 7-9yrs.; 15% 4-6yrs; 20% 1-3yrs; 9% student teaching; 11% not teaching Transition Training & Information Sources –46% inservice training; 26% transition course; 6% 2+ courses; 9% infused content; 17% conferences; 17% newsletters/online; 43% on-the- job training March, 2007

28. Satisfaction with Training Modules Families Kept attention86% SA or A80%89% Helped dev. Understand of content 95% SA or A93%92% Contributed to Learning: - Case-based activities - Interactive examples - Important points - text and written information 93% SA or A 87% SA or A 95% SA or A 94% NA 91% NA 91% Training was of high quality81% SA or A91%89% Important resource for the future 92% SA or A78%87% Effective way to offer inservice training 85% SA or A84%91% Best Practices Assessment March, 2007

29. Qualitative Data Results: Online Training Modules PRO’s Ease of use –Graphics –Navigation –Flexibility –Portfolio Information was current, useful & to the point Examples and interactive case studies CON’s Pre/Post Test Limited information Portfolio Printer-friendly version More reflective practice and application Links not working

30. Evaluation of Online Modules: Knowledge Gain Repeated measures design - Paired Sample t-test N= 282; Post-test (M=11.51, SD 2.77) significantly greater than pre-test (M=13.90, SD 2.86); t(164) = - 10.85 p<.00, d=.84 Working with Families during Transition N= 282; Post-test (M=11.51, SD 2.77) significantly greater than pre- test (M=13.90, SD 2.86); t(164) = -10.85 p<.00, d=.54 Best Practices in Transition Planning N= 282; Post-test (M=11.51, SD 2.77) significantly greater than pre- test (M=13.90, SD 2.86); t(124) = -14.318 p<.00, d=1.28 Transition Assessment

31. Random Sample-Control Group Study: 2 Online Modules Kim (2006). Kim & Morningstar (2006). Working with Families & Working with CLD Families and Youth Modules Control group (43; no intervention) & Intervention group (43; A-B; B-A) Content Knowledge test (pre/post) Competencies (level of preparation) + Attitudes (level of importance) Survey (.79-.94 reliability)

32. Results: Knowledge & Competency Gains 28/42 of control group completed pre & post tests 25/42 of intervention group completed study Comparison of groups –Chi Square tests = no significant difference among 2 groups for years teaching, types of students, training, certification, sources of training in transition –Content Knowledge: Experimental group showed significant changes in knowledge & control group showed no change –Perceived competencies: WWF: significant difference; WCLD: control & experimental groups showed no change –Perceived importance (Attitudes): no change pre/post both; WWF’s between groups ANCOVA on posttest was significant for control

33. Online Professional Development Evaluation Control Group 15-20 practitioners No training No online CoP Training Only 15-20 practitioners 2 training modules Training + CoP 15-20 practitioners 2 training modules Online CoP Comparisons Knowledge  Competencies & skills  Frequency of implementation Mixed Measures Pre-post test  Competencies surveys  Performance-based content analysis  Qualitative data Access to ALL online training upon completion of the study Recruitment & Sampling Access to ALL online training modules upon completion of the study

34. Recruitment of Participants Initial recruitment in 3 states (CO, ID, AZ) Contacted SDE staff at other states to find other participants NEW box on TC website – available to anyone Online agreement to participate Blocked from all training modules until research started Needed 150 participants with min. 90 completing research – recruited 156

35. Procedures Communication with participants was through email Participants in all groups (C, T, TCoP) were asked to begin research at the same time. Participants received email reminders and prompts to complete the research and were given approximate times for how long it would take to complete each step. All participants received incentives for completion of research (honorarium or graduate credit) and for completing various steps of the research within specific timelines (jump drives, bonuses, transition books) Phone support was provided to those participants needing intensive technical support using the website.

36. Participant Demographics 81 (86%) Educators 8 (9%) School Related Services Providers 5 (5%) Other –Community Service providers, Family members, Administrators 88 (94%) Serve secondary aged students Types of disability groups served –Across Multiple Disabilities (55%); High Incidence (33%) Low Incidence (7%); None (4%) Level of Education –64% Masters/specialist; 30% bachelors; 4% doctorate; 1% associates; 1% HS Diploma 94 Completed Research ( C=35; T=30; TCoP=29 )

37. Educators Educators Only - Certification Status –75% fully certified for current teaching assignment; 6% provisional/emergency certif.; 3% certif. in field other than teaching; 16% NA Educators Only - Years Teaching –64% 10+; 6% 7-9yrs.; 16% 4-6yrs; 13% 1-3yrs; 1% not teaching Educators Only - Transition Training & Information Sources –75% inservice training; 72% conferences; 64% newsletters/articles/books; 62% on-the-job training; 48% online; 20% professional associations; 9% 1+ transition course; 13% infused content

38. Measures STTS (pre/post) Best Practices pre/post tests Transition Assessment pre/post tests Satisfaction Surveys for each module Performance-based Assessment of 3 specific activities within each module (total of 6 activities across both modules)

39. DOMAINS Instructional Planning Curriculum and Instruction Transition Planning Assessment Collaboration Additional Competencies RELIABILITY TESTS: 557 teachers nationally Alpha scores:.96 Preparation subscale.94 Frequency subscale

40. Control Group Pre STTS Post STTS Best Practices Pre test Transition Assessment Pre test 1 Best Practices Post test 2 Transition Assessment Post test

41. Training Group Post STTS Pre STTS Best Practices 1. Pre test 2. Complete Module and all Activities 3. Post test 4. Satisfaction Survey Transition Assessment 1. Pre test 2. Complete Module and all Activities 3. Post test 4. Satisfaction Survey

42. Training + CoP Group Pre STTS Post STTS Participate in Community of Practice Best Practices 1. Pre test 2. Complete Module and all Activities 3. Post test 4. Satisfaction Survey Transition Assessment 1. Pre test 2. Complete Module and all Activities 3. Post test 4. Satisfaction Survey

47. Performance-based Assessment Developed, validated, user-tested, modified rubrics for 6 activities 3 reviewers met to reach consensus on scoring methods Compared scores on sample Scored all to determine Inter-rater Reliability –Range 75%-90% –Average 86% Mean Scores Comparisons between T & T+CoP groups

48. Results: Knowledge Gained Measure: pre/post tests for modules –Best Practice Module Statistically significant gain in knowledge for both training groups; high effect size Statistically significant difference in knowledge on post-tests between control and training groups, accounting for 35% of the variance in scores –Assessment Module Statistically significant gain in knowledge for both training groups; high effect size Statistically significant difference in knowledge on post-tests between control and training groups, accounting for 8% of the variance in scores

49. Results: Satisfaction with Training Kept attention91.2% SA or A94.8% Helped dev. Understand of content 96.4% SA or A94.8% Contributed to Learning: - Case-based activities - Interactive examples - Practice activities - text and written information 93.1% SA or A 86.2% SA or A 93.1% SA or A 91.3% SA or A 100% 91.4% 94.8% 96.6% Training was of high quality94.7% SA or A96.5% Relevant to my job98.2% SA or A98.3% Important resource for the future 96.5% SA or A96.6% Effective way to offer inservice training 94.8% SA or A96.5%

50. Results: Perceived Skill & Frequency Measure: STTS pre/post tests –Statistically significant increase in perceived skills for training groups –CoP group showed more realistic perceptions of their skills and frequency on the post-test

51. Current & Future Evaluation Efforts: Personnel Preparation Increased Transition Knowledge Participant satisfaction Improved Transition Planning Attention to teaching transition content Increased communication & linkages Competency Surveys PBA QI Program Evaluation Goal Attainment Scaling Focus groups Content Analysis of Discussions