Online Professional Development for Transition: Evaluating the Effectiveness Mary E. Morningstar, PhD – Dana L. Lattin, MSEd – Amy Gaumer Erickson, PhD – Ryan Kellems, MEd – October 18, 2007 DCDT International Conference – Orlando, FL

Late 1960’s Department of Special Education commitment to secondary special education and career development starts in late 1960s First, a little background…. Late 1960’s 1980’s ’s 2004-?1990’s 1970’s  Federal grant for secondary teacher education personnel preparation model,  Federal RETOOL project for higher education faculty on career education for students with disabilities, ’s  National study of high school programs,  1985 Department of Defense Dependents’ Schools Summer Institute on career/ vocational  Summer transition workshops late 1980s  Secondary Special Education/ Transition personnel preparation grant, ’s  Kansas Systems Change Project - Statewide Transition Training  Kansas Transition Network – contract with KSDE  KU Secondary/Transition Personnel Preparation Program  KU Secondary/Transition Outreach project Interprofessional/Transition Personnel Preparation, National Transition Outreach Project (National Transition Coalition), National Online Specialization in Transition Project, Present KU TransCert: Online Transition Graduate Certificate Program Evaluating the Effectiveness of Online Professional Development for Transition KU Secondary Transition Leadership Program

Continuing Ed. Online Classes Overview of Transition Transition Assessment Interagency Collaboration Employment Student & Family Involvement SEAs Online training Face to face State Needs Assess COPs

Online Resources Searchable Databases Models of Success Online Training User friendly and interactive information Making sense of complex information Applying information Connect & network Non-Instructor Led Online Professional Development

Online Training Modules Currently Available Working with Families Transition Assessment: The Big Picture Best Practices in Transition and IDEA 2004 Cultural Diversity and Transition Students with ED/BD and Employment In Design Phase Aligning Transition IEPs with Standards Self-Determination SBR & Transition Students with Significant Disabilities & PBS

Online Module Framework oLearning Objectives oCase-Based Examples & Activities oConceptual Models/Unifying Frameworks oImportant Point oNow It’s Your Turn oMore Information & My Library oInteractivity: Click & Compare; True/False; Games; Audio/video oSession Summary oPre/Post Test oLearning activities oReflective questions Impact on teaching Barriers to implementation Reflection of general learning content GeneralContent UserPortfolio

Learning Objectives

Case-based Examples & Activities

Conceptual Models & Unifying Frameworks

Important Point

Now It’s Your Turn

More Information & My Library

True/False or Myth/Reality

Click & Compare

Games

Audio/Video

Session Summary

Demographic Data of All Users 2405 practitioners –68% educators (sped, gen ed, transition, related services, administrators) ~ 50% serving secondary-aged students –18% College/university students Types of disability groups served –30% LD; 18% multiple groups; 5% MR; 4% ED/BD; 2% autism; 5% SMD; 25% Programmatic –57% regular schools; 7% special schools; 6.5% community agencies Level of Education –44% Masters/specialist; 44% bachelors; 4% doctorates; 3% associates March, 2007

Educators Certification Status –43% fully certified for current teaching assignment; 3% emergency certif.; 7% certif. in field other than teaching; 8% provisional; 17% NA Years Teaching –36% 10+; 8% 7-9yrs.; 15% 4-6yrs; 20% 1-3yrs; 9% student teaching; 11% not teaching Transition Training & Information Sources –46% inservice training; 26% transition course; 6% 2+ courses; 9% infused content; 17% conferences; 17% newsletters/online; 43% on-the- job training March, 2007

Satisfaction with Training Modules Families Kept attention86% SA or A80%89% Helped dev. Understand of content 95% SA or A93%92% Contributed to Learning: - Case-based activities - Interactive examples - Important points - text and written information 93% SA or A 87% SA or A 95% SA or A 94% NA 91% NA 91% Training was of high quality81% SA or A91%89% Important resource for the future 92% SA or A78%87% Effective way to offer inservice training 85% SA or A84%91% Best Practices Assessment March, 2007

Qualitative Data Results: Online Training Modules PRO’s Ease of use –Graphics –Navigation –Flexibility –Portfolio Information was current, useful & to the point Examples and interactive case studies CON’s Pre/Post Test Limited information Portfolio Printer-friendly version More reflective practice and application Links not working

Evaluation of Online Modules: Knowledge Gain Repeated measures design - Paired Sample t-test N= 282; Post-test (M=11.51, SD 2.77) significantly greater than pre-test (M=13.90, SD 2.86); t(164) = p<.00, d=.84 Working with Families during Transition N= 282; Post-test (M=11.51, SD 2.77) significantly greater than pre- test (M=13.90, SD 2.86); t(164) = p<.00, d=.54 Best Practices in Transition Planning N= 282; Post-test (M=11.51, SD 2.77) significantly greater than pre- test (M=13.90, SD 2.86); t(124) = p<.00, d=1.28 Transition Assessment

Random Sample-Control Group Study: 2 Online Modules Kim (2006). Kim & Morningstar (2006). Working with Families & Working with CLD Families and Youth Modules Control group (43; no intervention) & Intervention group (43; A-B; B-A) Content Knowledge test (pre/post) Competencies (level of preparation) + Attitudes (level of importance) Survey ( reliability)

Results: Knowledge & Competency Gains 28/42 of control group completed pre & post tests 25/42 of intervention group completed study Comparison of groups –Chi Square tests = no significant difference among 2 groups for years teaching, types of students, training, certification, sources of training in transition –Content Knowledge: Experimental group showed significant changes in knowledge & control group showed no change –Perceived competencies: WWF: significant difference; WCLD: control & experimental groups showed no change –Perceived importance (Attitudes): no change pre/post both; WWF’s between groups ANCOVA on posttest was significant for control

Online Professional Development Evaluation Control Group practitioners No training No online CoP Training Only practitioners 2 training modules Training + CoP practitioners 2 training modules Online CoP Comparisons Knowledge  Competencies & skills  Frequency of implementation Mixed Measures Pre-post test  Competencies surveys  Performance-based content analysis  Qualitative data Access to ALL online training upon completion of the study Recruitment & Sampling Access to ALL online training modules upon completion of the study

Recruitment of Participants Initial recruitment in 3 states (CO, ID, AZ) Contacted SDE staff at other states to find other participants NEW box on TC website – available to anyone Online agreement to participate Blocked from all training modules until research started Needed 150 participants with min. 90 completing research – recruited 156

Procedures Communication with participants was through Participants in all groups (C, T, TCoP) were asked to begin research at the same time. Participants received reminders and prompts to complete the research and were given approximate times for how long it would take to complete each step. All participants received incentives for completion of research (honorarium or graduate credit) and for completing various steps of the research within specific timelines (jump drives, bonuses, transition books) Phone support was provided to those participants needing intensive technical support using the website.

Participant Demographics 81 (86%) Educators 8 (9%) School Related Services Providers 5 (5%) Other –Community Service providers, Family members, Administrators 88 (94%) Serve secondary aged students Types of disability groups served –Across Multiple Disabilities (55%); High Incidence (33%) Low Incidence (7%); None (4%) Level of Education –64% Masters/specialist; 30% bachelors; 4% doctorate; 1% associates; 1% HS Diploma 94 Completed Research ( C=35; T=30; TCoP=29 )

Educators Educators Only - Certification Status –75% fully certified for current teaching assignment; 6% provisional/emergency certif.; 3% certif. in field other than teaching; 16% NA Educators Only - Years Teaching –64% 10+; 6% 7-9yrs.; 16% 4-6yrs; 13% 1-3yrs; 1% not teaching Educators Only - Transition Training & Information Sources –75% inservice training; 72% conferences; 64% newsletters/articles/books; 62% on-the-job training; 48% online; 20% professional associations; 9% 1+ transition course; 13% infused content

Measures STTS (pre/post) Best Practices pre/post tests Transition Assessment pre/post tests Satisfaction Surveys for each module Performance-based Assessment of 3 specific activities within each module (total of 6 activities across both modules)

DOMAINS Instructional Planning Curriculum and Instruction Transition Planning Assessment Collaboration Additional Competencies RELIABILITY TESTS: 557 teachers nationally Alpha scores:.96 Preparation subscale.94 Frequency subscale

Control Group Pre STTS Post STTS Best Practices Pre test Transition Assessment Pre test 1 Best Practices Post test 2 Transition Assessment Post test

Training Group Post STTS Pre STTS Best Practices 1. Pre test 2. Complete Module and all Activities 3. Post test 4. Satisfaction Survey Transition Assessment 1. Pre test 2. Complete Module and all Activities 3. Post test 4. Satisfaction Survey

Training + CoP Group Pre STTS Post STTS Participate in Community of Practice Best Practices 1. Pre test 2. Complete Module and all Activities 3. Post test 4. Satisfaction Survey Transition Assessment 1. Pre test 2. Complete Module and all Activities 3. Post test 4. Satisfaction Survey

Performance-based Assessment Developed, validated, user-tested, modified rubrics for 6 activities 3 reviewers met to reach consensus on scoring methods Compared scores on sample Scored all to determine Inter-rater Reliability –Range 75%-90% –Average 86% Mean Scores Comparisons between T & T+CoP groups

Results: Knowledge Gained Measure: pre/post tests for modules –Best Practice Module Statistically significant gain in knowledge for both training groups; high effect size Statistically significant difference in knowledge on post-tests between control and training groups, accounting for 35% of the variance in scores –Assessment Module Statistically significant gain in knowledge for both training groups; high effect size Statistically significant difference in knowledge on post-tests between control and training groups, accounting for 8% of the variance in scores

Results: Satisfaction with Training Kept attention91.2% SA or A94.8% Helped dev. Understand of content 96.4% SA or A94.8% Contributed to Learning: - Case-based activities - Interactive examples - Practice activities - text and written information 93.1% SA or A 86.2% SA or A 93.1% SA or A 91.3% SA or A 100% 91.4% 94.8% 96.6% Training was of high quality94.7% SA or A96.5% Relevant to my job98.2% SA or A98.3% Important resource for the future 96.5% SA or A96.6% Effective way to offer inservice training 94.8% SA or A96.5%

Results: Perceived Skill & Frequency Measure: STTS pre/post tests –Statistically significant increase in perceived skills for training groups –CoP group showed more realistic perceptions of their skills and frequency on the post-test

Current & Future Evaluation Efforts: Personnel Preparation Increased Transition Knowledge Participant satisfaction Improved Transition Planning Attention to teaching transition content Increased communication & linkages Competency Surveys PBA QI Program Evaluation Goal Attainment Scaling Focus groups Content Analysis of Discussions