When You Wish You’d Done a Pretest: Comparing Pre/Post and Retrospective Pre/Post Surveys of Knowledge and Skills Catherine Callow-Heusser, Ph.D. Audrey Hoffmann, M.S., BCBA EndVision Research and Evaluation

Think-Write-Share What have you done in the past when you haven’t collected baseline data and you wish you had?

Transition Specialist Master’s Degree Program The Utah State University Transition Specialist Program (TSP) is funded by the U.S. Department of Education - Office of Special Education Programs, Grant # 84.325K. Project staff synthesized relevant research to create a list of competencies deemed essential for teachers exiting the TSP. The competencies were organized into six domains and included evidence-based transition practices that met current needs in the field of transition. The list of competencies helped ensure that graduates of the TSP would be competent in the relevant and essential areas for a transition specialization.

Pre/Post Survey of Transition Competence Twenty TSP students completed a pre/post survey based on the transition competencies. They were asked to rate on a 5-point scale from novice to expert their knowledge and skills based on transition competencies. Nine Cohort I students completed the survey upon graduation, including a retrospective report of their initial knowledge and skills. Eleven Cohort II students completed the survey immediately prior to beginning coursework and again upon graduation from the TSP.

Timelines Cohort 1 Starts, Fall 2011 Cohort 1 Graduates, Spring/Summer 2013 Cohort 2 Starts, Fall 2013 Cohort 2 Graduates, Spring/Summer 2015 Cohort 3 Starts, Fall 2015 Transition Competencies Developed Transition Competencies Survey Administered Summer 2013 Cohort 1: Retrospective Pre, Post Cohort 2: Pre Spring 2015 Cohort 2: Post

Transition Competency Survey Assessment and Evaluation Level of Competence Novice Expert Competency Operational Definition/Examples   ⚪️ 🔘 ⚪ ⚪ ⚪ 5. Matching/aligning strengths, preferences, and interests to skills and/or demands required by post secondary, employment, residential, and other community-based settings Matching the outcomes from assessments to jobs and community-based opportunities, e.g., preference assessment results indicate a young adult prefers working inside and with children, and the young adult’s skills include talking with others, reading, providing instruction in a small group, so possible employment could be working in a day care or child care setting; results from assessments indicate a young adult would like to receive post- secondary training related to welding and the young adult possess the necessary skills to enroll in a post- secondary setting, so a possible option would be an applied technology college in a welding program. 🔘 ⚪️ ⚪ ⚪ ⚪ 6. Interpreting results of transition assessments for students, families, and other professionals Interpreting and explaining assessment results, answering questions related to assessment results, using language that is understandable when talking about assessment results, and ensuring all members of the transition planning team, including parents, understand results and potential implications.

Means and Standard Deviations SD Cohort 1 Pre 2.23 0.40 Cohort 2 Pre 2.35 0.47 Cohort 1 Post 4.11 0.31 Cohort 2 Post 4.40 0.24

Correlations Item Averages (n = 67) Correlation Cohorts 1 and 2 Pretest Correlation 0.79* Cohorts 1 and 2 Posttest Correlation 0.59* Cohort 1 Pre-Post Correlation 0.63* Cohort 2 Pre-Post Correlation 0.45* * p < 0.01

Cohort Comparisons The two cohorts were very similar at the beginning the program… And t-tests showed the groups were not statistically significantly different at pretest (p > 0.05).

Cohort Comparisons The two cohorts were very similar at the end of the program… while the t-tests showed the groups were statistically significantly different at posttest (p < 0.01), the range of average values for the items is quite restricted.

Findings Statistically significant differences between Pre/post ratings Cohorts 1 and 2 post ratings of transition competence

Assessment and Evaluation Applying results of student assessments to transition plans Using a variety of formal and informal post secondary, career, vocational, and transition assessment methods to identify transition needs Assessing young adults’ interests/preferences related to post secondary goals and educational and vocational experiences Selecting assessments with consideration for cultural and linguistic diversity and family values Matching/aligning strengths, preferences, and interests to skills and/or demands required by post secondary, employment, residential, and other community-based settings Interpreting results of transition assessments for students, families, and other professionals Coordinating/facilitating assistive technology needs assessments Using student data to evaluate effectiveness of transition teaching Observing/assessing post secondary, vocational or employment settings for potential placement opportunities Assessing natural support systems (i.e., available in existing environment) in a variety of settings Implementing strategies for shared use of student assessment data across different agencies Assessing young adults’ work- related skills in the actual training environment or workplace

Transition Planning Including appropriate team members in the transition planning process Using transition planning and communication strategies to facilitate input from team members Coordinating and facilitating IEP meetings/transition planning meetings Developing the young adult’s skills to participate to the level of their ability in the development of their IEP Developing transition goals and objectives based on the young adult’s interests and preferences Developing transition IEP goals and objectives based on strengths and needs identified from career, vocational, and transition assessments Developing transition-related goals and objectives that focus on post- secondary education or training, integrated employment, continuing and adult education, independent living, and/or community participation Including transition services and courses of study in the IEP that will reasonably enable the young adult to meet his or her post secondary goals Aligning IEP goals and objectives with state and local core standards Aligning post secondary goals with instructional activities Including assistive technology in the IEP as needed Identifying and addressing young adults’ non academic personal issues that impact transition Providing information about school, post secondary, and community services/agencies available to specific populations of individuals with exceptional needs Facilitating referrals related to school, post secondary, and community services/agencies Knowing about and applying Department of Labor regulations

Instructional Planning Knowing and applying models or philosophies of transition programs and practices Knowing how the history of national transition initiatives affects current directions Selecting and/or developing appropriate job placements Selecting and/or developing appropriate community-based educational programs Developing natural support systems that facilitate transition to post secondary environments Identifying supports needed for young adults to be successful in work experiences and training opportunities Facilitating appropriate accommodations/adaptations across a variety of settings Planning/developing educational experiences that correspond with IEP post secondary goals and objectives

Curriculum and Instruction Being familiar with and evaluating transition education methods and curricula Implementing evidence based transition practices Coordinating instructional activities related to young adult’s post secondary goals Providing community-based instruction Teaching self-advocacy and self- determination skills Teaching appropriate job readiness skills based on developmental needs and functional limitations Teaching daily living skills Teaching vocational/work-related skills Teaching social skills Teaching learning strategies and study skills Providing experiences and teaching leisure and recreational skills Implementing behavior management strategies in academic, community and/or work environments Teaching young adults how to access and use assistive technology in academic, work, and community environments if needed Teaching job retention skills identified by employers as essential for successful employment Providing experiences to increase awareness and knowledge of careers Ensuring young adults with disabilities have the same career and vocational development opportunities as their peers without disabilities

Communication and Collaboration Providing transition-related resources and materials to students, parents, educators, service providers, and employers Facilitating and/or providing training on transition-related topics Monitoring participation in transition planning and implementation Knowing and implementing procedures and requirements for referring young adults to community service agencies and other post-secondary organizations Resolving differences that may arise in collaborative relationships and interagency agreements Training, managing, coordinating and evaluating job coaches and school-based paraprofessionals working with young adults in vocational and community settings Collaborating with professionals (e.g., general educators, administrators, group home providers, employers, etc.) to improve transition outcomes Facilitating partnerships among service providers within the school and community to maximize transition outcomes Specifying roles and responsibilities of each team member and/or agency

Family Involvement Ensuring transition processes and outcomes are consistent with families’ cultures, beliefs, practices, and values Including and collaborating with parents and family members in planning, decision-making, and implementing transition services Providing information to families about transition services, programs, and post- secondary options for their young adult Identifying family service needs related to transition outcomes Assisting families to network with extended family, neighbors, church, employers, etc., to promote transition outcomes Developing relationships to ensure commitment of family members or other caretakers to assist in transition Working with family systems and the dynamics of individual families and understanding how systems impact the transition process

Effect Size Differences The Hedge’s g (IES Procedures and Standards Handbook, ver. 3.0, 2013) measure of effect size is calculated by dividing the difference in means by the pooled standard deviation.

Effect Size Differences Hedge’s g Cohorts 1 and 2 Pretest Comparison 0.27 Cohorts 1 and 2 Posttest Comparison 1.05* Cohort 1 Pre-Post Comparison 5.29* Cohort 2 Pre-Post Comparison 5.56* * p < 0.01

Retrospective Surveys In what cases can retrospective surveys of knowledge and skills work? When will they NOT work?

Discussion & Questions Cathy Callow-Heusser Secondary Math Specialist State Systemic Improvement Plan Multi-Tiered System of Supports Project Utah State Office of Education 801-538-7952 Cathy.Callow-Heusser@schools.utah.gov President EndVision Research and Evaluation, LLC cheusser@endvision.net 435-757-2724