1. Zarrow Center TAGG: A New On-Line Transition Assessment Jim Martin and Amber McConnell Dept. of Educational Psychology Zarrow Center Dept. of Educational Psychology University of Oklahoma

2. Zarrow Center Agenda 1.Legal Requirements and Overview 2.TAGG Overview 3.TAGG Development 4.TAGG Demonstration 5.TAGG Overview 6.Select Validity Evidence 7.Obtain the TAGG

3. Zarrow Center Transition Assessment Legal Requirements & Overview

4. Zarrow Center 4 The Purpose of SPED... a free appropriate public education that emphasizes special education and related services designed to meet students’ unique needs and to prepare them for further education, employment, and independent living.

5. Zarrow Center IDEA 2004 Goals IEPs must include postsecondary and annual transition goals – based upon age-appropriate transition assessment s – related to training, education, employment, and independent living (when needed) Consider independent living is always needed unless data say otherwise

6. Zarrow Center Transition & The Courts Prince, Plotner, & Yell (2014) examined district court findings and recommend – Use multiple assessments across transition domains – DO not solely use informal assessments This means at least one transition assessments need supporting validity evidence – Maximize student participation in the transition planning process

7. Zarrow Center DCDT Transition Standards Use valid and reliable transition assessments Use assessments on an on-going basis Use assessments to identify student interests, skills, and needs Interpret results for students and families Involve students in transition planning

8. Zarrow Center Basic Validity and Reliability Questions What is the purpose? Who was it designed for? Where did items come from? Does ample evidence exist for – Firm factor structure – Internal reliability – Test-retest reliability – No or minimal bias by gender, placement, GPA, grade, disability category, SES – Predictive validity

9. Zarrow Center Case Law Decision Case involved not conducting adequate transition assessments. Decision: Told school to focus on meaningful non-academic goals to prepare students for post-school life. Massachusetts Bureau Of Special Education Appeals And Currently Under Appeal In Federal Court, Dracut Public Schools, BSEA #08- 5330, 15 MSER 78 (2009).

10. Zarrow Center Transition Goals Postsecondary Goals set the direction Annual Transition Goals teach students essential skills needed to obtain postsecondary goals. – Postsecondary goal often change – Build annual transition goals using generalizable non-academic behaviors Behaviors useful for any postsecondary goal

11. Zarrow Center TAGG OVERVIEW

12. Zarrow Center TAGG Developed with a Grant from the National Center for Special Education Research and OU Zarrow Center Funds THANKS TO NATIONAL CENTER FOR SPECIAL EDUCATION RESEARCH

13. Zarrow Center Purpose Assess non-academic skills associated with and predictive of post-school further education and employment To provide IEP team student strengths, needs, a written summary, and annual transition goals matched to common core standards to facilitate writing I-13 compliant IEPs

14. Zarrow Center Who? TAGG Designed to Assess – Secondary-aged students with IEPs who plan to be competitively employed and/or enrolled in further education after graduation Each TAGG set includes 3 versions – Student – Family – Professional

15. Zarrow Center TAGG Web-Generated Results Profile Graphic results by constructs Written summary Relative and greatest strengths Relative and greatest needs Annual transition goals Components may be copied and pasted into IEP

16. Zarrow Center Disability Awareness Profile

17. Zarrow Center Combined Score Profile

18. Zarrow Center Combined Score The overall score is a weighted combination of all items. The overall score is not an average of all the construct scores.

19. Zarrow Center Summary Statement for IEP Chad Bailey’s skills were assessed using the TAGG, a norm-referenced assessment with research-based items known to be associated with post-school employment and education. Compared to similar students, Chad’s scores are average. Results indicate greatest strengths are in the areas of Goal Setting and Attainment. Chad’s relative strengths include Disability Awareness and Student Involvement in the IEP. Greatest needs are in the area of Strengths and Limitations, with Employment being a relative need.

20. Zarrow Center Greatest and Relative Strengths

21. Zarrow Center Areas of Greatest and Relative Need

22. Zarrow Center Suggested Annual Transition Goals To prepare for success in employment, the student will write an essay describing three situations where the student used his or her strengths with 90% grammar and context accuracy by the end of the essay writing unit.

23. Zarrow Center Versions and Format On-line written English – May be printed and taken by hand, but item scores must be entered into website to produce results – In next few months a TAGG versions in Spanish, will be added If needed now we have a paper version Users may choose to listen to audio or watch ASL videos for TAGG instructions and items

24. Zarrow Center Reading Level Professional 10.4 grade level Family 5.7 grade level Student 4.8 grade level

25. Zarrow Center TAGG DEVELOPMENT

26. Zarrow Center Standards for Educational and Psychological Assessment Guided TAGG Development

27. Zarrow Center Began Work on TAGG 10 Years Ago Studies began to appear identifying non- academic behaviors associated with post-high school education and employment Doctoral Seminar on Transition Education Located many studies and created pilot assessment Socially validated assessment Used this pilot to obtain IES funding

28. Zarrow Center Development of TAGG Items TAGG items derived from research studies that identified behaviors of former students with disabilities engaged in post-high school employment and/or further education The research team initially used the research studies to develop – 10 construct definitions – Items developed from constructs 15 iterative TAGG versions were created before field testing began

29. Zarrow Center Initial Structure: Ten Initial Constructs Knowledge of strengths and limitations Actions related to strengths and limitations Disability awareness Employment Goal setting and attainment Persistence Proactive involvement Self-advocacy Supports Utilization of resources

30. Zarrow Center

31. Establish Initial TAGG Structure Users from six states completed the initial test-version TAGG – 349 high school students with disabilities – 271 family members – 39 professionals Applied various factor analyses statistics Went from 10 constructs to 8 Went from 75 items to 34

32. Zarrow Center After FA: Professional and Family TAGG Stayed 1.Strengths and Limitation 2.Disability Awareness 3.Persistence 4.Interacting with Others 5.Goal Setting and Attainment 6.Employment 7.Student Involvement in IEP 8.Support Community Dropped 1.Actions Related to Strengths and Limitation 2.Utilization of Resources TAGG-P: (    df=499, RMSEA=.058, CFI=.92, TLI=.91, RMSR=.0597) TAGG-F: (  2 =862.74, df=499, RMSEA=.057, CFI=.91, TLI=.90, RMSR=.058)

33. Zarrow Center Student Version Constructs After FA After FA Constructs 1.Strengths and Limitations & Support Community 2.Disability Awareness 3.Persistence 4.Student Involvement in IEP 5.Interacting with Others 6.Goal Setting and Attainment 7.Employment Dropped Constructs 1.Actions Related to Strengths and Limitation 2.Utilization of Resources Combined Constructs 1.Strengths and Limitations 2.Support Community TAGG-S: (  2 =819.00, df=505, RMSEA=.047, CFI=.89, TLI=.88, RMSR=.064)

34. Zarrow Center Item Response Theory Advantages of IRT include – The ability to scale different item types – Provides a common metric for scales with different number of items – Weights items differentially by their validity for assessing the construct of interest

35. Zarrow Center IRT Algorithms Produce Results Profile 1.Placed each scale onto a common score metric 2.Projected item characteristics (e.g. item difficulty) onto the scale score metric 3.Conducted a within-student comparison of scale scores across constructs to determine relative strengths and weaknesses 4.Conducted a within-construct comparison of a student’s scale score to item responses (e.g. difficulty) to generate appropriate goals for identified weaknesses

36. Zarrow Center TAGG DEMONSTRATION

37. Zarrow Center Use of TAGG Website From Around the World

38. Zarrow Center Use of TAGG Website Across US

39. Zarrow Center SELECTED VALIDITY EVIDENCE

40. Zarrow Center Available at NO Cost At the OU Zarrow Center’s Web Site

41. Zarrow Center Three Years of Data Collection 2,556 participants from 42 states and 162 schools – 1,291 secondary students with disabilities who had postsecondary further education and/or competitive employment goals – 172 professional completed TAGG on 7 to 8 of their students – 847 family members completed TAGG on their child

42. Zarrow Center Replicated Factor Analysis Used Categorical Factor Analytic Procedures 846 Student Participants in the Phase 3 study Fit Indices – all very good – Fits a simple structure model (no cross loading and no correlated errors), which is fabulous! RMSEA results acceptable if less than.08) CFI and TLI acceptable if greater than.90 – TAGG P: (  2 =1190.41, df=499, RMSEA=.065, CFI=.97, TLI=.97) – TAGG F: (  2 =914.26, df=499, RMSEA=.06, CFI=.95, TLI=.95) – TAGG S: (  2 =870.84, df=499, RMSEA=.04, CFI=.95, TLI=.94)

43. Zarrow Center Predictive Validity Process Follow-up of 297 former high school students who completed the TAGG while in high school – One to Two Years After High School Logistic regressions examined relations between TAGG non-academic behavior constructs and postsecondary education and employment

44. Zarrow Center Constructs Predicting Further Education Student Involvement in the IEP Interacting with Others Support Community Goal Setting and Attainment

45. Zarrow Center Constructs Predicting Employment Student Involvement in IEP Interacting with Others Support Community Employment

46. Zarrow Center Internal Reliability Generally a score between.7 and.8 is considered “good” – Each TAGG version has great overall internal consistency and satisfactory subscale consistency (ranging from α =. 89 to α =. 95)

47. Zarrow Center Test-Retest Reliability Scores of.7 or higher represent good or satisfactory test-retest reliability – 14 weeks after the first TAGG was completed, same users completed the TAGG again. – A large correlation was found between the first and the second administration.80 for professional TAGG.70 for family TAGG.70 for student TAGG

48. Zarrow Center Impact of Placement and GPA 650 students, 72 high schools, across 9 states TAGG and General Education Placement – No meaningful connection TAGG and GPA – Overall TAGG: Non significant results – TAGG Versions: nothing meaningful Implications??

49. Zarrow Center

50. Fairness Validity Evidence: Gender Do differences exist by gender? – No overall difference by gender on TAGG-P, TAGG-S – On TAGG-F small overall differences – Some construct differences exist. On TAGG-S – females rated themselves higher on student involvement than males – Males rated higher on employment

51. Zarrow Center

52. Structure Fit By Disability Category Structure held across all disability categories The three TAGG versions are appropriate for students regardless of – Students’ time in general education classes – Number of transition education classes completed – High School grade level – Disability category

53. Zarrow Center Fairness Validity Evidence: Disability Categories Construct Differences were as expected but small – On most constructs OHI and LD higher than Autism, ED, and ID Students with ED and Autism lower scores on interacting with others

54. Zarrow Center For Example, Students with Autism Students with Autism scored themselves as having no problems with interacting with others. – (Professionals and Family members scored students with Autism lower than ID, OHI, SLD) Students with Autism did not score significantly higher than any other disability group on any construct.

55. Zarrow Center Free/reduced lunch eligibility No significant differences for construct scores on TAGG-P or TAGG-S. Only small differences for TAGG-F scores. Family employment No significant differences for construct scores Family education Significant differences- Highest family education lower TAGG scores Fairness Validity Evidence: SES

56. Zarrow Center How Close Are Students, Professionals, and Family TAGG Scores? How closely do the different TAGG versions assess the same student? – Medium correlations across Parent, Educator, and Student versions when assessing the same student. – This is excellent for this type of assessment

57. Zarrow Center TAGG & AIR Self-Determination Assessment Same users completed TAGG and AIR Self- Determination Assessment – Medium Correlation This implies the TAGG addresses some self- determination skills and assesses other skills, too.

58. Zarrow Center OBTAIN THE TAGG

59. Zarrow Center TAGG Details $3 per set (Professional, Student, Family versions) – Used to pay for on-going TAGG development and operational costs TAGG profiles saved for 7 years Data kept on OU high-speed secure cloud servers Purchased credits may be transferred to other registered TAGG users Unused credits refunded for one-year after purchase

60. Zarrow Center TAGG Web Site Location 1.The OU Zarrow Center’s Web Page http://zarrowcenter.ou.edu 2.The TAGG Section of the ZC Web Page https://tagg.ou.edu/tagg

61. Zarrow Center Contact Information Jim Martin jemartin@ou.edu Amber McConnell ambermcc@ou.edu Phone: (405) 325-8951 Website: https://tagg.ou.edu/tagg/https://tagg.ou.edu/tagg/