1. Who are the Students in Alternate and Modified Achievement Standards Assessments? Jacqueline F. Kearns, Ed.D., NAAC Martha Thurlow, Ph.D., NCEO Elizabeth Towles-Reeves, Ph.D., NAAC OSEP Project Directors’ Conference July 22, 2008

2. Part I Alternate Achievement Standards Assessments

3. Topics 1. Alternate Achievement Standards Assessments and the Validity Evaluation 2. Current Research from NAAC-Who are the Students in Alternate Achievement Standards Assessments? 3. Implications for the Validity Evaluation

4. OBSERVATION INTERPRETATION COGNITION  Student Population  Academic content  Theory of Learning  Assessment System  Test Development  Administration  Scoring  Reporting  Alignment  Item Analysis & DIF/Bias  Measurement error  Scaling and Equating  Standard Setting VALIDITY EVALUATION  Empirical evidence  Theory & logic (argument)  Consequential features The Assessment Triangle & Validity Evaluation Marion & Pellegrino (2006)

5. Cognition Vertex Validity Questions 1) Is the assessment appropriate for the students for whom it was intended? 2) Is the assessment being administered to the appropriate students? Both are important for the validity evaluation

6. Issues in Teaching/Assessing Students in Alternate Achievement Standards Assessments Varied levels of symbolic communication Attention to salient features of stimuli Memory Limited motor response repertoire Generalization Self-Regulation Meta-cognition Skill Synthesis Sensory Deficits Special Health Care Needs Kleinert, H., Browder, D., Towles-Reeves, E. (in press). Models of cognition for students with significant cognitive disabilities: Implications for assessment. Review of Educational Research. Kleinert, H., Browder, D., Towles-Reeves, E. (in press). Models of cognition for students with significant cognitive disabilities: Implications for assessment. Review of Educational Research.

7. Learner Characteristics Demographic Variables Learner Characteristics (all on a continuum of skills): Expressive Language Receptive Language Vision Hearing Motor Engagement Health Issues/Attendance Reading Mathematics Use of an Augmentative Communication System (dichotomous variable)

8. Methodology Seven partner states chose to participate during the 2006-2007 school year. States 1, 2, 3, 4, 5, and 6: gathered data in the administration process for their alternate achievement standards assessment (i.e., bubble sheet, paper/pencil version of the LCI, etc.) State 7: gathered data using Zoomerang, an online survey package.

9. States & LCI Response Rates StateGeographyParticipation RateSample NResponse Rate State 1North East0.96%2793100% State 2Mid West1.17%2216100% State 3East1.14%359575% State 4North East0.99%72293% State 5South East0.70%213487% State 6East0.76%46891% State 7West0.94%21947%

11. Expressive Language

12. Receptive Language

13. Use of Augmentative Communication Systems

14. Reading

15. Mathematics

16. Expressive Language Across Grade Bands

17. Reading Across Grade Bands

18. Mathematics Across Grade Bands

19. Who are the Kids in Alternate Achievement Standards Assessments? Represent ~1% or less of the total assessed population All disability categories were represented but primarily 3 emerge, Mental Retardation Multiple Disabilities Autism Highly varied levels of expressive/receptive language use Most students in the population use symbolic communication Level of symbolic language use does not significantly change across grade- bands The majority of students do not use AAC Most of the population read basic sight words and solve simple math problems with a calculator. Changes in skill progression in reading and math across grade bands most likely due to identification of students rather than teaching and learning

20. Cognition Vertex: Validity Evaluation Essential Questions Who is the population being assessed? Who is the population being assessed? How do we document and monitor the population? How do we document and monitor the population? What do we know about how they learn (theory of learning) academic content? What do we know about how they learn (theory of learning) academic content? What do our assessment results tell us about how the population is learning academic content? What do our assessment results tell us about how the population is learning academic content? Are our data about the population and theory of learning consistent with student performances on the assessment? Are our data about the population and theory of learning consistent with student performances on the assessment? If not, what assumptions are challenged? If not, what assumptions are challenged? What adjustments should be made? What adjustments should be made? Participation Participation Theory of Learning Theory of Learning Student Performance Student Performance

21. Alternate Achievement Standards Assessments References References Kleinert, H., Browder, D., Towles-Reeves, E. (in press). Models of cognition for students with significant cognitive disabilities: Implications for assessment. Review of Educational Research. Kleinert, H., Browder, D., Towles-Reeves, E. (in press). Models of cognition for students with significant cognitive disabilities: Implications for assessment. Review of Educational Research. Marion, S., & Pellegrino, J. (2006). A validity framework for evaluating the technical quality of alternate assessments. Educational Measurement: Issues and Practices, 25(4), 47-57. Marion, S., & Pellegrino, J. (2006). A validity framework for evaluating the technical quality of alternate assessments. Educational Measurement: Issues and Practices, 25(4), 47-57. Additional Resource Additional Resource Towles-Reeves, E., Kearns, J., Kleinert, H., & Kleinert, J. (2008, May 12). An analysis of the learning characteristics of students taking alternate assessments based on alternate achievement standards. Journal of Special Education. Retrieved June 2, 2008, from http://sed.sagepub.com/cgi/rapidpdf/0022466907313451v1. Towles-Reeves, E., Kearns, J., Kleinert, H., & Kleinert, J. (2008, May 12). An analysis of the learning characteristics of students taking alternate assessments based on alternate achievement standards. Journal of Special Education. Retrieved June 2, 2008, from http://sed.sagepub.com/cgi/rapidpdf/0022466907313451v1. http://sed.sagepub.com/cgi/rapidpdf/0022466907313451v1

22. Part II Modified Achievement Standards Assessments

23. Topics 1.Research and Regulation Advice 2.Current Practice 3.GSEG Project Work

24. Why Start from the Student? National Research Council – Pellegrino, Chudowsky, & Glaser, (2001) – Knowing What Students Know Luecht (2007) - good assessment design would apply human factors engineering principles by developing cognitive maps and cognitive construct models Pellegrino (2007) - Principled Assessment Design process that started with a clear student model as the basis for an evidence model that would, in turn, serve as the basis for a task model.

25. Why Start from the Student? Mislevy and Haertel (2007) - central point of agreement of these models is the necessity of first developing a good understanding of how people do or fail to do what is to be measured. Then tasks can be developed that let us observe what people do so we are able to make inferences that are more fully supported by clearer evidence.

26. Assessment as a Process of Reasoning from Evidence Cognition –model of how students represent knowledge & develop competence in the domain Observations –tasks or situations that allow one to observe students’ performance Interpretation –method for making sense of the data observationinterpretation cognition Must be coordinated! The Assessment Triangle

27. Cognition Vertex Validity Questions 1)Is the assessment appropriate for the students for whom it was intended? 2)Is the assessment being administered to the appropriate students? Both are important for the validity evaluation

28. April 9, 2007 Regulations Address Who the Students Are Preamble: The final regulations intentionally do not prescribe which students with disabilities are eligible to be assessed based on modified academic achievement standards; that is the determination of a student’s IEP Team, which includes the student’s parents, based on criteria developed by the State as part of the State’s guidelines for IEP Teams

29. April 9, 2007 Regulations Address Who the Students Are The student’s progress to date in response to appropriate instruction..., is such that, even if significant growth occurs, the IEP team is reasonably certain that the student will not achieve grade-level proficiency within the year covered by the student’s IEP. Section 200.1(e)(2)(ii)

30. April 9, 2007 Regulations Address Who the Students Are Inform IEP teams that students eligible to be assessed based on alternate or modified academic achievement standards may be from any of the disability categories listed in the IDEA. Section 200.1(e)(2)(ii)

31. From Cortiella (2007), Learning Opportunities for Your Child Through Alternate Assessments – Alternate Assessments based on Modified Achievement Standards

32. Current Practice In 2007, 5 states had assessments that they believed to be an AA-MAS before the April, 2007 regulation release – see Lazarus, Thurlow, Christensen, & Cormier (2007) Update study now being conducted – have pulled information on eligibility for AA- MAS

33. Eligibility Criteria in States In 2008, 10 states had assessments that they believed to be an AA-MAS CaliforniaNorth Carolina ConnecticutNorth Dakota KansasOklahoma LouisianaTexas MarylandVirginia

34. Number of States With Selected Eligibility Criteria Format of Criteria # of States Descriptions (bullets, chart, written description)10 Flowchart/Decision Tree 4 Checklist 5 2008 Study

35. Number of States With Selected Eligibility Criteria Disability Status Criteria # of States Student has IEP 10 Not based on disability category label 5 Not due only to ELL designation or being on 504 plan 3 Not due to being identified as having a significant cognitive disability 2 Student may be in any of the disability categories 6 2008 Study

36. Number of States With Selected Eligibility Criteria Classroom Learning/Teaching Criteria # of States Student not progressing at rate expected to reach grade level proficiency within school year covered by IEP 8 Student is learning grade-level content 6 Not due to receiving instruction based on extended or alternate standards or being eligible to take AA-AAS 6 Not based on attendance (extended/excessive absence) 5 Student receives specialized instruction 3 2008 Study

37. Number of States With Selected Eligibility Criteria Classroom Learning/Teaching Criteria – cont. # of States Student requires differentiated content for classroom assessment 3 Student needs accommodations during classroom instruction 2 Student’s classroom achievement and performance significantly below grade-level peers 2 Student consistently requires instruction in pre- requisite skills to the grade-level indicators 2 2008 Study

38. Number of States With Selected Eligibility Criteria Classroom Learning/Teaching Criteria – cont. # of States Not based on placement setting 2 Not based on amount of time in general or special education services 2 2008 Study

39. Number of States With Selected Eligibility Criteria Previous Performance Criteria # of States Student passed or failed AA-AAS or other large-scale tests 5 Student cannot demonstrate knowledge on regular assessment even with accommodations 3 Student has been tested on multiple, valid, objective measures over time 3 Student’s previous performance on multiple measures is considered 3 2008 Study

40. Number of States With Selected Eligibility Criteria IEP Goals & Other # of States IEP includes goals that are based on grade-level content standards 4 Not due to social, cultural, language, economic, or environmental factors 2 2008 Study

41. Identifying Accommodations for AA-MAS State Approaches Accommodations issues for regular assessment Integration of “accommodations” and universal design principles into the regular assessment first, then in the design of the AA-MAS

42. State Approaches – Accommodations Incorporated into AA-MAS Design Accommodation No. of States Fewer items/page3 Larger font size3 Calculator2 Breaks as needed2 Key text underlined/bolded1

43. State 1 (long time ) Reading = 19%Math = 17% State 2 (long time) Reading = 10.5%Math = 9.9% AA-MAS Participation Rates (2006-07) State 3 (long time) Reading = 23%Math = 21% State 4 (newer) Reading = 31%Math = 29% Rates based on # students with IEPs

44. State 1 (long time ) Reading = 24%Math = 33% State 2 (long time) Reading = 4.3%Math = 2.6% AA-MAS Rates Proficient (2006-07) State 3 (long time) No DataNo Data State 4 (newer) Reading = 52%Math = 54% Rates based on # students with IEPs

46. Alternate Achievement Standards Assessments: Alternate Achievement Standards Assessments: Consideration of students with high reading and math abilities Consideration of students with high reading and math abilities Assessment design for a highly varied population Assessment design for a highly varied population Considering symbolic language use Considering symbolic language use Skill progressions in reading and math Skill progressions in reading and math Considerations

47. Considerations Modified Achievement Standards Assessments: Modified Achievement Standards Assessments: Moving from student characteristics to an assessment based on grade-level content, but with modified achievement standards Moving from student characteristics to an assessment based on grade-level content, but with modified achievement standards Clearly defining the relationships among the general assessment, the AA-AAS, the AA-MAS, and the AA- GLAS, if one exists Clearly defining the relationships among the general assessment, the AA-AAS, the AA-MAS, and the AA- GLAS, if one exists Separating instructional issues from assessment issues Separating instructional issues from assessment issues Providing training and assistance for good decisions about who needs which assessment Providing training and assistance for good decisions about who needs which assessment

48. Contact Information Jacqueline Kearns, Ed.D. 1 Quality Street, Suite 722 Lexington, Kentucky 40507 859-257-7672 859-323-1838 Jacqueline.kearns@uky.edu Elizabeth Towles-Reeves, Ph.D. 1 Quality Street, Suite 722 Lexington, Kentucky 40507 859-257-7672 859-323-1838 Liztowles-reeves@uky.edu http://www.naacpartners.org/

49. Contact Information Martha Thurlow, Ph.D. 207 Pattee Hall 150 Pillsbury Drive SE Minneapolis, MN 55455 612-624-4826 THURL001@umn.edu www.nceo.info