Alternate Assessments on Alternate Achievement Standards Student Population Jacqueline F. Kearns, Ed.D. Elizabeth Towles-Reeves, MS

The Assessment Triangle & Validity Evaluation Marion & Pellegrino (2006) OBSERVATION INTERPRETATION COGNITION Assessment System Test Development Administration Scoring Reporting Alignment Item Analysis & DIF/Bias Measurement error Scaling and Equating Standard Setting VALIDITY EVALUATION Empirical evidence Theory & logic (argument) Consequential features Student Population Academic content Theory of Learning

Cognition Vertex Validity Questions Is the assessment appropriate for the students for whom it was intended? Is the assessment being administered to the appropriate students? Both are important for the validity evaluation

More Different Than Alike 2% Mulitple disabilities (Student example: Leslie) SOURCE: Education Week analysis of data from the U.S. Department of Education, Office of Special Education Programs, Data Analysis System, 2002-03

Issues in Teaching/Assessing Students in Alternate Assessments Varied levels of symbolic communication Attention to salient features of stimuli Memory Limited motor response repertoire Generalization Self-Regulation Meta-cognition Skill Synthesis Sensory Deficits Special Health Care Needs Kleinert, H., Browder, D., & Towles-Reeves, E. (2005). The assessment triangle and students with significant cognitive disabilities: Models of student cognition. National Alternate Assessment Center, Human Development Institute, University of Kentucky, Lexington. (PDF File)

Previous Data 165 Students across 7 states Extensive documentation through 111 item inventory Findings suggest: 64% routinely use verbal language 46% routinely understand pictures used to represent objects 11% don’t understand pictures used to represent objects. Almond & Bechard (2005) An In Depth Look at students who take alternate assessments: What do we know. Colorado EAG.

Learner Characteristics Demographic Variables Learner Characteristics (all on a continuum of skills): Expressive Language Receptive Language Vision Hearing Motor Engagement Health Issues/Attendance Reading Mathematics Use of an Augmentative Communication System (dichotomous variable)

Methodology Four partner states chose to participate States 1, 2, and 3: gathered data in the administration process for their AA-AAS via scannable document (i.e., bubble-sheet) State 4: gathered data using Zoomerang, an online survey package. N= 7,075

States & LCI Response Rates Geography Demographic Sample N Response Rate 1 Eastern Rural Suburban 3595 75% 2 North Eastern Urban- 2793 100% 3 Urban 468 91% 4 Western 219 47%

Alternate Assessment Participation Rates : % Total population State 1 .959% State 2 1.14% State 3 .766% State 4 .55%

“Most significant cognitive disabilities” Those kids that are PreSymbolic and also have low vision, low motor, and low engagement characteristics. Same for Emerging and Symbolic! “Most significant cognitive disabilities”

Expressive Language To communicate expressively, most students in each state used verbal or written words, signs, Braille, or language-based augmentative systems to request, initiate, and respond to questions, describe things or events, and express refusal (79%, 70%, 78%, and 75% respectively in States 1, 2, 3 and 4). A smaller group of the population in each state used understandable communication through such modes as gestures, pictures, objects/textures, points, etc., to clearly express a variety of intentions (13%, 18%, 13%, 17% respectively). An even smaller group of students primarily used cries, facial expressions, change in muscle tone, etc., to communicate, but these students had no clear use of objects/textures, regularized gestures, pictures, signs, etc., to communicate (8%, 11%, 9%, and 8% respectively).

Receptive Language Receptively, students in each state fell into two primary groups: those students who independently followed 1-2 step directions presented through words (e.g. words could be spoken, signed, printed, or any combination) while not requiring additional cues (53%, 45%, 55%, and 56% respectively in state 1, 2, 3, and 4); or those students who required additional cues (e.g., gestures, pictures, objects, or demonstrations/models) to follow 1-2 step directions (37%, 43%, 36%, and 34%). A smaller group (8%, 10%, 8%, and 7%) alerted to sensory input from another person (auditory, visual, touch, movement) but required actual physical assistance to follow simple directions. Finally, less than three percent of the population in each state displayed an uncertain response to sensory stimuli (e.g., sound/voice; sight/gesture; touch; movement; smell).

Use of Augmented Communication In State 1, results suggest that 47.2% of the students identified as communicating “primarily through cries, facial expressions, change in muscle tone, etc., but with no clear use of objects/textures, regularized gestures, pictures, signs, etc.” do not use a communication system. In addition, 54.6% of the students identified as “using intentional communication but not at the symbolic level” do not use a communication system. In State 2, results suggest that 51.2% of the students identified as communicating “primarily through cries, facial expressions, change in muscle tone, etc., but with no clear use of objects/textures, regularized gestures, pictures, signs, etc.” do not use a communication system. In addition, 49.7% of the students identified as “using intentional communication but not at the symbolic level” do not use a communication system. In State 3, results suggest that 23.3% of the students identified as communicating “primarily through cries, facial expressions, change in muscle tone, etc., but with no clear use of objects/textures, regularized gestures, pictures, signs, etc.” do not use a communication system. In addition, 53.2% of the students identified as “using intentional communication but not at the symbolic level” do not use a communication system. In State 4, results suggest that 77% of the students identified as communicating “primarily through cries, facial expressions, change in muscle tone, etc., but with no clear use of objects/textures, regularized gestures, pictures, signs, etc.” do not use a communication system. In addition, 57% of the students identified as “using intentional communication but not at the symbolic level” do not use a communication system.

Reading In States 1, 2, and State 3, teachers noted that 2-4% of the population read fluently with critical understanding in print or Braille. State 2 did not provide this option on the inventory. 15% of the students in State 1, 17% in State 2, and 15% in State 3, and 34% in State 4 were rated as being able to read fluently with basic (literal) understanding from paragraphs/short passages with narrative/ informational texts in print or Braille. The majority of students (50%, 46%, 39%, and 33%) were rated as being able to read basic sight words, simple sentences, directions, bullets, and/or lists in print or Braille, but not fluently from text with understanding. A smaller percentage of students (21%, 18%, 25%, and 18%) were rated as not yet having a sight word vocabulary, but being aware of text/Braille, following directionality, making letter distinctions, or telling a story from pictures. Finally, teachers noted 13% of students in State 1, 20% of students in State 2, and 17% of students in State 3, and 13% of students in State 4 had no observable awareness of print or Braille.

Mathematics Across the states, 3-6% of students applied computational procedures to solve real-life or routine word problems from a variety of contexts. The largest category of students within each state (57%, 44%, 45%, and 51% respectively) was able to complete computational procedures with or without a calculator. Nearly 21% of students in State 1, 25% of students in State 2, and 23% of students in State 3, and 27% of students in State 4 were described as performing at the more basic level of counting with one-to-one correspondence to at least 10, and/or make numbered sets of items. A smaller percentage still (6%, 10%, 13%, and 6%) were described as being able to count by rote to 5, but without the higher skill sequences of one-to-one correspondence or computation. Finally, teachers noted that nearly 12% of students in State 1, 18% of students in State 2, and 14% of students in State 3, and 12% of students in State 4 had no observable awareness or use of numbers.

Skill level 1: Students with 3 for Expressive, 4 for Receptive, 3-5 for Reading and 3-5 for Math

Who are the Kids? Represent ~1% or less of the total assessed population All disability categories were represented but primarily 3 emerge, Mental Retardation Multiple Disabilities Autism Highly varied levels of expressive/receptive language use Most students in the population use symbolic communication Level of symbolic language distribution is similar across grade-bands Only about 50% of the pre and emerging symbolic language users use ACS Pre-symbolic expressive language users are more likely to have additional complex characteristics. Most of the population read basic sight words and solve simple math problems with a calculator. Lack of skill progression in reading across grade bands (elementary, middle & high) Skill progression apparent in mathematics across grade bands but still small

Limitations Only four state participants Small sample size Global items in reading and math Participation rates at 1% or less

Cognition Vertex: Validity Evaluation Essential Questions Who is the population being assessed? How do we document and monitor the population? What do we know about how they learn (theory of learning) academic content? What do our assessment results tell us about how the population is learning academic content? Are our data about the population and theory of learning consistent with student performances on the assessment? If not, what assumptions are challenged? What adjustments should be made? Participation Theory of Learning Student Performance

References Agran, M., Fodor-Davis, Moore, & Martella, (1992). Effects of peer-delivered self-instructional training on a lunch-making task for students with severe disabilities. Education and Training in Mental Retardation, 27, 230-240. Billingsley, F., Gallucci, C., Peck, C., Schwartz, I., & Staub, D. (1996).  "But those kids can't even do math:  An alternative conceptualization of outcomes in special education.  Special Education Leadership Review, 3 (1), 43-55. Brown, L., Nisbet, J., Ford, A., Sweet, M., Shiraga, B., York, J., Loomis, R. (1983). The critical need for non-school instruction in educational programs for severely handicapped students. Journal of the Association of the Severely Handicapped. 8, 71-77. CAST (2002). Fox, (1989). Stimulus Generalization of skills and persons with profound mental handicaps. Education and Training in Mental Retardation, 24,219-299. Haring, N. (1988). Generalization for students with severe handicaps: Strategies and solutions. Seattle, WA: University of Washington Press. Hughes, C. & Agran, M. (1993). Teaching persons with severe disabilities to use self-instruction in community settings: An analysis of the applications. Journal of the Association for Persons with severe Handicaps, 18, 261-274. Hughes, C., Hugo, K., & Blatt, J. (1996). Self-instructional intervention for teaching generalized problem-solving with a functional task sequence. American Journal of Mental Retardation, 100 565-579. Westling, D., & Fox, L. (2004). Teaching Students with Severe Disabilities. Columbus: Pearson (Merrell). Whitman, T. L. (1990). Self-regulation and mental retardation. American Journal on Mental Retardation, 94, 347-362.

Contact Information Jacqueline Kearns, Ed.D. Elizabeth Towles-Reeves, MS 1 Quality Street, Suite 722 Lexington, Kentucky 40507 859-257-7672 X 80243 859-323-1838 Jacqueline.kearns@uky.edu 1 Quality Street, Suite 722 Lexington, Kentucky 40507 859-257-7672 X 80255 859-323-1838 Liztowles-reeves@uky.edu www.naacpartners.org