Consequential Validity A presentation of methods and selected results

Background Policy, Validity Framework, Study Design

Peer Review Guidance - Critical Element 4.1 (p.35) “Has the State ascertained that the decisions based on the results of its assessments are consistent with the purposes for which the assessments were designed?” “Has the State ascertained whether the assessment produces intended and unintended consequences?”

An Argument-based Validity Framework State the intended consequences Define the evidence that supports each consequence Collect data to ascertain the intended consequences Collect data to ascertain the unintended consequences

Original Validity Arguments (Lane & Stone, 2004) Teacher and administrator motivation and effort will be evident. Professional development support will be provided. Instruction and curriculum will be adapted. Student motivation & effort will be evident. Performance will improve and be related to the changes above.

Additional Validity Arguments NCRRC & 3 State Consortium Access to augmentative and alternative communication will be improved. Parental involvement in student’s academic career will be increased.

Collecting Data Teacher Surveys Classroom Observation Administrator Surveys

Survey Data Analysis Procedures Design Arguments Stakeholders Sample Stratify Random Cluster Collect Data Pilot Version 1 Version 2

Survey Data Analysis Procedures Prepare Data CFA Rasch Scaling Analyze Descriptive HLM Discriminant Function Conclude? Evidence of Consequential Validity

Classroom Observation Procedures Identify Elements Teaching IEP Development Student Profile Observe Modified NAAC Observation Tool Sample in a sample Evaluate Is there evidence of consequential validity?

Examining Changes Evidential Variables Contextual Variables Student Performance (Slope of Change) Contextual Variables (Relatively Stable) Evidential Variables

Baseline Data Contextual Variables Evidential Variables Gender Race SES Grade/Age School Size LEP Teacher* familiarity Teacher attitudes and beliefs Professional development Instruction in content areas Access to AAC Parent involvement *Includes administrators

Transferring the Framework General Education Special Education A history of assessment Adequate psychometric methods to construct and evaluate assessments Academic instruction is the foundation of education In the absence of disability, opportunity to learn and do well on the assessment Assessment is relatively new Limited psychometric methods to construct and evaluate assessments Academic instruction is special education reform Limited opportunity to learn

Selected Results Descriptive Data and Frequency Counts

Validity Evidence 1.a Teachers are familiar with the assessment

I believe I have a sufficient understanding of the purpose of the AA-AAS.

I believe I have a sufficient understanding of using the results to improve instruction.

Validity Evidence 1.b What are attitudes and beliefs toward the assessment?

I believe the current AA-AAS is an effective way to assess students. *Results are similar for math & science

It is important to include students with disabilities in statewide assessment and accountability systems

It is important to include students with disabilities in statewide assessment and accountability systems (Responses by assessment complexity)

Unintended (negative) consequences From a list generated from the pilot study

Unintended (negative) Consequences

Validity Evidence 2 Professional development opportunities are being provided

Rate the usefulness of the professional development activities that you have participated in since June 2007

Validity Evidence 3 Curriculum is aligned with the State’s Alternate Academic Standards

The AA-AAS has helped me align my classroom instruction with my State’s Alternate Academic Content Standards * Results are similar for math & science

Validity Evidence 4 Students are motivated to learn the material and do well on the assessment

Teaching academic content based on the Alternate Academic Content Standards for students with the most significant cognitive disabilities has had the following effect on student performance in the classroom.

Validity Evidence 5 Student scores are improving as a result of these factors

Some Take-Away Messages This framework can be used by any state to gather evidence of consequential validity. Regardless of the maturity of the Alternate Assessment, studies such as these remind us that development of the AA-AAS is a beginning, not an end.