1. Consequential Validity
A presentation of methods and selected results

2. Background
Policy, Validity Framework, Study Design

3. Peer Review Guidance - Critical Element 4.1 (p.35)
“Has the State ascertained that the decisions based on the results of its assessments are consistent with the purposes for which the assessments were designed?”
“Has the State ascertained whether the assessment produces intended and unintended consequences?”

4. An Argument-based Validity Framework
State the intended consequences
Define the evidence that supports each consequence
Collect data to ascertain the intended consequences
Collect data to ascertain the unintended consequences

5. Original Validity Arguments (Lane & Stone, 2004)
Teacher and administrator motivation and effort will be evident.
Professional development support will be provided.
Instruction and curriculum will be adapted.
Student motivation & effort will be evident.
Performance will improve and be related to the changes above.

6. Additional Validity Arguments NCRRC & 3 State Consortium
Access to augmentative and alternative communication will be improved.
Parental involvement in student’s academic career will be increased.

7. Collecting Data Teacher Surveys Classroom Observation
Administrator Surveys

8. Survey Data Analysis Procedures
Design
Arguments
Stakeholders
Sample
Stratify
Random Cluster
Collect Data
Pilot
Version 1
Version 2

9. Survey Data Analysis Procedures
Prepare Data
CFA
Rasch Scaling
Analyze
Descriptive
HLM
Discriminant Function
Conclude?
Evidence of Consequential Validity

10. Classroom Observation Procedures
Identify Elements
Teaching
IEP Development
Student Profile
Observe
Modified NAAC Observation Tool
Sample in a sample
Evaluate
Is there evidence of consequential validity?

11. Examining Changes Evidential Variables Contextual Variables
Student Performance
(Slope of Change)
Contextual Variables
(Relatively Stable)
Evidential Variables

12. Baseline Data Contextual Variables Evidential Variables Gender Race
SES
Grade/Age
School Size
LEP
Teacher* familiarity
Teacher attitudes and beliefs
Professional development
Instruction in content areas
Access to AAC
Parent involvement
*Includes administrators

13. Transferring the Framework
General Education
Special Education
A history of assessment
Adequate psychometric methods to construct and evaluate assessments
Academic instruction is the foundation of education
In the absence of disability, opportunity to learn and do well on the assessment
Assessment is relatively new
Limited psychometric methods to construct and evaluate assessments
Academic instruction is special education reform
Limited opportunity to learn

14. Selected Results
Descriptive Data and Frequency Counts

15. Validity Evidence 1.a
Teachers are familiar with the assessment

16. I believe I have a sufficient understanding of the purpose of the AA-AAS.

17. I believe I have a sufficient understanding of using the results to improve instruction.

18. Validity Evidence 1.b
What are attitudes and beliefs toward the assessment?

19. I believe the current AA-AAS is an effective way to assess students.
*Results are similar for math & science

20. It is important to include students with disabilities in statewide assessment and accountability systems

21. It is important to include students with disabilities in statewide assessment and accountability systems (Responses by assessment complexity)

22. Unintended (negative) consequences
From a list generated from the pilot study

23. Unintended (negative) Consequences

24. Validity Evidence 2
Professional development opportunities are being provided

25. Rate the usefulness of the professional development activities that you have participated in since June 2007

26. Validity Evidence 3
Curriculum is aligned with the State’s Alternate Academic Standards

27. The AA-AAS has helped me align my classroom instruction with my State’s Alternate Academic Content Standards
* Results are similar for math & science

28. Validity Evidence 4
Students are motivated to learn the material and do well on the assessment

29. Teaching academic content based on the Alternate Academic Content Standards for students with the most significant cognitive disabilities has had the following effect on student performance in the classroom.

30. Validity Evidence 5
Student scores are improving as a result of these factors

31. Some Take-Away Messages
This framework can be used by any state to gather evidence of consequential validity.
Regardless of the maturity of the Alternate Assessment, studies such as these remind us that development of the AA-AAS is a beginning, not an end.