1. Links for Academic Learning: Planning An Alignment Study
Click the sound icon(s) on each slide.
Part 3 addresses planning the alignment study and decisions that need to be made either with a vendor or within the state.
September 25, 2007

2. Issues to Consider in Planning
Guiding Principles: Linn
Validity
Fairness
Credibility
Utility
Gong
Development of content standards
Purpose of assessment
Prioritization of content
Impartial reviews for alignment are beneficial to ensure objectivity. While a state is welcome to be included in the alignment process, it is important that the partnership between external vendor and state remains unbiased pertaining to alignment outcomes. It is imperative, however, that the state’s values and rationale for decisions are captured within the study. For example, if a state chooses to prioritize one strand within math over another, the test blueprint and the thinking behind this prioritization should be reflected accurately in the study. It is critical that a state takes the time during the development of the extended standards and alternate assessment to convey their values and that these values are shared by all stakeholders involved. The researchers listed on this slide have multiple resources regarding the need for the articulation of guiding principles regarding the development of the assessment system.
September 25, 2007

3. Describing the Principles and Characteristics of Your AA-AAS
What are the policy elements?
What does the state want to accomplish by conducting the alignment study?
Who was included in the development of your AA-AAS?
What are the unique issues of your AA-AAS?
First, it is critical that the vendor or external reviewer understands the purpose of the AA-AAS system for the state. For example, if accountability is indicated as a part of the purpose, making inferences about student learning from the AA system will be overt within the policy written regarding outcomes expected from the assessment. Next, it is important to identify the reason for the alignment study. Does the state have a new AA-AAS? Is information needed about instruction in relation to the extended and/or grade level content standards? Third, who has and should be involved throughout this process? For example, who should be included in the development and revision of extended standards? Who should be involved in the standard setting process? Finally, it is important that the state identifies the unique approaches or issues to the development of their AA-AAS system. The alignment methodology used to evaluate the AA-AAS system should be flexible enough so that the format of the assessment should not limit the ability to report the alignment issues within the assessment system.
September 25, 2007

4. Specific Questions to Ask Vendor
Who will be the reviewers?
What educational components will be included in the alignment study?
How will the results be reported?
What criteria will be used in evaluating the degree of alignment?
Specific questions regarding the study should also be answered prior to the beginning of the study. First, who and how many reviewers will be used? How will reliability be measured? It is important that at least two or more reviewers are used. Content and special education experts should be used depending upon the content being evaluated. Next, it is important to discuss which pieces within the assessment system the external evaluator is willing to include in the alignment study. Unique to AA-AAS is the opportunity for states to include extended or alternate content standards. Also, as discussed earlier, as academic content may be new for special educators to teach, it may also be important to capture the instructional link. The purpose of the alignment study should guide the expectations of which components are included in the alignment study. It is also important to ask how student evidence or work samples will be included in the study. Good information about instruction and its link to the content standards can be found in the evidence provided. For example, a value for this population is the use of age appropriate materials. This can be measured within the student evidence of the AA-AAS. A third question is what data will be reported and in what manner? It is recommended that data should provide information beyond USDOE peer review criteria to help in future development and revision of the assessment system, and in professional development initiatives. A fourth critical question is what criteria will be used to evaluate alignment. The vendor should be clear on the expected values or judgments used. Finally, will the vendor include potential reasons for any lack of alignment. This information would be helpful to know why items did not align.
September 25, 2007

5. Getting the Most From the Alignment Study
Involve stakeholders in planning and responding to the alignment study
Consider the cost-benefit of including instructional alignment in the study
Use the alignment study results to look at inferences about student learning and potential evidence of validity of AA-AAS outcomes
States should look to get something more from the alignment study besides the USDOE peer review requirements. The quality enhancement of educational services for students participating in alternate assessments should help maximize the money spent on the alignment study. First, the alignment study and its outcomes can help build capacity for educators related to the teaching of academic content standards and its relationship with the teaching of functional skills. Best practice for the population of students who participate in AA-AAS such as self determination skills, generalization of content mastered, and assistive technology usage can be fostered within the alternate assessment. Measuring instructional alignment can provide the state with important data which can be used to create targeted professional development opportunities for educators in needed areas. Finally, gathering information regarding what actually counts in the student score is crucial. As states are charged to use AA-AAS to make inferences about student learning, states need information about what is being used to develop a student score. It is important to understand the difference between student performance and quality program information.
September 25, 2007

6. Contact Information
Diane Browder:
Claudia Flowers:
Shawnee Wakeman:
We would like to thank you for participating in today’s phone call. Please contact us at UNC Charlotte with any questions regarding the LAL and visit the NAAC or UNCC website for additional resources and information.
September 25, 2007