1. Test Blueprints for Adaptive Assessments
Tony Alpert
Oregon Department of Education

2. Purposes of Blueprints
Communicates with stakeholders including test takers regarding the design and content of the assessment
Serves as a source of evidence for demonstrating compliance with federal and state laws and regulations
Serves as method of codifying more detailed agreements regarding the design and content of the assessment for staff and contractors

3. Sources of Information Regarding Requirements for Blueprints
The joint Standards for Educational and Psychological Testing (AERA, APA, NCME)
Operational Best Practices for Statewide Large-Scale Assessment Programs (CCSSO and ATP)
Existing Blueprints of Large Scale Assessment Systems
Peer Review Guidance (USED)

4. Examples of Blueprint Content
Joint Standards
3.2 Describe the purpose of the test and the domain
3.3 Specifications should define the content of the test, proposed number of items, item formats, desired psychometric properties of the items, and the item and section arrangement, time for testing,
Also describes directions to test takers, test administration and scoring procedures and other relevant information
3.4 Procedure to interpret scores
CCSSO/ATP
2.1 Test Specifications for each grade and content area
2.1.2 In addition to 3.3 Should also include use of supporting materials like calculators

5. Examples of Blueprint Content (cont)
Standard 3.5 relevant experts external to the testing program should review the test specifications
CCSSO/ATP clarifies that these specifications should precede item development

6. CAT and Blueprints
CAT designed to be individualized, blueprints are designed to generalize
Assessment Systems have dealt with this problem when using multiple forms

7. Issues With Clear Solutions
Purpose of the test, item types, psychometric approach to scoring, format
Test Length, Approximate Time Required
Students that are eligible
Achievement level descriptors

8. Issues Not Unique to CAT
Content standards to which the test is aligned
State assessments sample content from a larger domain. Each student does not demonstrate proficiency on each required element of content at every level of complexity
Some content may not be assessed well on a summative assessment
Achievement level descriptors
Represent a description of the knowledge and skills students likely have achieved not necessarily whether they have demonstrated knowledge about each element of content (i.e. 100% correct isn’t required for any test)
Validity is tied to educator review and on the standard setting process

9. Issues Unique to CAT Allocation of Content
CATs access item pools and are caged by constraints
Constraints can take many forms (e.g. Content, Cognitive Complexity, Item types, etc)
Interaction between item pool and constraints
Percent Correct Isn’t Helpful
Each student should get about 50% of the items correct
P-values for items don’t work either

12. Issues Not Unique To CAT – But Complicated For CAT
Evaluation of Test Specifications
Prior to administration of tests use simulations
During and after administration, use real data
Determine if discrepancies are due to the algorithm, constraints, item pool, or interaction of the three
Evaluation of Item Pool
Sufficient Size
Optimally in proportion to depth and breadth as expected in each test

13. Issues Related To CAT Policy
Termination Criteria
Impacts test length, Allocation of content, testing time required, item pool requirements
Documentation in specifications should likely be more complex to address these more complex approaches
Use of Interactive Simulations, Audio and Speech
Paper may not be sufficient medium for test specifications
Need to consider test specifications that only can be delivered online