Evaluating Assessment for the NRS Evaluating Assessments for the NRS American Institutes for Research February 2005

Evaluating Assessment for the NRS 2 Student Assessment Process Adapted from D.M. Kenyon and C. Van Duzer, Valid, Reliable, and Appropriate Assessments for Adult English Language Learners, November Administrator Underlying competencies Performance Student Task Task qualities and conditions Underlying K/S/A Scale/CriteriaRater Underlying competencies Scale/criteria [NRS levels] and scoring conditions Score/Measure

Evaluating Assessment for the NRS 3 Need for Standardized Assessment Review Process Promotes compatibility of assessment information across states Standard process and criteria for determining use to measure educational gain Without it, inconsistent approval of assessments across states

Evaluating Assessment for the NRS 4 Assessment Review Process Required review procedure for states and at federal level All assessments to be reviewed Plan to formally regulate requirements, to take effect in 2006 Guided by Higher Education Ability to Benefit approach To be published in new NRS Implementation Guidelines (draft in handouts)

Evaluating Assessment for the NRS 5 Assessment Review Process (2) Step 1: Application—State or test publisher requests review and submits: Source materials (e.g., manuals, report) Reports with psychometric evidence (e.g., validity, reliability) Step 2: Rating on Psychometric Criteria Using NRS criteria on test requirements

Evaluating Assessment for the NRS 6 Assessment Review Process (3) Step 3: Formal Review Panel of 2 or 3 experts independently reviews assessment and makes recommendation Step 4: Documentation Formal report and recommendation to OVAE

Evaluating Assessment for the NRS 7 Questions, Discussion Questions?

Evaluating Assessment for the NRS 8 NRS Assessment Database Access to an online assessment database through Database will contain assessments that are appropriate for use in adult education programs Assessments will be approved for inclusion in the database by ED Additional assessments may be added after review and approval by ED

Evaluating Assessment for the NRS 9 Content Database will include 4 main sections: List of tests included in the database List of features to be included Results pages (for search and comparison functions) Definitions of terms

Evaluating Assessment for the NRS 10 Assessments by Name/Title

Evaluating Assessment for the NRS 11 Features to Be Included Title of Test (Acronym) Narrative Summary—short statement Test Use and Purpose Intended Population Purpose (e.g., Placement, Diagnostic, Progress, Achievement/Proficiency)

Evaluating Assessment for the NRS 12 Features to Be Included (cont’d.) Test Specifications and Features Test Content (skills measured) Range of Skill Levels Items and Subtests Alternate Test Forms Required/Recommended Training for Test Administrators Test Administration Procedures Scoring and Reporting Time Needed for Assessment

Evaluating Assessment for the NRS 13 Features to Be Included (cont’d.) Ordering Information/Availability Publisher/Company/Source for Obtaining the Test Versions, Publication Date(s) Product Web Site Additional Evaluator Comments

Evaluating Assessment for the NRS 14 Functions Two search methods: Search for an assessment by name Search for an assessment according to specific features

Evaluating Assessment for the NRS 15 Search Results by Test Name/Title

Evaluating Assessment for the NRS 16 Comparison Function Compare up to three different assessments across features in a side-by-side table format

Evaluating Assessment for the NRS 17 Comparison Chart of Two Tests

Evaluating Assessment for the NRS 18 NRS Assessment Database Questions? Comments/Feedback