2. Race to the Top Assessment Program: Public Hearings Race to the Top Assessment Program: Public Hearings Some Thoughts on Grades 3-8 and High School Assessments December 1, 2009 Denver, Colorado Presenter: Lloyd Bond, The Carnegie Foundation

3. Issues about which there is consensus Intended uses Intended uses Content, Formats, & Test Administration Content, Formats, & Test Administration Validation Validation Reporting Reporting

4. Consensus Issues Intended uses Monitoring student progress Monitoring student progress Teacher, principal, and school accountability Teacher, principal, and school accountability College and work readiness (?) College and work readiness (?) Content, Formats, & Test Administration Multiple measures of student success Multiple measures of student success Paper & pencil, computer assisted, MC, short answer, extended response, and portfolio Paper & pencil, computer assisted, MC, short answer, extended response, and portfolio Both mid-course and end-of-course assessments Both mid-course and end-of-course assessments Multiple opportunities and multiple ways to demonstrate mastery Multiple opportunities and multiple ways to demonstrate mastery (See “My Child Doesn’t Test Well” Carnegie Perspectives Web Site) (See “My Child Doesn’t Test Well” Carnegie Perspectives Web Site) Instructionally useful information to teachers and principals Instructionally useful information to teachers and principals

5. Consensus Issues (cont’d) Validation Appropriate evidence claims of content fidelity and college/work readiness (?) Appropriate evidence claims of content fidelity and college/work readiness (?)Reporting Reporting of assessment information (including assessment architecture, standards, and results) to the various non-technical stakeholders should be as straightforward as possible, understandable to literate adults Reporting of assessment information (including assessment architecture, standards, and results) to the various non-technical stakeholders should be as straightforward as possible, understandable to literate adults

6. Thoughts on Unresolved and/or Contentious Issues Broad & Shallow vs. Narrow & Deep Broad & Shallow vs. Narrow & Deep Consequential Validity - “Teaching to the Test” Consequential Validity - “Teaching to the Test” The concepts of college and work “readiness” The concepts of college and work “readiness” Familiar vs.aspirational assessment Familiar vs.aspirational assessment

7. Broad & Shallow vs... Narrow & Deep The delicate balance between (1) learning a lot of things marginally and (2) learning a modest number of important things well The delicate balance between (1) learning a lot of things marginally and (2) learning a modest number of important things well One of the distinguishing curricular features of countries that consistently do well on TIMMS and other assessments is that they error on the side of “(relatively) narrow and deep” One of the distinguishing curricular features of countries that consistently do well on TIMMS and other assessments is that they error on the side of “(relatively) narrow and deep”

8. Thoughts on the “what” of assessment The Grade 3-8 through High School assessments should move increasingly and continuously from: 1) assessment of declarative knowledge to assessment of procedural and conceptual knowledge 2) assessment of procedures and skills to assessment of problem solving and critical thinking

9. Thoughts on Validity Gathering evidence of content & predictive validity should be reasonably straightforward Gathering evidence of content & predictive validity should be reasonably straightforward Gathering evidence regarding consequential validity can be problematic Gathering evidence regarding consequential validity can be problematic

10. Consequential Validity of the Assessments Testing what is taught vs. teaching what is tested Testing what is taught vs. teaching what is tested Using the RTT assessments for accountability purposes guarantees that attempts to “teach to the test” will occur Using the RTT assessments for accountability purposes guarantees that attempts to “teach to the test” will occur We must therefore make the assessments worth teaching to We must therefore make the assessments worth teaching to (See “Teaching to the Test” Carnegie Perspectives Web Site) (See “Teaching to the Test” Carnegie Perspectives Web Site)

11. College & Work Readiness College and work “readiness” are as yet ill- defined and illusive concepts College and work “readiness” are as yet ill- defined and illusive concepts “Ready” for majoring in physical education at Podunk U or ready for majoring in electrical engineering at Cat Tech “Ready” for majoring in physical education at Podunk U or ready for majoring in electrical engineering at Cat Tech Ready for a cashier at McDonald’s or ready for office manager trainee at Merrill Lynch Ready for a cashier at McDonald’s or ready for office manager trainee at Merrill Lynch It is doubtful whether a single test can serve this multiple purposes It is doubtful whether a single test can serve this multiple purposes

12. Measuring important skills and abilities “We do not yet know how to measure all of the important things. Ergo, what we CAN measure becomes all important” “We do not yet know how to measure all of the important things. Ergo, what we CAN measure becomes all important”

13. Aspirational Assessments In addition to the usual and familiar, the RTT assessment program should encourage and support consortia that also move the profession forward In addition to the usual and familiar, the RTT assessment program should encourage and support consortia that also move the profession forward

14. Toward the Assessment of “Understanding” Students can “know” things and know how to do things without genuine understanding Students can “know” things and know how to do things without genuine understanding “Study to remember, and you will forget; study to understand, and you will remember.” ~Anonymous “Study to remember, and you will forget; study to understand, and you will remember.” ~Anonymous Teach for regurgitation, and students will forget; teach for understanding, and they will remember. Teach for regurgitation, and students will forget; teach for understanding, and they will remember. “The Assessment of Understanding” Carnegie Perspectives” Web Site “The Assessment of Understanding” Carnegie Perspectives” Web Site

15. Final Thoughts Beware of the assessment tail wagging the content dog Beware of the assessment tail wagging the content dog Don’t reinvent the wheel. Encourage consortia to take advantage of what has already been done and what we already know (e.g., College Board Accuplacer) Don’t reinvent the wheel. Encourage consortia to take advantage of what has already been done and what we already know (e.g., College Board Accuplacer) Encourage involvement of higher education faculty and industry personnel in the high school assessment Encourage involvement of higher education faculty and industry personnel in the high school assessment