1. Dramatically Better Assessment Systems: Advice for RTTT “Common Assessment” RFP Brian Gong Center for Assessment Presentation for the Input Meetings Sponsored by the U.S. Department of Education for the “Common Assessment” RFP, “Race to the Top” funding November 17, 2009 Atlanta, GA

2. 2Gong – USED Common Assessment RFP Input Mtg – 11/17/09 My Main Point The future of assessment in the United States will be shaped by what gets funded in this “Common Assessment” RFP. The future of assessment in the United States will be shaped by what gets funded in this “Common Assessment” RFP. USED should shape the RFP and fund it with a longer-term view of having in place dramatically better assessment systems in ten years. USED should shape the RFP and fund it with a longer-term view of having in place dramatically better assessment systems in ten years. When USED has to compromise, choose longer- term investments over short-term gains When USED has to compromise, choose longer- term investments over short-term gains Say very clearly what you want in the RFP Say very clearly what you want in the RFP Help foster good responses to the RFP Help foster good responses to the RFP

3. 3Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Personal recommendations Hedge bets by funding multiple ways to do multi-state common assessment, especially high school Hedge bets by funding multiple ways to do multi-state common assessment, especially high school Invest in six “game changers” that could make assessment dramatically better within a decade, but should not be framed as being operationally implemented on the short time schedule (“2012”) Invest in six “game changers” that could make assessment dramatically better within a decade, but should not be framed as being operationally implemented on the short time schedule (“2012”) Help foster good responses to the RFP and after Help foster good responses to the RFP and after

4. 4Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Short-term and Longer-term Investments Common Assessment RFP should fund For implementation by 2012, what we already know how to do in large-scale assessment but For implementation by 2012, what we already know how to do in large-scale assessment but With new set of content standards With new set of content standards With groups of multiple states (difficult to do) With groups of multiple states (difficult to do) For development through 2015, what we do not know how to do well at scale, but which has potential to lead to dramatically better assessment systems For development through 2015, what we do not know how to do well at scale, but which has potential to lead to dramatically better assessment systems

5. 5Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Implementing a new multi-state summative assessment takes years 2009 2010 2011 2012 2013 2014 2015 50 state systems, NAEP, TIMMS, PISA, PERLS, many LEA systems, NRTs, ACT/SAT, college’s tests, etc. Award RFP(s) (9/2009) Test Specifi- cations; Develop Items; Use specs, reports, equating design, administration agreements, etc. (2009-10) Pilot Test Items, promulgate high stakes policies, etc. (2010-11) First operational administra- tion & reporting, etc. (2011-12) Second operational administra- tion; first report using growth, etc. (2012-13) Fourth operational administra- tion; first graduating high school class, etc. (2014-15) Fast Implementation of RFP: 2012 (e.g., multi-state assessments with common content standards, “Peer Review” quality of things we know how to do) And aligning curriculum, instruction, accountability, and supports takes longer.

6. 6Gong – USED Common Assessment RFP Input Mtg – 11/17/09 RFP: Specify, Specify, Specify USED should specify its purpose, theory of action, and how the assessment results will be used so responders know the big picture USED should specify its purpose, theory of action, and how the assessment results will be used so responders know the big picture Specify what is wanted as an deliverable and the set parameters for responders’ creative proposals (e.g., time schedule) Specify what is wanted as an deliverable and the set parameters for responders’ creative proposals (e.g., time schedule) Specify the means an outcome should be done if USED really wants a specific means Specify the means an outcome should be done if USED really wants a specific means

7. 7Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Some Model Systems for 2012 Cross-state comparisons Cross-state comparisons Standards-based interpretation Standards-based interpretation Inform better instruction Inform better instruction Rapid turn-around Rapid turn-around Measure growth Measure growth Measure student performance for teacher/administrator evaluation Measure student performance for teacher/administrator evaluation

8. 8Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Cross-state Comparisons (2012) Purpose, TOA, Use: Hold students, schools, LEAs, and states accountable to a common performance standard by triggering sanctions Purpose, TOA, Use: Hold students, schools, LEAs, and states accountable to a common performance standard by triggering sanctions Outcome: Statistically robust reports of performance on common metric with no “wiggle room” – stronger than current NAEP mapping studies Outcome: Statistically robust reports of performance on common metric with no “wiggle room” – stronger than current NAEP mapping studies Means: Same content standards, same test specifications, same performance standards, single assessment across states, same administration procedures, strong equating across years Means: Same content standards, same test specifications, same performance standards, single assessment across states, same administration procedures, strong equating across years

9. 9Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Standards-based Interpretation (2012) Purpose, TOA, Use: Promote equity through holding students and schools to common opportunity-to-learn (content standards) and minimal performance standards Purpose, TOA, Use: Promote equity through holding students and schools to common opportunity-to-learn (content standards) and minimal performance standards Outcome: Valid reports of performance related to the designated standards Outcome: Valid reports of performance related to the designated standards Means: Aligned, grade-level only (?), matrix- sampled (?), high school (?), SWD (?), ELL (?) Means: Aligned, grade-level only (?), matrix- sampled (?), high school (?), SWD (?), ELL (?)

10. 10Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Inform Better Instruction (2012) Purpose, TOA, Use: Assess more complex and applied learning (monitor); model and encourage instruction (drive) Purpose, TOA, Use: Assess more complex and applied learning (monitor); model and encourage instruction (drive) Outcome: Incrementally better, more valid and reliable measurement of higher-order, complex student performances (?); more widespread “good” instruction (?) Outcome: Incrementally better, more valid and reliable measurement of higher-order, complex student performances (?); more widespread “good” instruction (?) Means: Curriculum-embedded assessments (e.g., standardized units, portfolios, graduation projects) (?); curricula with (local) matched assessments (?) Means: Curriculum-embedded assessments (e.g., standardized units, portfolios, graduation projects) (?); curricula with (local) matched assessments (?)

11. 11Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Rapid Turn-around (2012) Purpose, TOA, Use: Promote improvement through rapid feedback to inform actions Purpose, TOA, Use: Promote improvement through rapid feedback to inform actions Outcome: Reports of performance useful to decisions and actions, in appropriate timeframe (distinguish actions that are multi-year or annual monitoring from annual rich content analysis from shorter-term uses, down to course grades and student instructional feedback) Outcome: Reports of performance useful to decisions and actions, in appropriate timeframe (distinguish actions that are multi-year or annual monitoring from annual rich content analysis from shorter-term uses, down to course grades and student instructional feedback) Means: Trade-off speed for quality, cost: greater reliance on multiple-choice/machine-scored; trade- off centralized standardization for complex performances, local scoring; ignore administration variations (e.g., missing students) Means: Trade-off speed for quality, cost: greater reliance on multiple-choice/machine-scored; trade- off centralized standardization for complex performances, local scoring; ignore administration variations (e.g., missing students)

12. 12Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Measure Growth (2012) Purpose, TOA, Use: Accountability, program improvement, teacher accountability? Purpose, TOA, Use: Accountability, program improvement, teacher accountability? Outcome: Report of student progress over time related to what is/could be/should be: grade-level standards (?), own starting point (?), other students (?), program supports (?), “teacher’s contribution” (?); how to use in accountability (?) Outcome: Report of student progress over time related to what is/could be/should be: grade-level standards (?), own starting point (?), other students (?), program supports (?), “teacher’s contribution” (?); how to use in accountability (?) Means: Out-of-level testing (?), adaptive testing (?), vertical [moderated] scales (?), use math to predict reading for greater reliability (?), pre- post- measures within year (?) Means: Out-of-level testing (?), adaptive testing (?), vertical [moderated] scales (?), use math to predict reading for greater reliability (?), pre- post- measures within year (?)

13. 13Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Teacher/administrator evaluation (2012) Purpose, TOA, Use: Improve teacher quality by providing feedback (?); use in accountability or other high-stakes decisions (?) Purpose, TOA, Use: Improve teacher quality by providing feedback (?); use in accountability or other high-stakes decisions (?) Outcome: Changes in student performance associated with (attributable to ?) specific teachers, administrators, programs Outcome: Changes in student performance associated with (attributable to ?) specific teachers, administrators, programs Means: many statistical approaches (check assumptions, limitations) (?); combine with other information (?) Means: many statistical approaches (check assumptions, limitations) (?); combine with other information (?)

14. 14Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Personal recommendations Hedge bets on multiple ways to do multi- state common assessment, especially high school Hedge bets on multiple ways to do multi- state common assessment, especially high school Invest in six “game changers” that could make assessment dramatically better within a decade, but should not be framed as being operationally implemented on the short time schedule (“2012”) Invest in six “game changers” that could make assessment dramatically better within a decade, but should not be framed as being operationally implemented on the short time schedule (“2012”) Help foster good responses to the RFP and after Help foster good responses to the RFP and after

15. 15Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Invest in “Game Changers” - 1 Develop technology that provides evidence of more complex knowledge and skills (i.e., more valid) Develop technology that provides evidence of more complex knowledge and skills (i.e., more valid) E.g., interactive simulations, non-academic knowledge and skills E.g., interactive simulations, non-academic knowledge and skills Only use technology with an evidence- centered design approach to maintain construct relevance, most students Only use technology with an evidence- centered design approach to maintain construct relevance, most students

16. 16Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Invest in “Game Changers” - 2 Develop technology for validity Develop complex performance assessment Develop complex performance assessment Specify extended learning and content, real application contexts, student choice Specify extended learning and content, real application contexts, student choice Develop credible (local) administration and scoring Develop credible (local) administration and scoring Include all students (and teachers) Include all students (and teachers) Develop means of certifying validity and reliability, and of combining with other evidence Develop means of certifying validity and reliability, and of combining with other evidence

17. 17Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Invest in “Game Changers” - 3 Develop technology for validity Develop complex performance assessment Develop curricula that specify “what” and “how” of learning, and associated local assessment systems Develop curricula that specify “what” and “how” of learning, and associated local assessment systems Interim and formative assessments are needed to inform learning directly Interim and formative assessments are needed to inform learning directly Real assessment problem is informing “What should be done next?” – cannot solve without curriculum and teacher/administrator expertise Real assessment problem is informing “What should be done next?” – cannot solve without curriculum and teacher/administrator expertise

18. 18Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Invest in “Game Changers” - 4 Develop technology for validity Develop complex performance assessment Develop curricula, local assessment systems Develop new measurement models and technical criteria for assessments of complex knowledge and skills Develop new measurement models and technical criteria for assessments of complex knowledge and skills We know current models’ assumptions and limitations; do not impose on innovations! (Example: reliability vs. validity of complex performances; cognitive vs. unidimensional models) We know current models’ assumptions and limitations; do not impose on innovations! (Example: reliability vs. validity of complex performances; cognitive vs. unidimensional models)

19. 19Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Invest in “Game Changers” - 5 Develop technology for validity Develop complex performance assessment Develop curricula, comprehensive assessment systems Develop new measurement models and technical criteria Develop better accountability models and support better use of assessment results for program improvement Develop better accountability models and support better use of assessment results for program improvement Assessments, assessment use, and instruction are being distorted by our current accountability model Assessments, assessment use, and instruction are being distorted by our current accountability model

20. 20Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Invest in “Game Changers” - 6 Develop technology for validity Develop complex performance assessment Develop curricula, local assessment systems Develop new measurement models and technical criteria Develop better models of accountability and program improvement Develop model specifications for a coherent comprehensive assessment system that incorporates above five Develop model specifications for a coherent comprehensive assessment system that incorporates above five e.g., NAEP, state, e.g., NAEP, state,

21. 21Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Invest in “Game Changers” - 7 Technology for validity Complex performance assessment Curricula & comprehensive assessment systems New measurement models and technical criteria New measurement models and technical criteria Better accountability models and support for program improvement Better accountability models and support for program improvement

22. 22Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Personal Recommendation - 2 Invest in five assessment “game changers” Invest in five assessment “game changers” Hedge bets on multiple ways to do multi- state “2012” common assessment, especially high school Hedge bets on multiple ways to do multi- state “2012” common assessment, especially high school Good current models: all MC, mixed MC-CR, computer-based, end-of-course, survey, etc. Good current models: all MC, mixed MC-CR, computer-based, end-of-course, survey, etc. Interwoven with state policies (e.g., high school exit requirements) Interwoven with state policies (e.g., high school exit requirements) Help foster good responses to the RFP and after Help foster good responses to the RFP and after

23. 23Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Hedge bets on 2012 assessment End of course AND Grade 11 survey End of course AND Grade 11 survey Computer-based AND paper & pencil Computer-based AND paper & pencil All multiple choice AND modest short CR AND larger amount and more extensive CR All multiple choice AND modest short CR AND larger amount and more extensive CR Fund multiple “common content standards” Fund multiple “common content standards” To find out costs and benefits of multi-state common assessments To find out costs and benefits of multi-state common assessments Because no one set of content standards is clearly superior Because no one set of content standards is clearly superior Because no one approach is clearly superior Because no one approach is clearly superior Because reporting on a common score metric is less important Because reporting on a common score metric is less important

24. 24Gong – USED Common Assessment RFP Input Mtg – 11/17/09 RFP Portfolio of Awards Multiple (around 8) strong models that represent advances that can be implemented strongly by 2012 and that help get to the longer-term goal Multiple (around 8) strong models that represent advances that can be implemented strongly by 2012 and that help get to the longer-term goal Consider strategy: Do not fund strong models that will be adopted even if not funded Consider strategy: Do not fund strong models that will be adopted even if not funded Multiple (perhaps 12) strong “game changer” awards Multiple (perhaps 12) strong “game changer” awards

25. 25Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Personal Recommendation - 3 Invest in four assessment “game changers” Invest in four assessment “game changers” Hedge bets on multiple ways to do multi-state common assessment, especially high school Hedge bets on multiple ways to do multi-state common assessment, especially high school Help foster good responses to the RFP and after Help foster good responses to the RFP and after If USED wants certain outcomes of states working together, then promote leadership to make that happen among states, NGOs, test vendors, etc. If USED wants certain outcomes of states working together, then promote leadership to make that happen among states, NGOs, test vendors, etc.

26. 26Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Fostering Strong RFP Responses Provide clear RFP specs and different awards for “2012 implementation” and “game changers” Provide clear RFP specs and different awards for “2012 implementation” and “game changers” If USED wants states to have vendor partners in their RFP responses, need to indicate that early and facilitate it well (vs. states’ issuing an RFP) If USED wants states to have vendor partners in their RFP responses, need to indicate that early and facilitate it well (vs. states’ issuing an RFP) USED should think about what states who don’t get RTTT common assessment funds will do USED should think about what states who don’t get RTTT common assessment funds will do USED should think how what it funds will be adopted after RTTT and how that will shape what is available in the future USED should think how what it funds will be adopted after RTTT and how that will shape what is available in the future

27. 27Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Envision Intended & Unintended Consequences What if in 2012 there were five widely used assessments, all aligned to the same common content standards What if in 2012 there were five widely used assessments, all aligned to the same common content standards Four were commercially available from current test publishers (like the Achieve/Pearson Algebra 2 end-of-course exam) Four were commercially available from current test publishers (like the Achieve/Pearson Algebra 2 end-of-course exam) One was available by joining a consortium (like the WIDA ELP exams) One was available by joining a consortium (like the WIDA ELP exams) States were purchasing elementary math from one vendor and high school English from another vendor States were purchasing elementary math from one vendor and high school English from another vendor What if there were only one assessment being used? What if there were 46? What if there were only one assessment being used? What if there were 46?

28. 28Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Envision Intended & Unintended Consequences – 2 What if in 2012 each commercially available assessment came in five versions: What if in 2012 each commercially available assessment came in five versions: An all multiple-choice, computer-administered short form that took 20 minutes and cost $3/per student An all multiple-choice, computer-administered short form that took 20 minutes and cost $3/per student An all multiple-choice, computer or paper & pencil form that took 50 minutes and cost $7/per student An all multiple-choice, computer or paper & pencil form that took 50 minutes and cost $7/per student A computer or p & p version that took 120 minutes, had 40 multiple choice, 8 short constructed response, and 4 extended constructed response items and cost $15/per student A computer or p & p version that took 120 minutes, had 40 multiple choice, 8 short constructed response, and 4 extended constructed response items and cost $15/per student A computer of p & p version that took 150 minutes, had 40 multiple choice, 4 extended constructed response, and 2 long constructed response items and cost $60/per student A computer of p & p version that took 150 minutes, had 40 multiple choice, 4 extended constructed response, and 2 long constructed response items and cost $60/per student A version that included a standardized test like option 3 and had a curriculum-embedded project and other performance evidence that was centrally audited and cost $200/per student A version that included a standardized test like option 3 and had a curriculum-embedded project and other performance evidence that was centrally audited and cost $200/per student

29. 29Gong – USED Common Assessment RFP Input Mtg – 11/17/09 Center for Assessment www.nciea.org Brian Gong bgong@nciea.org For more information: