Download presentation
Presentation is loading. Please wait.
1
Gary W. Phillips Vice President & Institute Fellow American Institutes for Research Next Generation Achievement Standard Setting Symposium CCSSO NCSA New Orleans LA June 25, 2014 Using International Benchmarking to Help Establish Internationally Competitive Performance Standards
2
National and International Benchmarking 2 NAEP TIMSS PIRLS
3
3 National Benchmarking using NAEP ( simulated data ) Math, Grade 8NAEP-equivalents to State Score Page inEstimatedNAEP-StandardProportion BookmarkStateProportionEquivalentError ofin U.S. OrderedScaledAboveScaledNAEP-Above Item BookletScoresCut-ScoreScoreEquivalentNAEP-Equiv 418000.01 Ad = 333 350 10.9 0.03 407880.02341 8.0 0.05 397870.02340 7.8 0.05 387850.02339 7.4 0.06 377800.03335 6.7 0.07 367750.03 Proficient = 299 331 6.0 0.09 357700.04327 5.4 0.10 347660.05324 5.0 0.12 337600.07319 4.6 0.15 327550.09315 4.2 0.18 317480.12309 3.8 0.22 307450.13307 3.7 0.24 297400.16303 3.5 0.27 287350.19299 3.3 0.31 277300.22 Basic = 262 295 3.1 0.35 267260.25292 3.0 0.38 257200.30287 2.9 0.43 247150.34283 2.8 0.48 237120.37281 2.8 0.50 227050.43276 2.7 0.56 217000.48272 2.7 0.61 206990.49271 2.7 0.62 196900.58264 2.7 0.69 186850.62 Below Basic 260 2.8 0.73 176800.67256 2.8 0.76 166750.71252 2.9 0.79 156730.72250 3.0 0.81 146650.78244 3.2 0.85 136600.82240 3.3 0.87 126560.84237 3.5 0.89 116500.87232 3.7 0.91 106450.90228 4.0 0.93 96400.92224 4.3 0.94 86400.92224 4.3 0.94 76300.95216 5.0 0.96 66250.96213 5.5 0.97 56200.97209 6.1 0.98 46200.97209 6.1 0.98 36100.98201 7.6 0.99 26060.99198 8.4 0.99 16000.99193 9.8 0.99
4
4 National and International Benchmarking using NAEP and TIMSS ( simulated data ) Math, Grade 8NAEP-equivalents to State ScoreTIMSS-equivalents to State Score Page inEstimatedNAEP-StandardProportionTIMSS-StandardProportion BookmarkStateProportionEquivalentError ofin U.S.EquivalentError ofin Korea OrderedScaledAboveScaledNAEP-AboveScaledTIMSS-Above Item BookletScoresCut-ScoreScoreEquivalentNAEP-EquivScoreEquivalentTIMSS-Equiv 418000.01 Ad = 333 350 10.9 0.03 Ad=625 655 25 0.26 407880.02341 8.0 0.05635 20 0.34 397870.02340 7.8 0.05633 19 0.35 387850.02339 7.4 0.06630 19 0.36 377800.03335 6.7 0.07 High = 550 622 17 0.39 367750.03 Proficient = 299 331 6.0 0.09613 16 0.43 357700.04327 5.4 0.10605 15 0.47 347660.05324 5.0 0.12598 15 0.49 337600.07319 4.6 0.15588 14 0.54 327550.09315 4.2 0.18580 14 0.57 317480.12309 3.8 0.22568 13 0.62 307450.13307 3.7 0.24563 13 0.64 297400.16303 3.5 0.27555 13 0.68 287350.19299 3.3 0.31 Intermediate = 475 546 13 0.71 277300.22 Basic = 262 295 3.1 0.35538 13 0.74 267260.25292 3.0 0.38531 13 0.76 257200.30287 2.9 0.43521 13 0.80 247150.34283 2.8 0.48513 13 0.82 237120.37281 2.8 0.50508 13 0.83 227050.43276 2.7 0.56496 13 0.86 217000.48272 2.7 0.61488 13 0.88 206990.49271 2.7 0.62486 13 0.89 196900.58264 2.7 0.69 Low = 400 471 13 0.91 186850.62 Below Basic 260 2.8 0.73463 13 0.93 176800.67256 2.8 0.76454 13 0.94 166750.71252 2.9 0.79446 13 0.95 156730.72250 3.0 0.81442 13 0.95 146650.78244 3.2 0.85429 14 0.97 136600.82240 3.3 0.87421 14 0.97 126560.84237 3.5 0.89414 14 0.98 116500.87232 3.7 0.91404 14 0.98 106450.90228 4.0 0.93 Below Low 396 15 0.99 96400.92224 4.3 0.94387 15 0.99 86400.92224 4.3 0.94387 15 0.99 76300.95216 5.0 0.96370 16 0.99 66250.96213 5.5 0.97362 17 0.99 56200.97209 6.1 0.98354 18 1.00 46200.97209 6.1 0.98354 19 1.00 36100.98201 7.6 0.99337 20 1.00 26060.99198 8.4 0.99330 22 1.00 16000.99193 9.8 0.99320 24 1.00
5
5 National and International Benchmarking using NAEP and TIMSS ( simulated data ) Math, Grade 8NAEP-equivalents to State ScoreTIMSS-equivalents to State Score Page inEstimatedNAEP-StandardProportionTIMSS-StandardProportion BookmarkStateProportionEquivalentError ofin U.S.EquivalentError ofin Korea OrderedScaledAboveScaledNAEP-AboveScaledTIMSS-Above Item BookletScoresCut-ScoreScoreEquivalentNAEP-EquivScoreEquivalentTIMSS-Equiv 418000.01 Ad = 333 350 10.9 0.03 Ad=625 655 25 0.26 407880.02341 8.0 0.05635 20 0.34 397870.02340 7.8 0.05633 19 0.35 387850.02339 7.4 0.06630 19 0.36 377800.03335 6.7 0.07 High = 550 622 17 0.39 367750.03 Proficient = 299 331 6.0 0.09613 16 0.43 357700.04327 5.4 0.10605 15 0.47 347660.05324 5.0 0.12598 15 0.49 337600.07319 4.6 0.15588 14 0.54 327550.09315 4.2 0.18580 14 0.57 317480.12309 3.8 0.22568 13 0.62 307450.13307 3.7 0.24563 13 0.64 297400.16303 3.5 0.27555 13 0.68 287350.19299 3.3 0.31 Intermediate = 475 546 13 0.71 277300.22 Basic = 262 295 3.1 0.35538 13 0.74 267260.25292 3.0 0.38531 13 0.76 257200.30287 2.9 0.43521 13 0.80 247150.34283 2.8 0.48513 13 0.82 237120.37281 2.8 0.50508 13 0.83 227050.43276 2.7 0.56496 13 0.86 217000.48272 2.7 0.61488 13 0.88 206990.49271 2.7 0.62486 13 0.89 196900.58264 2.7 0.69 Low = 400 471 13 0.91 186850.62 Below Basic 260 2.8 0.73463 13 0.93 176800.67256 2.8 0.76454 13 0.94 166750.71252 2.9 0.79446 13 0.95 156730.72250 3.0 0.81442 13 0.95 146650.78244 3.2 0.85429 14 0.97 136600.82240 3.3 0.87421 14 0.97 126560.84237 3.5 0.89414 14 0.98 116500.87232 3.7 0.91404 14 0.98 106450.90228 4.0 0.93 Below Low 396 15 0.99 96400.92224 4.3 0.94387 15 0.99 86400.92224 4.3 0.94387 15 0.99 76300.95216 5.0 0.96370 16 0.99 66250.96213 5.5 0.97362 17 0.99 56200.97209 6.1 0.98354 18 1.00 46200.97209 6.1 0.98354 19 1.00 36100.98201 7.6 0.99337 20 1.00 26060.99198 8.4 0.99330 22 1.00 16000.99193 9.8 0.99320 24 1.00
6
6 National and International Benchmarking using NAEP and TIMSS ( simulated data ) Math, Grade 8NAEP-equivalents to State ScoreTIMSS-equivalents to State Score Page inEstimatedNAEP-StandardProportionTIMSS-StandardProportion BookmarkStateProportionEquivalentError ofin U.S.EquivalentError ofin Korea OrderedScaledAboveScaledNAEP-AboveScaledTIMSS-Above Item BookletScoresCut-ScoreScoreEquivalentNAEP-EquivScoreEquivalentTIMSS-Equiv 418000.01 Ad = 333 350 10.9 0.03 Ad=625 655 25 0.26 407880.02341 8.0 0.05635 20 0.34 397870.02340 7.8 0.05633 19 0.35 387850.02339 7.4 0.06630 19 0.36 377800.03335 6.7 0.07 High = 550 622 17 0.39 367750.03 Proficient = 299 331 6.0 0.09613 16 0.43 357700.04327 5.4 0.10605 15 0.47 347660.05324 5.0 0.12598 15 0.49 337600.07319 4.6 0.15588 14 0.54 327550.09315 4.2 0.18580 14 0.57 317480.12309 3.8 0.22568 13 0.62 307450.13307 3.7 0.24563 13 0.64 297400.16303 3.5 0.27555 13 0.68 287350.19299 3.3 0.31 Intermediate = 475 546 13 0.71 277300.22 Basic = 262 295 3.1 0.35538 13 0.74 267260.25292 3.0 0.38531 13 0.76 257200.30287 2.9 0.43521 13 0.80 247150.34283 2.8 0.48513 13 0.82 237120.37281 2.8 0.50508 13 0.83 227050.43276 2.7 0.56496 13 0.86 217000.48272 2.7 0.61488 13 0.88 206990.49271 2.7 0.62486 13 0.89 196900.58264 2.7 0.69 Low = 400 471 13 0.91 186850.62 Below Basic 260 2.8 0.73463 13 0.93 176800.67256 2.8 0.76454 13 0.94 166750.71252 2.9 0.79446 13 0.95 156730.72250 3.0 0.81442 13 0.95 146650.78244 3.2 0.85429 14 0.97 136600.82240 3.3 0.87421 14 0.97 126560.84237 3.5 0.89414 14 0.98 116500.87232 3.7 0.91404 14 0.98 106450.90228 4.0 0.93 Below Low 396 15 0.99 96400.92224 4.3 0.94387 15 0.99 86400.92224 4.3 0.94387 15 0.99 76300.95216 5.0 0.96370 16 0.99 66250.96213 5.5 0.97362 17 0.99 56200.97209 6.1 0.98354 18 1.00 46200.97209 6.1 0.98354 19 1.00 36100.98201 7.6 0.99337 20 1.00 26060.99198 8.4 0.99330 22 1.00 16000.99193 9.8 0.99320 24 1.00
7
7 National and International Benchmarking using NAEP and TIMSS ( simulated data ) Math, Grade 8NAEP-equivalents to State ScoreTIMSS-equivalents to State Score Page inEstimatedNAEP-StandardProportionTIMSS-Standard BookmarkStateProportionEquivalentError ofin U.S.EquivalentError ofTIMSS- OrderedScaledAboveScaledNAEP-AboveScaledTIMSS-Equivalent Item BookletScoresCut-ScoreScoreEquivalentNAEP-EquivScoreEquivalentRange 418000.01 Ad = 333 350 10.9 0.03 Ad=625 655 25 630-680 407880.02341 8.0 0.05635 20 615-655 397870.02340 7.8 0.05633 19 614-652 387850.02339 7.4 0.06630 19 611-649 377800.03335 6.7 0.07 High = 550 622 17 605-639 367750.03 Proficient = 299 331 6.0 0.09613 16 597-629 357700.04327 5.4 0.10605 15 590-620 347660.05324 5.0 0.12598 15 583-613 337600.07319 4.6 0.15588 14 574-602 327550.09315 4.2 0.18580 14 566-594 317480.12309 3.8 0.22568 13 555-581 307450.13307 3.7 0.24563 13 550-576 297400.16303 3.5 0.27555 13 542-568 287350.19299 3.3 0.31 Intermediate = 475 546 13 533-559 277300.22 Basic = 262 295 3.1 0.35538 13 525-551 267260.25292 3.0 0.38531 13 518-544 257200.30287 2.9 0.43521 13 508-534 247150.34283 2.8 0.48513 13 500-526 237120.37281 2.8 0.50508 13 495-521 227050.43276 2.7 0.56496 13 483-509 217000.48272 2.7 0.61488 13 475-501 206990.49271 2.7 0.62486 13 473-499 196900.58264 2.7 0.69 Low = 400 471 13 458-484 186850.62 Below Basic 260 2.8 0.73463 13 450-476 176800.67256 2.8 0.76454 13 441-467 166750.71252 2.9 0.79446 13 433-459 156730.72250 3.0 0.81442 13 429-455 146650.78244 3.2 0.85429 14 415-443 136600.82240 3.3 0.87421 14 407-435 126560.84237 3.5 0.89414 14 400-428 116500.87232 3.7 0.91404 14 390-418 106450.90228 4.0 0.93 Below Low 396 15 381-411 96400.92224 4.3 0.94387 15 372-402 86400.92224 4.3 0.94387 15 372-402 76300.95216 5.0 0.96370 16 354-386 66250.96213 5.5 0.97362 17 345-379 56200.97209 6.1 0.98354 18 336-372 46200.97209 6.1 0.98354 19 335-373 36100.98201 7.6 0.99337 20 317-357 26060.99198 8.4 0.99330 22 308-352 16000.99193 9.8 0.99320 24 296-344
8
International Benchmarking with PISA 8 The international benchmarking with PISA was replicated in in 2010 in three states (Hawaii, Oregon and Delaware) in three subjects (reading, mathematics and science). The statistical linking with PISA was accomplished by embedding secure PISA items in the state assessment through a legally binding Memo of Understanding and Confidentiality Agreement between AIR and the OECD.
9
Common item Equating for PISA 9 About 30 items are embedded as field test items in the computer adaptive test in the state. The computer adaptive field test algorithm randomly administered the PISA items. Number items per student - If the state had F = 400 items for field testing, P = 30 PISA items, and the test length was L = 40, then the expected number of PISA items administered to each student was E( p ) = 1/F * L * P = 3. Number students per item - If the student population size was N = 10,000, then the expected sample size per PISA item was E( n ) = 1/F * L * N = 1,000. Number of items per student does not provide enough information per student to get estimates of PISA scores (i.e., distributions of PISA- plausible values). Number students per item provides enough information per item to get good state-based estimates of the PISA item parameters.
10
PISA Grade 10 Mathematics 10 Results for State Assessment OrderedTheta Standard Error of ThetaPercentPISA-Standard Percent Reaching Percent Reaching ItemAssociated ReachingequivalentError ofOECDFinland BookletItemWith StateScaledPISA- PageParameterRP67 Cut-scoreScoreequivalent 643.754.50.15266011 48 633.494.20.13364010 613 623.374.10.13463110 715 613.294.00.13462510 817 603.173.90.12461510 20 PISA Level V = 607 (Top Performers) 592.943.60.0965977 1427 582.873.60.1265929 1529 572.803.50.1275879 1732 562.743.40.1175829 1834 552.713.40.0875797 1935 542.653.40.0885756 2037 532.613.30.1185729 2138 522.553.30.1195679 2341 512.483.20.11105619 2544 502.423.10.11105578 2646
11
PISA Grade 10 Mathematics 11 Results for State Assessment OrderedTheta Standard Error of ThetaPercentPISA-Standard Percent Reaching Percent Reaching ItemAssociated ReachingequivalentError ofOECDFinland BookletItemWith StateScaledPISA- PageParameterRP67 Cut-scoreScoreequivalent 492.343.10.11115519 2849 482.293.00.11125468 3051 PISA Level IV = 545 (Strong Performers) 472.263.00.10125448 3152 462.202.90.11125408 3254 452.182.90.10125388 3355 442.172.90.10125378 3455 432.112.80.10155328 3658 422.072.80.10155298 3759 412.042.70.10165278 3860 401.992.70.10175248 3962 391.972.70.10285228 4063 381.942.70.07285206 4164 371.942.60.10285198 4164 361.892.60.10285168 4265 351.892.60.10295158 4366
12
PISA Grade 10 Mathematics 12 Results for State Assessment OrderedTheta Standard Error of ThetaPercentPISA-Standard Percent Reaching Percent Reaching ItemAssociated ReachingequivalentError ofOECDFinland BookletItemWith StateScaledPISA- PageParameterRP67 Cut-scoreScoreequivalent 341.852.60.10295138 4467 331.822.50.10295108 4568 321.772.50.10295068 4770 311.742.40.10295048 4771 301.682.40.10294998 5073 291.652.40.10294978 5074 281.632.30.10304958 5174 271.592.30.10304928 5376 261.552.30.10304898 5477 251.512.20.10304868 5578 241.492.20.10304848 5679 PISA Level III = 482 (Moderate Performers) 231.452.20.10304818 5780 221.412.10.10304788 5981 211.392.10.10304768 5981
13
PISA Grade 10 Mathematics 13 Results for State Assessment OrderedTheta Standard Error of ThetaPercentPISA-Standard Percent Reaching Percent Reaching ItemAssociated ReachingequivalentError ofOECDFinland BookletItemWith StateScaledPISA- PageParameterRP67 Cut-scoreScoreequivalent 201.322.00.10344718 6283 191.252.00.10374658 6485 181.201.90.10384628 6586 171.161.90.10404598 6686 161.111.80.10424548 6888 151.081.80.10434528 6988 141.071.80.07434526 6988 131.031.70.10444498 7089 121.021.70.10454478 7189 110.961.70.10484438 7390 100.871.60.10664368 7592 90.841.60.07674346 7692 80.751.50.11674269 7893
14
PISA Grade 10 Mathematics 14 Results for State Assessment OrderedTheta Standard Error of ThetaPercentPISA-Standard Percent Reaching Percent Reaching ItemAssociated ReachingequivalentError ofOECDFinland BookletItemWith StateScaledPISA- PageParameterRP67 Cut-scoreScoreequivalent 70.751.50.11674268 7893 60.701.40.11674239 7994 PISA Level II = 420 (Moderate Performers) 50.631.30.11684179 8195 40.571.30.11684129 8395 30.371.10.11723979 8697 20.261.00.11953899 8898 10.050.80.12953729 9199
Similar presentations
