Analysis of AP Exam Scores Apples to Apples

The Problem How do we compare the pass rate on the AP exam for a particular AP course with the nationwide pass rate for that course? How do we compare the pass rate on the AP exam for a particular AP course from one school to another? How do we compare the pass rate on the AP exam for two different AP courses?

College Board Exams The College Board administers the AP exams, SAT and PSAT The College Board has data on millions of students who have taken multiple exams There is a correlation between how well students perform on the PSAT and how well they perform on certain AP exams Passing an AP exam is often considered to be a score of 3 or higher

PSAT – AP Correlation College Board statisticians looked at scores of over 200,000 students who took both the PSAT and the AP Calculus AB exam They determined that there was a significant correlation (0.55) between PSAT Math and performance on the AP Calculus AB exam They also determined that there was no significant correlation between PSAT Critical Reading or PSAT Writing and performance on the AP Calculus AB exam

PSAT – AP Relationship The higher students score on PSAT Math, the higher their probability of passing the AP Calculus AB exam Of students who took both the PSAT and the AP Calculus AB exam: 10% of those scoring 43 on PSAT Math passed the AP Calculus AB exam 50% of those scoring 57 on PSAT Math passed the AP Calculus AB exam 90% of those scoring 71 on PSAT Math passed the AP Calculus AB exam

PSAT – AP Relationship Depending on the AP course, success on the AP exam is related to different subsections of the PSAT Courses related to PSAT Critical Reading + Math Biology, Chemistry, Computer Science A, Environmental Science, Macro/Micro Economics, Physics B and C, Statistics, World History Courses related to PSAT Critical Reading + Writing Art History, English Language, English Literature

PSAT – AP Relationship Courses related to PSAT Math + Writing Music Theory Courses related to PSAT Critical Reading + Math + Writing European History, Government (US and Comparative), Human Geography, Psychology, US History All other AP courses show no significant correlation between exam success and PSAT scores

AP Potential Service offered by the College Board Helps school administrators identify students at their school who could be successful in various AP courses This is where the relationship between PSAT and AP exam success is published (along with many College Board publications) http://www.collegeboard.com/counselors/app/expectancy.html

AP Potential: Retrospectively After the AP exam, gather PSAT data on students If an AP Calculus AB class has any students with PSAT Math scores of 43, 10% of them should have passed the AP exam If an AP Calculus AB class has any students with PSAT Math scores of 71, 90% of them should have passed the AP exam That’s how similar students nationwide performed

AP Potential: Expected Passers For any PSAT Math score, we predict the student’s probability of passing the AP Calculus AB exam Assuming they perform like similar students nationwide Combining these predictions for all students in the AP Calculus AB class gives a prediction of how many students we should expect to pass Examine how students actually performed compared to how we should have expected them to perform

Graphical Analysis A graph will allow a quick visual evaluation of student performance One curve shows the relationship between PSAT score and AP exam pass rate nationwide One curve shows the relationship between PSAT score and AP exam pass rate for our class We would like to see the curve for our class above the nationwide curve

AP Government (US) Now let’s take a look at the AP exam for AP Government Define success as obtaining a score of 3 or higher on the AP exam Success is correlated with the sum of PSAT Critical Reading + Math + Writing

Dissecting the Graph blue dots, blue curve The smooth blue curve shows the overall relationship between PSAT score and AP pass rate for the AP Government exam nationwide Nationwide, 90% of students with a combined PSAT score of 195 pass the AP Government exam Nationwide, 10 % of students with a combined PSAT score of 123 pass the AP Government exam

Dissecting the Graph black dots This black dot represents a student who had a combined PSAT score of 130, but did not pass the AP exam, coded as 0 Each black dot represents a student in our dataset. This graph is analyzing 93 students who took both the PSAT and the AP Government exam, so there are 93 black dots This black dot represents a student who had a combined PSAT score of 130, and did pass the AP exam, coded as 1 All of the black dots are either at 1 or 0 on the y axis, 1 for pass (3 or higher) and 0 for fail (less than 3) on the AP exam

Dissecting the Graph green dots, green curve The green circles represent grouped data within our dataset The smooth green curve shows the overall relationship between PSAT score and AP pass rate for the AP Government exam for this dataset This green circle represents the 7 students who scored between 110 and 119 on the PSAT, none of whom passed the AP Government exam, for a pass rate of 0% This green circle represents the 20 students who scored between 120 and 129 on the PSAT, 5 of whom passed the AP Government exam, for a pass rate of 25% Generally speaking, green circles and a green curve above the blue curve is a good thing

Numerical Analysis Of these 93 US Government students, we should have expected 24.5 to pass the AP exam Assuming they perform as students similar to them perform nationwide, based on their PSAT scores In fact, 31 of these students passed the AP exam With a random sample of 93 students like these (with their PSAT scores) we would observe 31 or more pass the AP exam just 6% of the time Poissonian Binomial Distribution

Simulation Analysis Simulate these 93 students taking the AP Government exam Run the simulation for 1000 trials For each trial, record the number of students who pass the AP exam Create a graph of the simulation results

Simulation Results Thus 31 successes is not just more than we expected, it is on the threshold of being statistically significantly more than expected Of the 1000 trials, 57 (5.7%) resulted in 31 or more successes, and 29 (2.9%) resulted in more than 31 successes

Causation? We must be careful not to attach causation to correlation in general. In particular, there may be many different reasons why this particular group of students has performed better than expected Teacher experience Teacher training Teacher efficacy Class size Instructional materials Average PSAT score School culture Parental involvement

Success Usually success on AP exams is considered to be a score of 3, 4 or 5 Some high schools have very high pass rates Some colleges require a 4 on the AP exam (or even a 5) in order for the student to obtain college credit Some prefer to define success on AP exams as earning a 4 or 5 only Let’s look at two AP Government graphs

Government (US) Pass: 3, 4, 5 As we’ve already discussed, the curve for this class is nicely above the national curve. A graphical demonstration that 31 passed when only 24.5 were expected to pass.

Government (US) Pass: 4, 5 This agrees nicely with the numerical analysis, which shows that we would have expected 8.7 of these 93 students to get a 4 or 5, based on their PSAT scores, while we actually observed 10 of them get a 4 or 5. The curve for this class, with success now defined as 4 or 5, is in line with – if not a little above – the national curve.

Other Subjects 2014 - 2016

Biology Pass: 3, 4, 5 This z-shape squarish curve occurs when there is no overlap in the data – in this case every student with PSAT score under 106 failed the AP exam, and all those with PSAT score over 106 passed the AP exam. Statistical Note: Referred to as complete separation or quasi-complete separation, it causes problems with the regression convergence. This dataset has only 11 students, with 4.2 expected to pass, and 4 actually passed.

Calculus AB Pass: 3, 4, 5 This class curve is well below the national curve over the entire range of PSAT scores. This low number of passers is expected less than 1% of the time just by chance Nationwide, about 60% of students with a PSAT Math score of 60 pass the AP Calculus AB exam. This agrees nicely with the numerical analysis which revealed that only 12 students passed, although 21 were expected to pass. Students in this class with a PSAT Math score of 60 would be predicted to have only about a 42% pass rate.

Calculus AB Pass: 4, 5 This class curve is far below the national curve over the entire range of PSAT scores. This low number of 4s is expected less than 0.1% of the time just by chance. This agrees nicely with the numerical analysis which revealed that although 11 students were expected to get a 4 or better, only 3 students actually did.

English Language Pass: 3, 4, 5 This class curve is well below the national curve over the entire range of PSAT scores. For students with a PSAT CR + W score of 100, there is a 35% gap between the pass rate for this class and the pass rate nationwide. This agrees nicely with the numerical analysis which revealed that although 35 students were expected to pass, only 19 students actually passed. This low number of passers never happens just by chance.

English Language Pass: 4, 5 This class is even worse when we look at the 4s. This low number of 4s happens about 0.01% of the time just by chance. This agrees nicely with the numerical analysis which revealed that 11 students were expected to earn a 4 or better, but only 2 students actually got a 4 or better.

English Literature Pass: 3, 4, 5 The students in this class did substantially better than similar students nationwide. This many passers for a group of students like these only happens about 0.02% of the time just by chance! These graphs echo what the numerical analysis showed. This group of students would be expected to have 32 pass the AP exam, but in fact 46 passed the AP exam.

English Literature Pass: 4, 5 The results for this class do not look quite as good when we look at the 4s. Statistical Note: Quasi-complete separation is messing with the regression a bit here. Almost all of those who failed the AP exam had PSAT scores of 110 and below, while all the passers had PSAT scores of 110 and above. About 8 of these students were expected to earn a 4 or better, and 7 actually did.

Environmental Science Pass: 3, 4, 5 The students in this class did slightly worse than similar students nationwide. The biggest gap between the curves is about 10%. Results like this, or worse, happen about 14% of the time just by chance. Numerical analysis shows that about 16 of these students should have passed, but only 12 actually did pass.

Environmental Science Pass: 4, 5 The results for this class look about the same if success is 4 or higher. About 7.5 of these students were expected to earn a 4 or better, and 5 actually did. This type of result, or worse, happens about 18% of the time.

Psychology Pass: 3, 4, 5 For a group of students like this, we would never observe a result this bad – 7 students or worse pass the AP exam – just by chance. The students in this class performed far worse than their nationwide peers – in some cases a pass rate 50% below the nationwide pass rate. This class was expected to have 19 students pass, and instead only 7 did.

Psychology Pass: 4, 5 The students in this class performed slightly better – but still poorly – when we consider a passing score to be 4 or 5. For a group of students like this, we would expect 5 passers – with scores of 4 or 5 – only 0.3% of the time by chance alone. This class was expected to have 12 students earn a 4 or higher, but only 5 did. Interesting that only 7 of 41 students passed this exam, but 5 of the 7 that passed got a 4 or 5.

Statistics Pass: 3, 4, 5 This class curve is above the national curve for all values of PSAT score. For a group of students like this, we would expect to observe 18 or more passers only about 5% of the time just by chance. This class was expected to have 13 students pass, and 18 actually passed.

Statistics Pass: 4, 5 The students in this class also performed significantly better than similar students nationwide when we consider a passing score to be 4 or 5. For a group of students like this, we would expect 9 or more passers – with scores of 4 or 5 – less than 4% of the time by chance alone. This class was expected to have about 5 students earn a 4 or 5, but 9 students actually earned a 4 or 5.

US History Pass: 3, 4, 5 The students in this class performed worse than their nationwide peers. For a group of students like this, we would expect to see 14 passers, or worse, only 0.6% of the time by chance alone. This class was expected to have 23 students pass, and instead only 14 did.

US History Pass: 4, 5 This is about the worst a class curve will ever look, without being a flat line at 0. For a group of students like this, we would expect 2 passers – with scores of 4 or 5 – only 0.04% of the time by chance alone. This class of 69 students was expected to have 10 students earn a 4 or higher, but only 2 did.

World History 2014 – 2015: Old PSAT Pass: 3, 4, 5 The students in this class performed far worse than their nationwide peers – in some cases 40% worse. For a group of students like this, we would expect to see 3 passers, or worse, only 0.3% of the time by chance alone. This class was expected to have 10 students pass, but instead only 3 did.

World History 2016: New PSAT Pass: 3, 4, 5 The students in this class performed better than their nationwide peers for most of the range of PSAT scores. For a group of students like this, we would expect to see 21 passers, or better, about 13% of the time by chance alone. This class was expected to have 17 students pass, but actually 21 passed. Notice the different scale on the PSAT axis due to the new PSAT given in the fall of 2015.

World History 2016: New PSAT Pass: 4, 5 These 57 students, however, performed far worse than their nationwide peers when passing is considered to be a score of 4 or 5. For a group of students like this, we would expect to see only 1 score of 4 or 5 less than 1% of the time just by chance. These 57 students were expected to have 6 score a 4 or 5, but only 1 did.