1. Analysis of AP Exam Scores
Apples to Apples

2. The Problem
How do we compare the pass rate on the AP exam for a particular AP course with the nationwide pass rate for that course?
How do we compare the pass rate on the AP exam for a particular AP course from one school to another?
How do we compare the pass rate on the AP exam for two different AP courses?

3. College Board Exams
The College Board administers the AP exams, SAT and PSAT
The College Board has data on millions of students who have taken multiple exams
There is a correlation between how well students perform on the PSAT and how well they perform on certain AP exams
Passing an AP exam is often considered to be a score of 3 or higher

4. PSAT – AP Correlation
College Board statisticians looked at scores of over 200,000 students who took both the PSAT and the AP Calculus AB exam
They determined that there was a significant correlation (0.55) between PSAT Math and performance on the AP Calculus AB exam
They also determined that there was no significant correlation between PSAT Critical Reading or PSAT Writing and performance on the AP Calculus AB exam

5. PSAT – AP Relationship
The higher students score on PSAT Math, the higher their probability of passing the AP Calculus AB exam
Of students who took both the PSAT and the AP Calculus AB exam:
10% of those scoring 43 on PSAT Math passed the AP Calculus AB exam
50% of those scoring 57 on PSAT Math passed the AP Calculus AB exam
90% of those scoring 71 on PSAT Math passed the AP Calculus AB exam

6. PSAT – AP Relationship
Depending on the AP course, success on the AP exam is related to different subsections of the PSAT
Courses related to PSAT Critical Reading + Math
Biology, Chemistry, Computer Science A, Environmental Science, Macro/Micro Economics, Physics B and C, Statistics, World History
Courses related to PSAT Critical Reading + Writing
Art History, English Language, English Literature

7. PSAT – AP Relationship Courses related to PSAT Math + Writing
Music Theory
Courses related to PSAT Critical Reading + Math + Writing
European History, Government (US and Comparative), Human Geography, Psychology, US History
All other AP courses show no significant correlation between exam success and PSAT scores

8. AP Potential Service offered by the College Board
Helps school administrators identify students at their school who could be successful in various AP courses
This is where the relationship between PSAT and AP exam success is published (along with many College Board publications)

9. AP Potential: Retrospectively
After the AP exam, gather PSAT data on students
If an AP Calculus AB class has any students with PSAT Math scores of 43, 10% of them should have passed the AP exam
If an AP Calculus AB class has any students with PSAT Math scores of 71, 90% of them should have passed the AP exam
That’s how similar students nationwide performed

10. AP Potential: Expected Passers
For any PSAT Math score, we predict the student’s probability of passing the AP Calculus AB exam
Assuming they perform like similar students nationwide
Combining these predictions for all students in the AP Calculus AB class gives a prediction of how many students we should expect to pass
Examine how students actually performed compared to how we should have expected them to perform

11. Graphical Analysis
A graph will allow a quick visual evaluation of student performance
One curve shows the relationship between PSAT score and AP exam pass rate nationwide
One curve shows the relationship between PSAT score and AP exam pass rate for our class
We would like to see the curve for our class above the nationwide curve

12. AP Government (US)
Now let’s take a look at the AP exam for AP Government
Define success as obtaining a score of 3 or higher on the AP exam
Success is correlated with the sum of PSAT Critical Reading + Math + Writing

13. Dissecting the Graph blue dots, blue curve
The smooth blue curve shows the overall relationship between PSAT score and AP pass rate for the AP Government exam nationwide
Nationwide, 90% of students with a combined PSAT score of 195 pass the AP Government exam
Nationwide, 10 % of students with a combined PSAT score of 123 pass the AP Government exam

14. Dissecting the Graph black dots
This black dot represents a student who had a combined PSAT score of 130, but did not pass the AP exam, coded as 0
Each black dot represents a student in our dataset. This graph is analyzing 93 students who took both the PSAT and the AP Government exam, so there are 93 black dots
This black dot represents a student who had a combined PSAT score of 130, and did pass the AP exam, coded as 1
All of the black dots are either at 1 or 0 on the y axis, 1 for pass (3 or higher) and 0 for fail (less than 3) on the AP exam

15. Dissecting the Graph green dots, green curve
The green circles represent grouped data within our dataset
The smooth green curve shows the overall relationship between PSAT score and AP pass rate for the AP Government exam for this dataset
This green circle represents the 7 students who scored between 110 and 119 on the PSAT, none of whom passed the AP Government exam, for a pass rate of 0%
This green circle represents the 20 students who scored between 120 and 129 on the PSAT, 5 of whom passed the AP Government exam, for a pass rate of 25%
Generally speaking, green circles and a green curve above the blue curve is a good thing

16. Numerical Analysis
Of these 93 US Government students, we should have expected 24.5 to pass the AP exam
Assuming they perform as students similar to them perform nationwide, based on their PSAT scores
In fact, 31 of these students passed the AP exam
With a random sample of 93 students like these (with their PSAT scores) we would observe 31 or more pass the AP exam just 6% of the time
Poissonian Binomial Distribution

17. Simulation Analysis
Simulate these 93 students taking the AP Government exam
Run the simulation for 1000 trials
For each trial, record the number of students who pass the AP exam
Create a graph of the simulation results

18. Simulation Results
Thus 31 successes is not just more than we expected, it is on the threshold of being statistically significantly more than expected
Of the 1000 trials, 57 (5.7%) resulted in 31 or more successes, and 29 (2.9%) resulted in more than 31 successes

19. Causation?
We must be careful not to attach causation to correlation in general. In particular, there may be many different reasons why this particular group of students has performed better than expected
Teacher experience
Teacher training
Teacher efficacy
Class size
Instructional materials
Average PSAT score
School culture
Parental involvement

20. Success
Usually success on AP exams is considered to be a score of 3, 4 or 5
Some high schools have very high pass rates
Some colleges require a 4 on the AP exam (or even a 5) in order for the student to obtain college credit
Some prefer to define success on AP exams as earning a 4 or 5 only
Let’s look at two AP Government graphs

21. Government (US)
Pass: 3, 4, 5
As we’ve already discussed, the curve for this class is nicely above the national curve. A graphical demonstration that 31 passed when only 24.5 were expected to pass.

22. Government (US)
Pass: 4, 5
This agrees nicely with the numerical analysis, which shows that we would have expected 8.7 of these 93 students to get a 4 or 5, based on their PSAT scores, while we actually observed 10 of them get a 4 or 5.
The curve for this class, with success now defined as 4 or 5, is in line with – if not a little above – the national curve.

23. Other Subjects

24. Biology
Pass: 3, 4, 5
This z-shape squarish curve occurs when there is no overlap in the data – in this case every student with PSAT score under 106 failed the AP exam, and all those with PSAT score over 106 passed the AP exam.
Statistical Note:
Referred to as complete separation or quasi-complete separation, it causes problems with the regression convergence.
This dataset has only 11 students, with 4.2 expected to pass, and 4 actually passed.

25. Calculus AB
Pass: 3, 4, 5
This class curve is well below the national curve over the entire range of PSAT scores.
This low number of passers is expected less than 1% of the time just by chance
Nationwide, about 60% of students with a PSAT Math score of 60 pass the AP Calculus AB exam.
This agrees nicely with the numerical analysis which revealed that only 12 students passed, although 21 were expected to pass.
Students in this class with a PSAT Math score of 60 would be predicted to have only about a 42% pass rate.

26. Calculus AB
Pass: 4, 5
This class curve is far below the national curve over the entire range of PSAT scores.
This low number of 4s is expected less than 0.1% of the time just by chance.
This agrees nicely with the numerical analysis which revealed that although 11 students were expected to get a 4 or better, only 3 students actually did.

27. English Language Pass: 3, 4, 5
This class curve is well below the national curve over the entire range of PSAT scores.
For students with a PSAT CR + W score of 100, there is a 35% gap between the pass rate for this class and the pass rate nationwide.
This agrees nicely with the numerical analysis which revealed that although 35 students were expected to pass, only 19 students actually passed.
This low number of passers never happens just by chance.

28. English Language Pass: 4, 5
This class is even worse when we look at the 4s.
This low number of 4s happens about 0.01% of the time just by chance.
This agrees nicely with the numerical analysis which revealed that 11 students were expected to earn a 4 or better, but only 2 students actually got a 4 or better.

29. English Literature Pass: 3, 4, 5
The students in this class did substantially better than similar students nationwide.
This many passers for a group of students like these only happens about 0.02% of the time just by chance!
These graphs echo what the numerical analysis showed. This group of students would be expected to have 32 pass the AP exam, but in fact 46 passed the AP exam.

30. English Literature Pass: 4, 5
The results for this class do not look quite as good when we look at the 4s.
Statistical Note:
Quasi-complete separation is messing with the regression a bit here. Almost all of those who failed the AP exam had PSAT scores of 110 and below, while all the passers had PSAT scores of 110 and above.
About 8 of these students were expected to earn a 4 or better, and 7 actually did.

31. Environmental Science
Pass: 3, 4, 5
The students in this class did slightly worse than similar students nationwide. The biggest gap between the curves is about 10%.
Results like this, or worse, happen about 14% of the time just by chance.
Numerical analysis shows that about 16 of these students should have passed, but only 12 actually did pass.

32. Environmental Science
Pass: 4, 5
The results for this class look about the same if success is 4 or higher.
About 7.5 of these students were expected to earn a 4 or better, and 5 actually did.
This type of result, or worse, happens about 18% of the time.

33. Psychology
Pass: 3, 4, 5
For a group of students like this, we would never observe a result this bad – 7 students or worse pass the AP exam – just by chance.
The students in this class performed far worse than their nationwide peers – in some cases a pass rate 50% below the nationwide pass rate.
This class was expected to have 19 students pass, and instead only 7 did.

34. Psychology
Pass: 4, 5
The students in this class performed slightly better – but still poorly – when we consider a passing score to be 4 or 5.
For a group of students like this, we would expect 5 passers – with scores of 4 or 5 – only 0.3% of the time by chance alone.
This class was expected to have 12 students earn a 4 or higher, but only 5 did. Interesting that only 7 of 41 students passed this exam, but 5 of the 7 that passed got a 4 or 5.

35. Statistics
Pass: 3, 4, 5
This class curve is above the national curve for all values of PSAT score.
For a group of students like this, we would expect to observe 18 or more passers only about 5% of the time just by chance.
This class was expected to have 13 students pass, and 18 actually passed.

36. Statistics
Pass: 4, 5
The students in this class also performed significantly better than similar students nationwide when we consider a passing score to be 4 or 5.
For a group of students like this, we would expect 9 or more passers – with scores of 4 or 5 – less than 4% of the time by chance alone.
This class was expected to have about 5 students earn a 4 or 5, but 9 students actually earned a 4 or 5.

37. US History
Pass: 3, 4, 5
The students in this class performed worse than their nationwide peers.
For a group of students like this, we would expect to see 14 passers, or worse, only 0.6% of the time by chance alone.
This class was expected to have 23 students pass, and instead only 14 did.

38. US History
Pass: 4, 5
This is about the worst a class curve will ever look, without being a flat line at 0.
For a group of students like this, we would expect 2 passers – with scores of 4 or 5 – only 0.04% of the time by chance alone.
This class of 69 students was expected to have 10 students earn a 4 or higher, but only 2 did.

39. World History 2014 – 2015: Old PSAT Pass: 3, 4, 5
The students in this class performed far worse than their nationwide peers – in some cases 40% worse.
For a group of students like this, we would expect to see 3 passers, or worse, only 0.3% of the time by chance alone.
This class was expected to have 10 students pass, but instead only 3 did.

40. World History 2016: New PSAT Pass: 3, 4, 5
The students in this class performed better than their nationwide peers for most of the range of PSAT scores.
For a group of students like this, we would expect to see 21 passers, or better, about 13% of the time by chance alone.
This class was expected to have 17 students pass, but actually 21 passed.
Notice the different scale on the PSAT axis due to the new PSAT given in the fall of 2015.

41. World History 2016: New PSAT Pass: 4, 5
These 57 students, however, performed far worse than their nationwide peers when passing is considered to be a score of 4 or 5.
For a group of students like this, we would expect to see only 1 score of 4 or 5 less than 1% of the time just by chance.
These 57 students were expected to have 6 score a 4 or 5, but only 1 did.