Tracking and Analyzing Student Success Data Part 3

State/Territory Area (km 2 ) Afghanistan 645,807606,467 Albania 28,74825,643 Algeria 2,381,7412,556,276 American Samoa Andorra Anguilla 9688 Antigua & Barbuda Argentina 2,777,4092,537,555 Armenia 29,74325,859 Aruba Australia 7,682,5577,365,389 Austria 83,85881,434 Azerbaijan 86,53091,752 Bahamas 13,9629,125 Bahrain Bangladesh 142,61598,722 Barbados Belgium 30,51825,417 Belize 22,96520,648 Benin 112,620108,768 ……… Which column lists the correct areas?

A Bit of History Late 1800s: Astronomer Simon Newcomb noticed that the early pages of log table books were more grubby than the later pages  Users were looking-up numbers that started with digit 1 more often than numbers starting with, say, digit 5. If the leading (first) digit is d, then the frequency of occurrence (probability) of the leading digit is Log 10 (1 + 1/d)

If the leading (first) digit is d, then the frequency of occurrence (probability) of the leading digit is Log 10 (1 + 1/d) Leading digit (d) Probability of occurrence 30%18%12%10%8%7%6%5%< 5% NumberLeading (first) digit

A Bit of History (cont’d) Late 1930s: Physicist Frank Benford rediscovered Newcomb’s formula: Log 10 (1+1/d)  Benford’s Law Source: The Law of Anomalous Numbers, F. Benford, Proceedings of the American Philosophical Society, Vol. 78, 1938, pp:

State/Territory Area (km 2 ) Afghanistan 645,807606,467 Albania 28,74825,643 Algeria 2,381,7412,556,276 American Samoa Andorra Anguilla 9688 Antigua & Barbuda Argentina 2,777,4092,537,555 Armenia 29,74325,859 Aruba Australia 7,682,5577,365,389 Austria 83,85881,434 Azerbaijan 86,53091,752 Bahamas 13,9629,125 Bahrain Bangladesh 142,61598,722 Barbados Belgium 30,51825,417 Belize 22,96520,648 Benin 112,620108,768 ………

State/Territory Area (km 2 ) Afghanistan 645,807606,467 Albania 28,74825,643 Algeria 2,381,7412,556,276 American Samoa Andorra Anguilla 9688 Antigua & Barbuda Argentina 2,777,4092,537,555 Armenia 29,74325,859 Aruba Australia 7,682,5577,365,389 Austria 83,85881,434 Azerbaijan 86,53091,752 Bahamas 13,9629,125 Bahrain Bangladesh 142,61598,722 Barbados Belgium 30,51825,417 Belize 22,96520,648 Benin 112,620108,768 ……… Correct List Source: Improvised from R. M. Fewster, The American Statistician, Feb. 2009, Vol. 63, No. 1.

Benford’s Law in Mastering TM Rasil Warnakulasooriya Pearson, Boston

A Quick Demo of Mastering TM masteringphysics.com Thanks to Claire Masson for the Demo slides

A Tutorial Problem in MasteringPhysics ®

Student enters 0 to Part A

Students can open Hints

Declarative Hint Procedural Hint

Feedback within Hints

Student obtains the correct answers

Time to Completion Data StudentItemTime interval (sec.) [finished time-start time] First (leading) digit S1Tutorial X3503 S1EOC Y10481 S3EOC Y676 S4Tutorial Z …………

MasteringPhysics ® time data follow Benford’s Law closely

MasteringChemistry ® time data follow Benford’s Law closely (better than in physics)

But what can we do with it?

What can the deviations from the Benford’s Law tell us?

Deviation from Benford’s Law less deviationmore deviation

Final Exam Score vs. Deviation from Benford’s Law (Chemistry) Thanks to professors Randall Hall & Leslie Butler for providing the exam score data

Final Exam Score vs. Deviation from Benford’s Law For every 1/5 th of deviation from Benford’s Law, the exam score is decreased by 1/2 standard deviation

To what degree is the average time correlated with the final exam scores? (about 7 times less than the correlation found by using only the first digits; almost no correlation)

The correlation (-0.2) is small but surprisingly good (?) * Typical correlations encountered in educational research ~ 0.3 * High school GPA & SAT ® -math correlated at 0.23  * SAT ® -math and first-year college GPA correlated at 0.26   Source: College Board Research Report No : Validity of the SAT ® for Predicting First-Year College Grade Point Average, Jennifer L. Kobrin, et al.

Can we identify at-risk students?

Using average time: 0% of the students can be flagged Students who sat for the final exam Students who did not sit for the final exam

Using Benford’s Law: ~19% of the students can be flagged Students who sat for the final exam Students who did not sit for the final exam

* In time to completion data the first digit (or deviations from the Benford’s Law) carries more actionable information than all of the digits combined! Summary * Possibility of weighting the deviations from Benford's Law to identify at-risk students so that instructors can intervene effectively * Possibility of weighting the deviations from Benford’s Law in deciding on specific study plans/personalized learning resources within Mastering

State/Territory Area (miles 2 ) Afghanistan 249,347234,158 Albania 11,1009,901 Algeria 919,595986,983 American Samoa 7679 Andorra Anguilla 3734 Antigua & Barbuda Argentina 1,072,363979,755 Armenia 11,4849,984 Aruba 7581 Australia 2,966,2502,843,791 Austria 32,37831,442 Azerbaijan 33,40935,426 Bahamas 5,3913,523 Bahrain Bangladesh 55,06438,117 Barbados Belgium 11,7839,814 Belize 8,8677,972 Benin 43,48341,996 ………