1. It’s no secret Measuring the security and reliability of authentication via ‘secret’ questions Stuart Schechter, A. J. Bernheim Brush, Serge Egelman IEEE S&P ’09 Presented by: HAN Jin

2. Outline Motivation & Introduction Background Study recruitment and methodology Discussion 2

3. Motivation Forums, blogs, online-games may use may authenticate users who have forgotten their passwords via their email addresses, webmail services cannot always do so. All four of the most popular webmail providers – AOL, Google, Microsoft, and Yahoo! – rely on personal questions as the secondary authentication secrets used to reset account passwords. 3

4. Motivation The most recent burst: Yahoo! –2008 vice presidential nominee Sarah Palin’s Yahoo! account had been compromised by someone who researched the answer to the question: “Where did you meet your spouse?” one questionDespite the consequences of authentication failures, the four largest webmail providers require only one question be answered in order to reset an account’s password. 4

5. Introduction security and reliability authentication questionsTo quantify the security and reliability of personal authentication questions as they are used today, a user study These authors ran a user study for those questions used by all four top webmail providers. 5

6. Study Recruitment 4 separate days (March 22 ~ June 23, 2008) 130 participants (ppts) 64 male, 66 females A diversity of ages and professions 6

7. Participant recruitment Their recruiting team selected participants from a larger pool of potential participants they maintain for all studies at Microsoft. partnersAll participants were required to have partners and the categories of relationships between participants and their partners are broken down in Table 2c. 7

8. Initial laboratory visit Two-hour visit 8

9. Tricky parts Awards –They offered two prizes (an XBOX 360 and a Zune digital music player) and gave participants a virtual lottery ticket for each question they both answered and later recalled. Authors anticipated participants might 1. try to increase their chance of recalling their answers by providing the same answer for all questions –They added a rule that eliminated rewards for recalling the same answer numerous times 2. Participants might record their answers –They did not inform participants that we would follow- up to test their recollections in the future. 9

10. Reliability (memorability) follow-up Answers were judged as correct recollections if they differed from the original only in the use of white space, punctuation, and capitalization. To encourage participants to do their best at recalling their original answers, authors offered all participants a new incentive, again based on the percentage of answers they recalled. –The top quartile received an Amazon.com gift card worth $15, the second quartile received one worth $10, the third $5, and the last quartile received no performance-based gratuity. –In addition, all participants received some form of base gratuity just for participating 10

11. Answer comparison algorithms equality –an artifact in their study: the Illume survey software they used to collect the answers fails to store carriage returns substring –treated a guess as valid if it contained a substring that matched the original answer distance –Levenshtein edit distance algorithm 11

12. Answer comparison algorithms (cont.) distance algorithm: –reduced the cost of transpositions of two characters (‘swapped’  ‘sawpped’) from two to one –They allowed one error (an edit distance cost of one) for every five characters in the original answer Change from substring to distance: 2.5 –reduces the number of answers forgotten (not recalled within 5 attempts) by 2.5% (11.3% reduction) 1.4 –increased the percentage of answers guessed by participants’ partners by 1.4% (6.8% relative increase) 12

13. 13

14. Closely analysis The trade-off was well worth it: In 34 of the 40 cases where a guess was treated as incorrect by the substring algorithm but correct by the distance algorithm (80%), the guessing partner clearly knew the correct answer: The difference was a one character typing error that an attacker could easily fix with a second guess. 14

15. 15 Results

16. Real-world memorability results Resetting Hotmail passwords needs –An correct answer to a personal question & correct answer to zip code Only 43 out of 99 (43%) reported participants were able to successfully provide the correct answer to their personal question and zip code, the rest 57%: –75% unable to answer their personal question –31% unable to recall the zip code –A surprising 13% of participants suspected that the reason they could not answer their personal question was because they had intentionally provided a bogus answer when setting up their account. 16

17. Main results The results for all questions used by the top four webmail services (as of March, 2008) are summarized in Table 4. –Willingness to answer “not willing”, “unknown”, and “don’t have one” –Reliability (memorability) –Security against statistical guessing An answer is deemed vulnerable to this attack if it is among the five most popular answers provided by other participants (excluding the participant’s partner) –Security against guessing by acquaintance 17

18. 18

19. 19 Top left part of the result table

20. Top right of the result table 20

21. Results analysis Google’s questions performed the best since the overall guess rate was just 4%. Questions with answers that participants found easiest to recall appeared to be those that their partners found easiest to guess. A non-parametric Kendall test, examining the correlation between the fraction of answers recalled for each question and the fraction guessed by participants’ partners, indicates a strong correlation, 21

22. The security of user-written questions 22 2. Vulnerable with no personal knowledge other than geographic region (31 of 127, 24%) i.Answer can be found via simple web search (2, 2%) What’s your favorite cookie at Panera Bakery? ii.Answer space <= 5 (11, 8%), <= 10 (15, 12%) & <= 25 (18, 14%) How many children do I have? What is my blood type? iii.Answer high on easily searchable popularity lists, top 5 (6, 5%), top 25 (11, 7%) Favorite Food What sports team would you love to see lose 3. Vulnerable to coworkers, clients, or family members (32 of 127, 25%)

23. Discussions Improving questions to reduce vulnerability to statistical guessing attacks: –responses could be penalized in proportion to their popularity –reduce the proportion of popular answers: rejecting answers that exceed a certain threshold of popularity (e.g. 1%) Alternative backup authenticators –authentication via a code sent to an alternate email address – not viable for users’ primary email accounts –mobile phones – frequently shared, lost, and stolen –User-selected trustees vouch for the identity of the user. 23

24. Related Work Two other earlier works for the use of personal questions for authentication –Zviran and Haga in 1990 [17] –Podd et al. in 1996 They extend prior research by untrusted acquaintances –measuring the security of those questions against guessing not just by significant others, but by untrusted acquaintances as well statistical guessing attacks –They also examine the vulnerability of those questions to statistical guessing attacks. 24

25. Epilog On November 12, 2008, authors contacted AOL, Google, and Yahoo! to provide them with a draft of this paper and share their intent to publish at this symposium. In February 2009, Yahoo! had replaced all nine of the personal authentication questions that its users may choose from when signing up for a new account. 25

26. My conclusion Aim the top –webmail providers –conference Good funding Easy extendable 26