1. AUTHENTICATION MELEE A Usability Analysis of Seven Web Authentication Systems Scott Ruoti, Brent Roberts, Kent Seamons Internet Security Research Lab Brigham Young University http://isrl.byu.edu World Wide Web Conference 2015, Florence, Italy

2. Acknowledge co-authors Scott Ruoti Brent Roberts

3. Passwords Rule the Web Deployed everywhere Well-known security problems Many proposed systems to replace them Passwords have combination of usability, deployability, and security that is hard to beat Bonneau et al. The Quest to Replace Passwords, IEEE Security & Privacy 2012 May 20, 2015WWW 2015, Florence, Italy3

4. Our Research Agenda Develop Single Sign-on Protocols using Secure Remote Password (SRP) Analyze security and usability of our system We wanted to leverage experience from prior work in usable authentication What are the most usable authentication systems? How to measure the usability of an authentication system? May 20, 2015WWW 2015, Florence, Italy4

5. Where Are We? Looked at systems proposed in research literature No clear best system Lack of empirical analysis May 20, 2015WWW 2015, Florence, Italy5

6. Where Are We? Looked at systems proposed in research literature No clear best system Lack of empirical analysis Limitations Proposals are not evaluated using standard usability metrics Proposals are not compared against each other May 20, 2015WWW 2015, Florence, Italy6

7. Security vs. Usability Big Dog vs. little dog May 20, 2015WWW 2015, Florence, Italy7

8. Where Do We Want to Be? Elevate usability on an equal footing with security Truly secure systems must be both secure and usable Determine which proposals have the best overall usability Use a standard metric Head-to-head comparison of proposals Identify best in class systems Establish a basis for evaluating new proposals New proposals should not receive serious attention until they demonstrate acceptable usability Security researchers can be poor predictors of usability May 20, 2015WWW 2015, Florence, Italy8

9. Authentication Melee Conducted empirical analysis of seven web authentication systems Federated single sign-on: Google OAuth 2.0, Facebook Connect, Mozilla Persona Email-based: SAW, Hatchet QR Code-based: WebTicket, Snap2Pass Used the System Usability Scale (SUS) as a standard usability metric Organized systems into head-to-head competitions May 20, 2015WWW 2015, Florence, Italy9

10. Tournament Structure Difficult to do a full combinatorial study If each participant tests two systems, it requires a large number of participants If each participant tests all systems, it can lead to study fatigue Instead we structured our study into a tournament First round based on type of authentication system May 20, 2015WWW 2015, Florence, Italy10

11. Federated Single Sign-on Authentication is centralized into a single identifying party The website relies on the identifying party to verify the identity of users Systems Google OAuth 2.0 Widespread Facebook Connect Widespread Mozilla Persona Identifying party only handles authentication May 20, 2015WWW 2015, Florence, Italy11

12. Email-based Single sign-on where all email providers are identity providers Users verify their identity by demonstrating their ability to send or receive email Systems SAW Click on a link sent in an email message Hatchet Enter a code sent in an email message May 20, 2015WWW 2015, Florence, Italy12

13. QR Code-based Encodes authentication credentials into a QR code Two recent systems WebTicket Snap2Pass May 20, 2015WWW 2015, Florence, Italy13

14. QR Code-based Encodes authentication credentials into a QR code Two recent systems WebTicket Credentials encoded into a token that is printed out Token is shown to the website to authenticate the user May 20, 2015WWW 2015, Florence, Italy14

15. QR Code-based Encodes authentication credentials into a QR code Two recent systems WebTicket Credentials encoded into a token that is printed out Token is shown to the website to authenticate the user Snap2Pass The user’s phone acts as the identity provider The website sends information to the phone through QR codes May 20, 2015WWW 2015, Florence, Italy15

16. Methodology Four studies in total Federated single sign-on: 24 participants Email-based: 18 participants QR code-based: 25 participants Championship round: 30 participants Participants were from BYU Most were undergraduates Most were between 18 – 24 years old On average rated themselves as having intermediate technical skills May 20, 2015WWW 2015, Florence, Italy16

17. Study design Built two websites Forum website Bank website Implemented the seven authentication systems Existing implementations unavailable Consistent look and functionality Six tasks 2 registration tasks 4 authentication tasks Repeated same tasks for each system tested Questionnaire After each system After study as a whole May 20, 2015WWW 2015, Florence, Italy17

18. System Usability Scale Single numeric score between 0 and 100 (higher is better) Calculated based on user responses to 10 Likert-scale questions Individual participants’ SUS scores are averaged to give the overall SUS score May 20, 2015WWW 2015, Florence, Italy18

19. SUS Questions 1. I think that I would like to use this system frequently. 2. I found the system unnecessarily complex. 3. I thought the system was easy to use. 4. I think that I would need the support of a technical person to be able to use this system. 5. I found the various functions in this system were well integrated. 6. I thought there was too much inconsistency in this system. 7. I would imagine that most people would learn to use this system very quickly. 8. I found the system very cumbersome to use. 9. I felt very confident using the system. 10. I needed to learn a lot of things before I could get going with this system. May 20, 2015WWW 2015, Florence, Italy19

20. What Does the SUS Score Mean? If a system has a SUS score of 75, what does that mean? Bangor et al. examined SUS in over 200 usability studies and developed an adjective-oriented interpretation of a SUS score May 20, 2015WWW 2015, Florence, Italy20

21. Results: Federated Single Sign-on Winner: three way tie SUS scores between 71 and 72 Good Acceptable C grade Chose Google OAuth 2.0 to advance Mozilla Persona took longer to authenticate Difference was not mentioned in participants qualitative responses Trust issues with Google OAuth 2.0 and Facebook Connect May 20, 2015WWW 2015, Florence, Italy21 SystemSUS Score Google72.0 Facebook71.4 Mozilla71.8

22. Results: Email-based Winner: SAW Both systems performed poorly SAW OK Low-marginal acceptability D grade Participants disliked checking their email Hatchet OK Low-marginal acceptability F grade Users don’t want to leave their browser May 20, 2015WWW 2015, Florence, Italy22 SystemSUS Score SAW61.0 Hatchet53.5

23. Results: QR Code-based Winner: Snap2Pass WebTicket OK Low-marginal acceptability D grade Snap2Pass Good Acceptable B grade May 20, 2015WWW 2015, Florence, Italy23 SystemSUS Score WebTicket57.9 Snap2Pass75.7

24. Championship Round Participants: Google OAuth 2.0, SAW, Snap2Pass Google OAuth 2.0 and Snap2Pass tie SUS scores consistent with earlier scores Overall winners: Federated single sign-on Snap2Pass May 20, 2015WWW 2015, Florence, Italy24 SystemSUS Score Google75.0 SAW53.2 Snap2Pass68.4

25. Championship Round Participants were asked how the systems compared to each other and to passwords May 20, 2015WWW 2015, Florence, Italy25

26. System Usability Scale Repeatable results - consistent SUS scores between studies Good predictor of overall preference More accurate than mean time to authenticate Recommendation: All new proposals be evaluated using SUS New system proposals should not be seriously considered until they receive a score of at least 70 May 20, 2015WWW 2015, Florence, Italy26

27. Qualitative Feedback Users provided feedback via open-ended survey questions and in-person interviews The results provide interesting user perspectives on authentication May 20, 2015WWW 2015, Florence, Italy27

28. Transparency In usable security, transparency refers to hiding security details Transparency increases usability Tested this by modifying SAW to automate token retrieval Used participants from second usability study (email-based) Increased SUS score by 12.1 points Statistically significant difference (p=0.01) May 20, 2015WWW 2015, Florence, Italy28

29. Transparency Transparency can result in a lack of trust Similar phenomenon in our secure email research Confused Johnny: When Automatic Encryption Leads to Confusion and Mistakes Ruoti et al., SOUPS 2013 “I would like to understand more about how it works up-front. It doesn't feel secure.” “I think it was very straightforward to use. Once again like with the other system, perhaps an explanation of how it protected information would give me more confidence in using it.” May 20, 2015WWW 2015, Florence, Italy29

30. Single Sign-on Protocols Users liked the speed and convenience Users recognized the risk of putting all their eggs in one basket Suggested augmenting SSO with low-entropy passwords at the website Adds security if identity provider account is compromised May 20, 2015WWW 2015, Florence, Italy30

31. Single Sign-on Protocols Reputation of the identity provider is important Desire dedicated identity providers “I would be worried about security. I've heard that Facebook is ‘relatively’ easy to hack. I would want to be sure that it was all secure before I started using it.” “I trust Google with my passwords.” “I would make an account separate from my social network and mail specifically for functions like banking etc.” May 20, 2015WWW 2015, Florence, Italy31

32. The Coolness Factor Participants were most willing to adopt systems that they described as “cool” “Man was that cool!” “Also, the feel of it made me enjoy doing it. I felt technologically literate and the app felt futuristic as a whole, which I enjoyed.” May 20, 2015WWW 2015, Florence, Italy32

33. Biometrics We did not test or mention biometrics in our study Users consistently mentioned them as being a “cool” way to authenticate Indication that users may be accepting of viable biometric solutions “retinal scanner so i just sit in front of my computer and it scans my eye. dope.” “The ideal system would scan some part of my body - either eye or thumb - because these are literally ALWAYS with me.” May 20, 2015WWW 2015, Florence, Italy33

34. Conclusion We tested seven web authentication systems Found federated single sign-on and Snap2Pass to be the most usable First empirical analysis of a heterogeneous collection of authentication proposals System Usability Scale SUS is a good measure of usability for authentication proposals Repeatable results that allow for comparing heterogeneous systems Recommend it be used for all new authentication proposals Minimum score of 70 for serious consideration Future work Exploring the tradeoffs of transparency in authentication Low-entropy passwords with single sign-on Biometric-based web authentication May 20, 2015WWW 2015, Florence, Italy34

35. Questions? May 20, 2015WWW 2015, Florence, Italy35 Contact: seamons@cs.byu.edu

36. Questions?