With your instructor, Jeremy Hyland User Studies II With your instructor, Jeremy Hyland

Plan for Today Discuss the reading: Do a little testing of our own… Why Johnny Can’t Encrypt Johnny 2: Judgment Day Do a little testing of our own…

Why Johnny Can’t Encrypt Who’s Johnny and why can’t he encrypt? Posner says What’s Johnny trying to hide?

Why Johnny Can’t Encrypt Whitten and Tygar, 1999 http://www.usenix.org/publications/library/proceedings/sec99/full_papers/whitten/whitten_html/index.html A Usability Evaluation of PGP 5.0

Why Johnny Can’t Encrypt “Security mechanisms are only effective when used correctly” So: If Usable then else

Why Johnny Can’t Encrypt Defining Usable Security Software Whitten and Tygar: Security software is usable if the people who are expected to use it: are reliably made aware of the security tasks they need to perform. are able to figure out how to successfully perform those tasks don't make dangerous errors are sufficiently comfortable with the interface to continue using it.

Why Johnny Can’t Encrypt Why is usable security hard? McNealy says You have no usable security, get over it.

Why Johnny Can’t Encrypt Why is usable security hard? Five reasons: 1. The unmotivated users “Security is usually a secondary goal” 2. Policy Abstraction Programmers understand the representation but normal users have no background knowledge.

Why Johnny Can’t Encrypt Why is usable security hard? Five reasons: 3. The lack of feedback We can’t predict every situation. 4. The proverbial “barn door” Need to focus on error prevention. 5. The weakest link Attacker only needs to find one vulnerability

Why Johnny Can’t Encrypt Usability Evaluation PGP 5.0 Pretty Good Privacy Software for encrypting and signing data Plug-in provides “easy” use with email clients Modern GUI, well designed by most standards

Why Johnny Can’t Encrypt Usability Evaluation Whitten and Tygar focus their evaluation on a question based off their definition of usable secure software: If an average user of email feels the need for privacy and authentication, and acquires PGP with that purpose in mind, will PGP's current design allow that person to realize what needs to be done, figure out how to do it, and avoid dangerous errors, without becoming so frustrated that he or she decides to give up on using PGP after all? Loaded question?

Why Johnny Can’t Encrypt Usability Evaluation Cognitive walk through Mentally step through the software as if we were a new user. Attempt to identify the usability pitfalls. Focus on interface learnablity.

Why Johnny Can’t Encrypt Usability Evaluation Cognitive walk through results: Visual metaphors Public vs. Private keys Signatures and verification

Why Johnny Can’t Encrypt Usability Evaluation Cognitive walk through results: Different key types Compatibility increases complexity Keys listed as users

Why Johnny Can’t Encrypt Keys listed as users

Why Johnny Can’t Encrypt Usability Evaluation Cognitive walk through results: Key server Hidden? What is it doing? Revocation not automatic Would that help?

Why Johnny Can’t Encrypt Usability Evaluation Cognitive walk through results: Key management policy Unneeded confusion What’s the difference between trust and validity?

Why Johnny Can’t Encrypt Usability Evaluation Cognitive walk through results: Irreversible actions Need to prevent costly errors Consistency “Encoding”?!? Too much information More unneeded confusion Show the basic information, make more advanced information available only when needed.

Why Johnny Can’t Encrypt Usability Evaluation User Test PGP 5.0 with Eudora 12 participants all with at least some college and none with advanced knowledge of encryption Participants were given a scenario with tasks to complete within 90 min Tasks built on each other Participants could ask some questions through email

Why Johnny Can’t Encrypt Usability Evaluation User Test Results: 3 users accidentally sent the message in clear text 7 users used their public key to encrypt and only 2 of the 7 figured out how to correct the problem Only 2 users were able to decrypt without problems Only 1 user figured out how to deal with RSA keys correctly. A total of 3 users were able to successfully complete the basic process of sending and receiving encrypted emails. One user was not able to encrypt at all

Why Johnny Can’t Encrypt Conclusion If an average user of email feels the need for privacy and authentication, and acquires PGP with that purpose in mind, will PGP's current design allow that person to realize what needs to be done, figure out how to do it, and avoid dangerous errors, without becoming so frustrated that he or she decides to give up on using PGP after all? Nope Is this a failure in the design of the PGP 5.0 interface or is it a function of the problem of traditional usable design vs. design for usable secure systems? Security as the primary function vs. a secondary function

Johnny 2 Garfinkel and Miller, 2005 http://www.simson.net/clips/academic/2005.SOUPS.johnny2.pdf Follow-up to Why “Johnny Can’t encrypt” Test of new encryption technology Key Continuity Management S/MIME certificates Better interface Simple buttons

Johnny 2 Garfinkel and Miller: Johnny couldn’t encrypt because of the key architecture behind PGP. “….the fundamental usability barriers that Whitten identified could be overcome by replacing the underlying third-party certification model with Key Continuity Management.”

Johnny 2 User Test Tried to stay as close to the Johnny experiment as practical Same methods of user solicitation/selection Same basic scenario Similar user tasks Added attackers

Johnny 2 User Test Attacks: new key attack new identity attack unsigned message attack How well does the interface enable users to respond to these attacks?

Johnny 2 User Test Test application: CoPilot “Wizard of Oz” prototype S/MIME certificate handling: First time = Yellow Trusted certificate = Green Changed certificate = Red Unsigned message = White Unsigned message from a sender that normal sends signed messages = Gray Better tools allow for a more automated and scientific test

Johnny 2 User Test 43 test subjects Three groups: No KCM Color Color+Briefing

Johnny 2 User Test Results: Users generally understood the basics Little understanding of signature integrity guarantees Verifying attack message authenticity was difficult for most users No group resisted attacks 100% of the time Color and Color+Briefing resisted new key attack and the unsigned message attack better then No KCM The interface did not help against new identity attacks

Johnny 2 User Test: Conclusions A few surface interface issues Do not trust button Misconceptions about the security of sealed messages Generally, the new interface simplifies email encryption Still problems with determining certificate trust, however some of these problems may be unavoidable.

So Now What? Now its time to do your own test!

User Test 3 groups: Take a few minutes to create a simple user test Cell Phone CD player Calculator Take a few minutes to create a simple user test One member of each group switches to be a tester

User Test Guidance: Decide whose going to do what! Create a Use Case Scenario Define user tasks for completion of the scenario Set up metrics for results evaluation What qualifies as success vs. failure?

User Test Results!?