Caleb Stepanian, Cindy Rogers, Nilesh Patel Confused Johnny: When Automatic Encryption Leads to Confusion and Mistakes Caleb Stepanian, Cindy Rogers, Nilesh Patel

Outline Background Information Who is Johnny? What is usability? What is Johnny's problem? How can we fix it? Results Conclusions

Who’s Johnny?

What is usability? Security software is usable if the people using it: Know the security tasks they need to perform Are able to figure out how to perform them Don’t make dangerous errors Are comfortable enough to continue use

Problem Statement Johnny finds it confusing to encrypt his emails. Email encryption (PGP) is not very usable chicken-and-egg problem lots of manual tools and background knowledge are needed

Hypothesis Johnny doesn’t encrypt because current solutions are not transparent enough.

Transparency To be considered transparent: 1. Cannot require too much effort 2. Must solve chicken-and-egg problem for keys 3. Handle encryption automatically, hiding cipher text

Experiment Have Johnny try transparent encryption and opaque encryption methods to determine his preferences.

Experimental Methodology Find a transparent system that meets criteria Find other more opaque solutions Run one user study for each other solution comparing it to the transparent one Find System Usability Scale (SUS) score for each Draw conclusions

System Usability Scale A set of ten questions that allows one to access the usability of a system on a sliding scale.

Experimental Setup Transparent: Pwm (Private Webmail) Browser extension that overlays automatic encryption over web mail Opaque: MP (Message Protector) Manual encryption with external program

Other Methods Tested Depot Base: Voltage(Voltage SecureMail Cloud) Sign up for an account and verify it. Generic: Encrypt.it (Bookmarklet) Allows you to encrypt the text in any field with a password.

Pwm Example

MP Example

Results Because people did not see the encryption happening, 10% of users didn’t encrypt their emails correctly & some users didn’t innately trust the system. Manual encryption (copy pasting while seeing the ciphertext) and clear separation gave users more confidence in the system.

Comparison Results: PWM v. MP 28 users tried both MP and Pwm Metric Percent users Pwm Study Percent users MP study Successful Decryption 86% 93% Successful Encryption 83% 97% Comprehension* 76% Intuitively decrypt 72% 100% Preferred System 41% *Correctly identifying who would be able to read encrypted messages

Study Results Pwm Usability Study PWM v. Voltage Preference Metric Successful Users out of 25 Setup Pwm 24 Successful Decryption Reply with Encrypted Message 23 Send Encrypted Message Direct 22 44% users reported Voltage was cumbersome to encrypt and decrypt a message 19% preferred Voltage

MP vs. Encipher.it Task 1: Install the given system Task 2: Open Gmail and send encrypted message, decrypt response Task 3: Open Facebook and send encrypted message, then decrypt reply System Task 2 Task 3 MP 89% 96% Encipher.it 57% / 50% 82% / 61%

Conclusions of MP vs. Encipher.it MP had a SUS score of 72.23 Encipher.it had a SUS score of 61.25 MP qualifies as “acceptable” Encipher.it ranks as “low marginal”

Conclusion Encryption needs to be somewhat manual so that users feel secure and know the difference between encryption and plaintext

Limitations User studies were short term lab studies First SUS question was “I think that I would like to use this system frequently”. First MP study assumed secrets were already shared Second MP study assumed Pwm was installed

Thank you! Any Questions?

PGP (Pretty Good Privacy) public and private keypairs private key needed to sign and decrypt public key needed to encrypt and verify signature A user needs to generate a keypair and share their public key before an encrypted message can be sent to them

Key escrow server Trusted third party that generates and stores key material for users Has ability to read all messages and masquerade as any user

Example SUS Survey Choose from 1 (strongly disagree) to 5 (strongly agree). I think that I would like to use this system frequently I found the system unnecessarily complex I thought the system was easy to use I think that I would need the support of a technical person to be able to use this system I found the various functions in this system were well integrated I thought there was too much inconsistency in this system I found the system very cumbersome to use I would imagine that most people would learn to use this system very quickly I felt very confident using the system I needed to learn a lot of things before I could get going with this system