1. Caleb Stepanian, Cindy Rogers, Nilesh Patel
Confused Johnny: When Automatic Encryption Leads to Confusion and Mistakes
Caleb Stepanian, Cindy Rogers, Nilesh Patel

2. Outline Background Information Who is Johnny? What is usability?
What is Johnny's problem?
How can we fix it?
Results
Conclusions

3. Who’s Johnny?

4. What is usability? Security software is usable if the people using it:
Know the security tasks they need to perform
Are able to figure out how to perform them
Don’t make dangerous errors
Are comfortable enough to continue use

5. Problem Statement Johnny finds it confusing to encrypt his emails.
encryption (PGP) is not very usable
chicken-and-egg problem
lots of manual tools and background knowledge are needed

6. Hypothesis
Johnny doesn’t encrypt because current solutions are not transparent enough.

7. Transparency To be considered transparent:
1. Cannot require too much effort
2. Must solve chicken-and-egg problem for keys
3. Handle encryption automatically, hiding cipher text

8. Experiment
Have Johnny try transparent encryption and opaque encryption methods to determine his preferences.

9. Experimental Methodology
Find a transparent system that meets criteria
Find other more opaque solutions
Run one user study for each other solution comparing it to the transparent one
Find System Usability Scale (SUS) score for each
Draw conclusions

10. System Usability Scale
A set of ten questions that allows one to access the
usability of a system on a sliding scale.

11. Experimental Setup Transparent: Pwm (Private Webmail)
Browser extension that overlays automatic encryption over web mail
Opaque: MP (Message Protector)
Manual encryption with external program

12. Other Methods Tested Depot Base: Voltage(Voltage SecureMail Cloud)
Sign up for an account and verify it.
Generic: Encrypt.it (Bookmarklet)
Allows you to encrypt the text in any field with a password.

13. Pwm Example

14. MP Example

15. Results
Because people did not see the encryption happening, 10% of users didn’t encrypt their s correctly & some users didn’t innately trust the system.
Manual encryption (copy pasting while seeing the ciphertext) and clear separation gave users more confidence in the system.

16. Comparison Results: PWM v. MP
28 users tried both MP and Pwm
Metric
Percent users Pwm Study
Percent users MP study
Successful Decryption
86%
93%
Successful Encryption
83%
97%
Comprehension*
76%
Intuitively decrypt
72%
100%
Preferred System
41%
*Correctly identifying who would be able to read encrypted messages

17. Study Results Pwm Usability Study PWM v. Voltage Preference Metric
Successful Users out of 25
Setup Pwm 24
Successful Decryption
Reply with Encrypted Message 23
Send Encrypted Message Direct 22
44% users reported Voltage was cumbersome to encrypt and decrypt a message
19% preferred Voltage

18. MP vs. Encipher.it Task 1: Install the given system
Task 2: Open Gmail and send encrypted message, decrypt response
Task 3: Open Facebook and send encrypted message, then decrypt reply
System
Task 2
Task 3 MP
89%
96%
Encipher.it
57% / 50%
82% / 61%

19. Conclusions of MP vs. Encipher.it
MP had a SUS score of Encipher.it had a SUS score of MP qualifies as “acceptable” Encipher.it ranks as “low marginal”

20. Conclusion
Encryption needs to be somewhat manual so that users feel secure and know the difference between encryption and plaintext

21. Limitations User studies were short term lab studies
First SUS question was “I think that I would like to use this system frequently”.
First MP study assumed secrets were already shared Second MP study assumed Pwm was installed

22. Thank you!
Any Questions?

23. PGP (Pretty Good Privacy)
public and private keypairs
private key needed to sign and decrypt
public key needed to encrypt and verify signature
A user needs to generate a keypair and share their public key before an encrypted message can be sent to them

24. Key escrow server
Trusted third party that generates and stores key material for users
Has ability to read all messages and masquerade as any user

25. Example SUS Survey
Choose from 1 (strongly disagree) to 5 (strongly agree).
I think that I would like to use this system frequently
I found the system unnecessarily complex
I thought the system was easy to use
I think that I would need the support of a technical person to be able to use this system
I found the various functions in this system were well integrated
I thought there was too much inconsistency in this system
I found the system very cumbersome to use
I would imagine that most people would learn to use this system very quickly
I felt very confident using the system
I needed to learn a lot of things before I could get going with this system