2. “We’re on the Same Page”: A Usability Study of Secure Email Using Pairs of Novice Users Scott Ruoti, Jeff Andersen, Scott Heidbrink, Mark O'Neill, Elham Vaziripour, Justin Wu, Daniel Zappala, Kent Seamons Internet Security Research Lab Brigham Young University https://isrl.byu.edu 34th Annual ACM Conference on Human Factors in Computing Systems (CHI ‘16)

3. Email Security  Email was built without security  Security has since been bolted on  Email is still often insecure  In transit and at rest  Email was built without security  Security has since been bolted on  Email is still often insecure  In transit and at rest

4. End-to-end Encryption Alice Bob Email Sever Mallory

5. What’s the Hold Up?  Secure email systems have largely been unusable  Only viable in corporate settings  How do we get the masses to adopt secure email?  We’ve mostly relied on expert-led adoption  Can email be adopted in a grassroots style?  No previous study has examined this question  Secure email systems have largely been unusable  Only viable in corporate settings  How do we get the masses to adopt secure email?  We’ve mostly relied on expert-led adoption  Can email be adopted in a grassroots style?  No previous study has examined this question

6. Methodology  Brought in pairs of novice participants  Friends  Johnny emailed Jane asking for help with his taxes  Johnny told to encrypt his email and given the URL for a secure email system  Jane was told to wait for an email from Johnny  Johnny and Jane had to collaboratively set up and use secure email  Brought in pairs of novice participants  Friends  Johnny emailed Jane asking for help with his taxes  Johnny told to encrypt his email and given the URL for a secure email system  Jane was told to wait for an email from Johnny  Johnny and Jane had to collaboratively set up and use secure email

7. Pwm Integrates with Gmail Identity-based encryption

8. Tutanota Email depot PGP and password- based encryption

9. Virtru Integrated and email depot Custom key escrow scheme

10. Results  Within subjects  25 participant pairs  50 total participants  Quantitative metrics  Qualitative feedback  Within subjects  25 participant pairs  50 total participants  Quantitative metrics  Qualitative feedback

11. System Usability Scale

12. PwmTutanotaVirtru

13. Task Completion Time

15. Mistakes  No mistakes with Pwm  One mistake with Virtru  Many mistakes with Tutanota  Two-third (68%, n=17) sent password through regular email  Half (48%, n=12) selected easily guessed passwords  No mistakes with Pwm  One mistake with Virtru  Many mistakes with Tutanota  Two-third (68%, n=17) sent password through regular email  Half (48%, n=12) selected easily guessed passwords

16. Favorite System

17. Paired-participants  Two novices  Struggled using Tutanota’s password-based encryption  Two perspectives  Jane strongly preferred Pwm  More natural behavior  Relaxed during the study  Willing to believe other side made mistakes  Two novices  Struggled using Tutanota’s password-based encryption  Two perspectives  Jane strongly preferred Pwm  More natural behavior  Relaxed during the study  Willing to believe other side made mistakes

18. Quotes “I thought it was good, I dunno, might’ve taken the pressure off too, where it’s like, ‘Okay, he’s figuring this out too’, so I can just, y’know, I don’t have to feel as ‘under-the-microscope’ in the study.”

19. Quotes “...I was more at ease probably than I would’ve been if it was someone random on the other end...It would’ve felt more mechanical, robotic, whereas I know [her] and I was calling my wife, ‘Hi wife! What’s the password?’ It felt a lot more personable for me I think....”

20. Quotes “It was good in that you saw the troubles, like the third system [Tutanota], I didn’t even know how it worked, so I ended up sending an email to myself on Gmail so then I could see what was happening on her end, to know like how it works on the other end. So I think it’s good to have two people on each end that don’t know what’s going on, because if it weren’t I’d assume the person on the other side had done it before...”

21. Other Lessons Learned  Hiding security details leads to a lack of trust  Participants prefer integrated secure email  Integrated tutorials are essential  Users are interested in secure email  Hiding security details leads to a lack of trust  Participants prefer integrated secure email  Integrated tutorials are essential  Users are interested in secure email

22. Summary  Paired participant studies are helpful  Assesses the usability of grassroots adoption  Allows for interesting interactions between two novice users  Leads to more natural participant behavior  We are getting closer to usable, secure email for the masses  Paired participant studies are helpful  Assesses the usability of grassroots adoption  Allows for interesting interactions between two novice users  Leads to more natural participant behavior  We are getting closer to usable, secure email for the masses

23. Questions