1. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 1 Understanding the Human in the Loop January 16, 2008

2. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 2 Humans “Humans are incapable of securely storing high- quality cryptographic keys, and they have unacceptable speed and accuracy when performing cryptographic operations. (They are also large, expensive to maintain, difficult to manage, and they pollute the environment. It is astonishing that these devices continue to be manufactured and deployed. But they are sufficiently pervasive that we must design our protocols around their limitations.)” -- C. Kaufman, R. Perlman, and M. Speciner. Network Security: PRIVATE Communication in a PUBLIC World. 2nd edition. Prentice Hall, page 237, 2002.

3. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 3 Humans are weakest link Most security breaches attributed to “human error” Social engineering attacks proliferate Frequent security policy compliance failures Automated systems are generally more predictable and accurate than humans

4. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 4 Why are humans in the loop at all? Don’t know how or too expensive to automate Human judgments or policy decisions needed Need to authenticate humans

5. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 5 The human threat Malicious humans who will attack system Humans who don’t know when or how to perform security-critical tasks Humans who are unmotivated to perform security-critical tasks properly or comply with policies Humans who are incapable of making sound security decisions

6. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 6 Need to better understand humans Do they know they are supposed to be doing something? Do they understand what they are supposed to do? Do they know how to do it? Are they motivated to do it? Are they capable of doing it? Will they actually do it?

7. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 7 Proposed framework Cranor interactions article: What do they "indicate?": evaluating security and privacy indicators The Handbook of Warnings, edited by Michael S. Wogalter Wogalter’s Communication-Human Information Processing (C-HIP) Model Applied C-HIP to security indicators evaluation from interactions article Expanded it to model other types of human interaction with secure systems Developed “Human in the loop security framework” and “Human threat identification and mitigation process” - paper under review Need validation and more work on mitigation and how to operationalize process

8. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 8 C-HIP Model Communication- Human Information Processing (C-HIP) Model Wogalter, M. 2006. Communication- Human Information Processing (C-HIP) Model. In Wogalter, M., ed., Handbook of Warnings. Lawrence Erlbaum Associates, Mahwah, NJ, 51-61.

9. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 9 Human in the loop security framework Human Receiver Communication Processing Application Communication Delivery Intentions Attention Switch Attention Maintenance Comprehension Knowledge Retention Knowledge Transfer Motivation Attitudes and Beliefs Knowledge Acquisition Communication Behavior Personal Variables Knowledge and Experience Demographics and Personal Characteristics Communication Impediments Interference Environmental Stimuli Capabilities

10. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 10 Communication processing model Framework is based on communication processing model Many models in the literature Used to model all sorts of different types of communications: individual, group, media, etc. Most end-user security actions are triggered by some form of communication Pop-up alert, email, manual, etc. Expert self-discovery of a security process can be modeled as communication to oneself

11. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 11 Communication Human Receiver Communication Processing Application Communication Delivery Intentions Attention Switch Attention Maintenance Comprehension Knowledge Retention Knowledge Transfer Motivation Attitudes and Beliefs Knowledge Acquisition Communication Behavior Personal Variables Knowledge and Experience Demographics and Personal Characteristics Communication Impediments Interference Environmental Stimuli Capabilities

12. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 12 Types of security communications Warnings Alert users to take immediate action to avoid hazard Notices Inform users about characteristics of entity or object Status indicators Inform users about system status information Training Teach users about threat and how to respond Policy Inform users about policies

13. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 13 Active versus passive communications Firefox Anti-Phishing Warning ActivePassive Bluetooth indicator in Mac menu bar Indicators with audio alerts Indicators with animation

14. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 14 Communication impediments Human Receiver Communication Processing Application Communication Delivery Intentions Attention Switch Attention Maintenance Comprehension Knowledge Retention Knowledge Transfer Motivation Attitudes and Beliefs Knowledge Acquisition Communication Behavior Personal Variables Knowledge and Experience Demographics and Personal Characteristics Communication Impediments Interference Environmental Stimuli Capabilities

15. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 15 Environmental stimuli Divert user’s attention Greatest impact on passive communication Examples Other communications Ambient light and noise User’s primary task

16. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 16 Interference Anything that may prevent a communication from being received as the sender intended Caused by Malicious attackers Technology failures Environmental stimuli that obscure the communication Focus of traditional secure systems analysis How can attacker interfere with communications?

17. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 17 Human receiver Human Receiver Communication Processing Application Communication Delivery Intentions Attention Switch Attention Maintenance Comprehension Knowledge Retention Knowledge Transfer Motivation Attitudes and Beliefs Knowledge Acquisition Communication Behavior Personal Variables Knowledge and Experience Demographics and Personal Characteristics Communication Impediments Interference Environmental Stimuli Capabilities “The human in the loop”

18. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 18 Communication delivery Attention switch Noticing communication Attention maintenance Paying attention long enough to process communication Breakdowns Environmental stimuli, interference Characteristics of communication Habituation  Tendency for the impact of stimuli to decrease over time Just because the communication appeared on the user’s screen, doesn’t mean the user actually saw it

19. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 19 Communication processing Comprehension Ability to understand communication Knowledge acquisition User’s ability to learn what to do in response Breakdowns Unfamiliar symbols, vocabulary, complex sentences, conceptual complexity Even if a user understands the communication, they still may not know what they are supposed to do

20. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 20 Application Knowledge retention Ability to remember communication Knowledge transfer Ability to recognize situations where the communication is applicable and figure out how to apply it Some security communications are always applied immediately (for example, pop-up warnings) so retention and transfer may not be necessary

21. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 21 Personal variables Demographics and personal characteristics Age, gender, culture, education, occupation, disabilities Knowledge and experience Education, occupation, prior experience

22. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 22 Intentions Attitudes and beliefs Beliefs about communication accuracy Beliefs about whether they should pay attention Self-efficacy - whether they believe they can complete actions effectively Response-efficacy - whether they believe the actions they take will be effective How long it will take General attitudes - trust, annoyance, etc. Motivation Incentives, disincentives

23. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 23 Capabilities User’s level of ability Cognitive or physical skills Availability of necessary software or devices

24. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 24 Behavior Human Receiver Communication Processing Application Communication Delivery Intentions Attention Switch Attention Maintenance Comprehension Knowledge Retention Knowledge Transfer Motivation Attitudes and Beliefs Knowledge Acquisition Communication Behavior Personal Variables Knowledge and Experience Demographics and Personal Characteristics Communication Impediments Interference Environmental Stimuli Capabilities

25. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 25 Behavior Users may complete recommended action, but do so in a way that follows a predictable pattern that can be exploited by attackers Example: password choice Users may intend to comply, but may fail to complete necessary action

26. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 26 Gulfs Don Norman. The Design of Every Day Things.1988. Gulf of Execution Gap between a person’s intentions to carry out an action and the mechanisms provided by a system to facilitate that action  “I can’t figure out how to make it do what I want it to do” Gulf of Evaluation When a user completes an action but is unable to interpret the results to determine whether it was successful  “I can’t figure out whether it worked”

27. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 27 Generic Error-Modeling System James Reason. Human Error. 1990. Mistakes When people formulate action plans that will not achieve the desired goal Lapses When people formulate suitable action plans, but forget to perform a planned action (for example, skipping a step) Slips When people perform actions incorrectly (for example, press the wrong button)

28. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 28 Human threat identification and mitigation process Task Identification Task Automation Failure Identification Failure Mitigation Human-in-the-loop Framework User Studies Task identification Identify all points where the system relies on humans to perform security- critical functions Task automation Find ways to partially or fully automate some of these tasks Failure identification Identify potential failure modes for remaining tasks Failure mitigation Find ways to prevent these failures

29. Why don’t users follow password policies?

30. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 30 Typical password policy Pick a hard to guess password Don’t use it anywhere else Change it often Don’t write it down

31. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 31 Typical password practice Bank = b3aYZ Amazon = aa66x! Phonebill = p$2$ta1

32. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 32 Why don’t users follow password policies? Task Identification Task Automation Failure Identification Failure Mitigation Human-in-the-loop Framework User Studies

33. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 33 Why don’t users follow password policies? Human Receiver Communication Processing Application Communication Delivery Intentions Attention Switch Attention Maintenance Comprehension Knowledge Retention Knowledge Transfer Motivation Attitudes and Beliefs Knowledge Acquisition Communication Behavior Personal Variables Knowledge and Experience Demographics and Personal Characteristics Communication Impediments Interference Environmental Stimuli Capabilities

34. Why don’t user’s heed browser security warnings?

35. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 35 Do users notice them? “What lock icon?” Few users notice lock icon in browser chrome, https, etc.

36. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 36 Do users know what they mean? Web browser lock icon: “I think that it means secured, it symbolizes some kind of security, somehow.” Web browser security pop-up: “Yeah, like the certificate has expired. I don’t actually know what that means.” J. Downs, M. Holbrook, and L. Cranor. Decision Strategies and Susceptibility to Phishing. In Proceedings of the 2006 Symposium On Usable Privacy and Security, 12-14 July 2006, Pittsburgh, PA.

37. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 37 Do they do what they advise? “I would probably experience some brief, vague sense of unease and close the box and go about my business.”

38. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 38 Why don’t users heed browser security warnings? Task Identification Task Automation Failure Identification Failure Mitigation Human-in-the-loop Framework User Studies

39. Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor http://cups.cs.cmu.edu/courses/ups.html/ 39 Why don’t users heed browser security warnings? Human Receiver Communication Processing Application Communication Delivery Intentions Attention Switch Attention Maintenance Comprehension Knowledge Retention Knowledge Transfer Motivation Attitudes and Beliefs Knowledge Acquisition Communication Behavior Personal Variables Knowledge and Experience Demographics and Personal Characteristics Communication Impediments Interference Environmental Stimuli Capabilities