C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate, Carnegie Mellon University Presented at IIS seminar, 1/30/2008

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Everyday Privacy and Security Problem 2

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images This entire process known as phishing 3

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Still a growing problem Estimated 1 in 122 s are phishing Average 31,000 unique phishing sites reported each month in 2007 Estimated 3.5 million people have fallen for phishing in 2006 Estimated $ 350m – $ 2b direct loss a year More profitable to phish than rob the bank! 4

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Project: Supporting Trust Decisions Goal: help people make better online trust decisions Currently focusing on anti-phishing Large multi-disciplinary team project at CMU Computer science, human-computer interaction, public policy, social and decision sciences, CERT

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Our Multi-Pronged Approach Human side Interviews to understand decision-making PhishGuru embedded training Anti-Phishing Phil game Understanding effectiveness of browser warnings Computer side PILFER anti-phishing filter CANTINA web anti-phishing algorithm Automate where possible, support where necessary

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Our Multi-Pronged Approach Human side Interviews to understand decision-making PhishGuru embedded training Anti-Phishing Phil game Understanding effectiveness of browser warnings Computer side PILFER anti-phishing filter CANTINA web anti-phishing algorithm What do users know about phishing?

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Interview Study Interviewed 40 Internet users (35 non-experts) “Mental models” interviews included role play and open ended questions Brief overview of results (see paper for details) J. Downs, M. Holbrook, and L. Cranor. Decision Strategies and Susceptibility to Phishing. In Proceedings of the 2006 Symposium On Usable Privacy and Security, July 2006, Pittsburgh, PA.

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Little knowledge of phishing Only about half knew the meaning of term “phishing” 55% say that they had never noticed an unexpected or strange-looking URL 55% reported being cautious when asked for sensitive financial information But very few reported being suspicious of asking for passwords Knowledge of financial phish reduced likelihood of falling for these scams But did not transfer to other scams, such as an amazon.com password phish 9

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Naive Evaluation Strategies The most frequent strategies don’t help much in identifying phish This appears to be for me It’s normal to hear from companies you do business with Reputable companies will send s “I will probably give them the information that they asked for. And I would assume that I had already given them that information at some point so I will feel comfortable giving it to them again.”

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Summary of Findings People generally not good at identifying scams they haven’t specifically seen before People don’t use good strategies to protect themselves Large-scale survey across multiple cities in the US confirm finding Downs, J. S., Holbrook, M. B., and Cranor, L. F. Behavioral Response to Phishing. In eCrime ’07: Proceedings of the 2007 e-Crime Researchers summit

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Outline Human side Interviews to understand decision-making PhishGuru embedded training Anti-Phishing Phil game Understanding effectiveness of browser warnings Computer side PILFER anti-phishing filter CANTINA web anti-phishing algorithm Can we train people not to fall for phish?

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Web Site Training Study Laboratory study of 28 non-expert computer users Asked participants to evaluate 20 web sites Control group evaluated 10 web sites, took 15 min break to read or play solitaire, evaluated 10 more web sites Experimental group same as above, but spent 15 min break reading web-based training materials Experimental group performed significantly better identifying phish after training Less reliance on “professional-looking” designs Looking at and understanding URLs Web site asks for too much information People can learn from web-based training materials, if only we could get them to read them!

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images How Do We Get People Trained? Most people don’t proactively look for training materials on the web Companies send “security notice” s to employees and/or customers We hypothesized these tend to be ignored Too much to read People don’t consider them relevant People think they already know how to protect themselves Led us to idea of embedded training

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Embedded Training Can we “train” people during their normal use of to avoid phishing attacks? Periodically, people get sent a training Training looks like a phishing attack If person falls for it, intervention warns and highlights what cues to look for in succinct and engaging format P. Kumaraguru, Y. Rhee, A. Acquisti, L. Cranor, J. Hong, and E. Nunge. Protecting People from Phishing: The Design and Evaluation of an Embedded Training System. CHI 2007.

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Subject: Revision to Your Amazon.com Information Please login and enter your information Embedded training example

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Intervention #1 – Diagram

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Intervention #1 – Diagram Explains why they are seeing this message

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Intervention #1 – Diagram Explains what a phishing scam is

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Intervention #1 – Diagram Explains how to identify a phishing scam

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Intervention #1 – Diagram Explains simple things you can do to protect self

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Intervention #2 – Comic Strip

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Summary of Evaluation Results Study setup: Role play as Bobby Smith at Cognix Inc going through companies s 10 participants in each condition, screened for novice Evaluation I: Lab study comparing our prototypes to standard security notices Existing practice of security notices is ineffective Embedded training is effective Comic strip intervention worked best Evaluation II: Have to fall for phishing to be effective? How well do people retain knowledge?

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Results of Evaluation #2 Have to fall for phishing to be effective? How well do people retain knowledge after a week?

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Results of Evaluation #2 Have to fall for phishing to be effective? How well do people retain knowledge after a week? Correctness

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Results of Evaluation #2 Have to fall for phishing to be effective? How well do people retain knowledge after a week? Correctness

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Anti-Phishing Phil A game to teach people not to fall for phish Embedded training focuses on Our game focuses on web browser Goals How to parse URLs Where to look for URLs Use search engines for help S. Sheng, B. Magnien, P. Kumaraguru, A. Acquisti, L. Cranor, J. Hong, and E. Nunge. Anti-Phishing Phil: The Design and Evaluation of a Game That Teaches People Not to Fall for Phish. In Proceedings of the 2007 Symposium On Usable Privacy and Security, Pittsburgh, PA, July 18-20,

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Anti-Phishing Phil

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Summary of Evaluation of Anti-Phishing Phil Test participants’ ability to identify phishing web sites before and after training up to 15 min 10 web sites before training, 10 after, randomized order Evaluation I: Lab study How do Phil perform with existing training materials? Evaluation II: Online study How well do people retain what they learned?

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Results Phil had the best performance overall, with lowest false positives Novice users improve by 47%, intermediate users by 25% People remembered what they learned one week after the training Over 52,000 people played the game in the last three months 35

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Game results 36

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images 37

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Teaching users about phishing attacks can be a reality!

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Outline Human side Interviews to understand decision-making PhishGuru embedded training Anti-Phishing Phil game Understanding effectiveness of browser warnings Computer side PILFER anti-phishing filter CANTINA web anti-phishing algorithm Do people see, understand, and believe web browser warnings?

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Screenshots Internet Explorer – Passive Warning

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Screenshots Internet Explorer – Active Block

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Screenshots Mozilla FireFox – Active Block

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images How Effective are these Warnings? Tested four conditions FireFox Active Block IE Active Block IE Passive Warning Control (no warnings or blocks) “Shopping Study” Setup some fake phishing pages and added to blacklists Users were phished after purchases Real accounts and personal information Spoofing eBay and Amazon (2 phish/user) We observed them interact with the warnings

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images How Effective are these Warnings?

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images How Effective are these Warnings?

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Discussion of Phish Warnings Nearly everyone will fall for highly contextual phish Passive IE warning failed for many reasons Didn’t interrupt the main task Slow to appear (up to 5 seconds) Not clear what the right action was Looked too much like other ignorable warnings (habituation) Bug in implementation, any keystroke dismisses

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Screenshots Internet Explorer – Passive Warning

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Discussion of Phish Warnings Active IE warnings Most saw but did not believe it  “Since it gave me the option of still proceeding to the website, I figured it couldn’t be that bad” Some element of habituation (looks like other warnings) Saw two pathological cases Egelman, S, Cranor, L, Hong, J. You’ve been Warned. In CHI 2008.

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Screenshots Internet Explorer – Active Block

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Outline Human side Interviews to understand decision-making PhishGuru embedded training Anti-Phishing Phil game Understanding effectiveness of browser warnings Computer side PILFER anti-phishing filter CANTINA web anti-phishing algorithm Can we automatically detect phish s?

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images PILFER Anti-Phishing Filter Philosophy: automate where possible, support where necessary Goal: Create filter that detects phishing s Spam filters well-explored, but how good for phishing? Can we create a custom filter for phishing? I. Fette, N. Sadeh, A. Tomasic. Learning to Detect Phishing s. In W W W 2007.

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images PILFER Anti-Phishing Filter Heuristics combined in SVM IP addresses in link ( Age of linked-to domains (younger domains likely phishing) Non-matching URLs (ex. most links point to PayPal) “Click here to restore your account” HTML Number of links Number of domain names in links Number of dots in URLs ( JavaScript SpamAssassin rating

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images PILFER Evaluation Ham corpora from SpamAssassin (2002 and 2003) 6950 good s Phishingcorpus 860 phishing s

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images PILFER Evaluation

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images PILFER Evaluation PILFER now implemented as SpamAssassin filter

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Outline Human side Interviews to understand decision-making PhishGuru embedded training Anti-Phishing Phil game Understanding effectiveness of browser warnings Computer side PILFER anti-phishing filter CANTINA web anti-phishing algorithm How good is phish detection for web sites? Can we do better?

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Lots of Phish Detection Algorithms Dozens of anti-phishing toolbars offered Built into security software suites Offered by ISPs Free downloads (132 on download.com) Built into latest version of popular web browsers

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Lots of Phish Detection Algorithms Dozens of anti-phishing toolbars offered Built into security software suites Offered by ISPs Free downloads (132 on download.com) Built into latest version of popular web browsers But how well do they detect phish? Short answer: still room for improvement

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Testing the Toolbars November 2006: Automated evaluation of 10 toolbars Used phishtank.com and APWG as source of phishing URLs Evaluated 100 phish and 510 legitimate sites Y. Zhang, S. Egelman, L. Cranor, J. Hong. Phinding Phish: An Evaluation of Anti-Phishing Toolbars. NDSS 2006.

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Testbed System Architecture

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Results 38% false positives 1% false positives PhishTank

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Results Only one toolbar >90% accuracy (but high false positives) Several catch 70-85% of phish with few false positives

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Results Only one toolbar >90% accuracy (but high false positives) Several catch 70-85% of phish with few false positives Can we do better? Can we use search engines to help find phish? Y. Zhang, J. Hong, L. Cranor. CANTINA: A Content- Based Approach to Detecting Phishing Web Sites. In W W W 2007.

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Robust Hyperlinks Developed by Phelps and Wilensky to solve “404 not found” problem Key idea was to add a lexical signature to URLs that could be fed to a search engine if URL failed Ex. How to generate signature? Found that TF-IDF was fairly effective Informal evaluation found five words was sufficient for most web pages

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Adapting TF-IDF for Anti- Phishing Can same basic approach be used for anti-phishing? Scammers often directly copy web pages With Google search engine, fake should have low page rank FakeReal

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images How CANTINA Works Given a web page, calculate TF-IDF score for each word in that page Take five words with highest TF-IDF weights Feed these five words into a search engine (Google) If domain name of current web page is in top N search results, we consider it legitimate N=30 worked well No improvement by increasing N Later, added some heuristics to reduce false positives

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Fake eBay, user, sign, help, forgot

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Real eBay, user, sign, help, forgot

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Evaluating CANTINA PhishTank

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Weaknesses in CANTINA Bad guys may try to subvert search engines Only works if legitimate page is indexed Intranets May be confused if same login page in multiple places

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Summary Whirlwind tour of our work on anti-phishing Human side: how people make decisions, training, UIs Computer side: better algorithms for detecting phish More info about our work at cups.cs.cmu.edu

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Acknowledgments Alessandro Acquisti Lorrie Cranor Sven Dietrich Julie Downs Mandy Holbrook Norman Sadeh Anthony Tomasic Umut Topkara Supported by NSF, ARO, CyLab, Portugal Telecom Serge Egelman Ian Fette Ponnurangam Kumaraguru Bryant Magnien Elizabeth Nunge Yong Rhee Steve Sheng Yue Zhang

C MU U sable P rivacy and S ecurity Laboratory Steve Sheng Engineering and Public Policy

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Is it phish? Our label YesNo YesTrue positiveFalse negative NoFalse positiveTrue negative

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Minimal Knowledge of Lock Icon “I think that it means secured, it symbolizes some kind of security, somehow.” 85% of participants were aware of lock icon Only 40% of those knew that it was supposed to be in the browser chrome Only 35% had noticed https, and many of those did not know what it meant

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Solution Space Phishing Legal Social Technical 83

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Phishing continues to evolve Spear-phishing on the rise for US military and other organizations aiming sensitive information Voice over IP phishing becoming more prevalent Phishing techniques continue to evolve 84

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Research Problem As phishing continues to evolve, what can and should stakeholders do to better fight it? 85

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Summary of Thesis Statement Identify phishing stakeholders and their stakes Find gaps in the countermeasures pursued by each stakeholder Generate and evaluate policy options to better fight phishing now and in the future Case studies on the effectiveness of anti-phishing toolbars and game-based anti-phishing education 86

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Overview Stakeholder analysis Understand Countermeasures Effectiveness Incentives for stakeholders Advantages and disadvantages Expert analysis Interview 7-10 experts Current and future threats countermeasures Gaps Find gaps between countermeasures and attacks Contrast stakeholder actions with expert analysis Recommen -dations Who is in the best position? How to shift incentives? Is there some kind of tools to build? Case study: Antiphishing toolbars Case study: Anti-phishing Phil 87

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Outline Background Relevant literature Prior Work Public Policy Analysis Case Study in Anti-phishing toolbars Case Study in User Education Schedule 88

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Outline Background Relevant literature Prior Work Public Policy Analysis Case Study in Anti-phishing toolbars Case Study in User Education Schedule 89

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Stakeholders Consumers-- OrganizationsUS Military, Universities, Corporations Financial InstitutionsBank of America, Citibank, Wachovia, Paypal MerchantseBay, Amazon Internet Service ProvidersSBC, Comcast, AOL ProvidersGmail, YahooMail, Hotmail, Outlook, Thunderbird BrowsersInternet Explorer, Firefox, Safari, Opera, Netscape. DNS authoritiesVerisign, various NICs Software VendorsGoogle, Microsoft, Symantec, RSA, MarkMonitor Law EnforcementsFederal Bureau of Investigation(FBI), CERT, Secret Service, Identity Theft Divisions in Law enforcements Government RegulatorsFederal Financial Institutions Examination Council (FFIEC), Federal Trade Commission (FTC) Academic InstitutionsCarnegie Mellon University, Indiana University Industry ConsortiumFinancial Services Technology Council(FSTC), Anti- Phishing Working Group (APWG), Messaging Anti-Abuse Working Group(MAAWG) Direct stakeholders Indirect stakeholders Primary Victims Secondary Victims Vendors Enforcement Oversight / Coordination / Research Market based 90

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Phishing Countermeasures Prevention Detection Shutdown Warn user Block s / websites Authenticati on / Fraud detection 91 Education

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Economics of information Security - Externalities Does successfully combating phishing depends on the efforts of the laziest and most cowardly family? or the most valiant knight? or sum of efforts? If it is all of above, which part requires what kinds of efforts? 92

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Hypothesis Consumers are the weakest link The problem can be solved if a solution has ubitiquous coverage and near perfect performance, and browsers are the most likely candidate. In which case, phishers will use other channels Effective law enforcements require the sum of all efforts 93

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Longitudinal Trends 94

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Estimating problems 95

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Estimating Countermeasures 1) What advantages, constraints does each stakeholder have in their phishing countermeasures? 2) What kind of solutions best fit each type of stakeholder? 96

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Understanding Constraints 97

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Expert Interviews Goal: To further understand current and future phishing threats, relevant countermeasures, and with an eye on tomorrow, countermeasures should be put in place. 12 experts from industry associations, academia, industry, law enforcements, and volunteer organizations 98

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Expert Interviews SectorsExamplesNumber to Interview Industry AssociationsAnti-Phishing Working Group (APWG), Messaging Anti-abuse Working Group (MAAWG), Financial Services Technology Council (FSTC) 2-3 officers IndustryMicrosoft, Google, RSA, Symantec, MarkMonitor, McAfee, MessageLabs, and CloudMark 3-6 experts Law EnforcementsFederal Bureau of Investigation (FBI), Secret Service, CERT 2-4 experts AcademiaCMU and other institutions3-5 faculty Volunteer OrganizationsPhishTank, CastleCorps2 experts 99

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images High Level Questions Phishing threats What do you think the current state of phishing? How phishing are costing various stakeholders? What kinds of attacks would likely to happen in the near future and long term? Countermeasures What kinds of solutions are stakeholders adopting? What are some effective ways to combat phishing? In light of the evolving phishing threats, what are some of the most promising ways? Is there anything missing in the countermeasures? 100

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images High Level Questions Policy Related Who is the best position to solve the problem? and what kind of solutions you see are lacking? What additional investments are needed? How should we prioritize our spending on prevention, detection, shutdown, and education? Where are we wasting our money at? 101

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Methodology Semi-structured interviews Refine the objective and questions; outline a design; draft the interview questions; pilot test with 3 CMU experts, iterate on it more based on the results Conduct interviews from May 2008 to October 2008 Follow up surveys with some organizations 102

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Overview Stakeholder analysis Understand Countermeasures Effectiveness Incentives for stakeholders Advantages and disadvantages Expert analysis Interview 7-10 experts Current and future threats countermeasures Gaps Find gaps between countermeasures and attacks Contrast stakeholder actions with expert analysis Recommen- dations Who is in the best position? How to shift incentives? Is there some kind of tools to build? 103

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Gap Analysis Map countermeasures with attack vectors Contrast stakeholders actions with expert analysis and recommendations 104

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Gap Analysis AttackPreventionDetectionWarningBlock / Shutdown Website Instant Messaging Auto Dialer News, Chat Room, Blog Bulletin Board Wireless LANs P2P or Interactive Games Malware 105

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Phishing Life Cycle Source: Financial Service Technology Consortium,

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Phishing Life Cycle PlanningSetupAttackCollectionFraud Post Attack 107

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Why People fall People judge a website's legitimacy by its “look and feel” (Rachna et al. 2006, Wu et al. 06) Many do not understand or trust web browser indicator (Downs et al. 2007) Awareness do not link to different behaviors or strategies (Downs et al. 2007) Perceived severity of the consequences does not predict their behaviors (Downs et al. 2007) 108

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Cost of Phishing Direct costs Consumers lose money, banking fraud Estimated 350 – 2 billion Indirect costs Erosion of consumer trust Impact on brand name Increase in customer call centers Opportunity costs 109

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Recent Developments VOIP phishing Spear phishing Rock phish and fast flux 110

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images 111

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Phil Online User Study Conducted in 9/25 – 10/10 Validate Lab study results Test for retention of knowledge Condition Control: N = 2702 (12 websites + game) Game: N = 2021 (674 complete one week later) (6 website + game + 6 website + 6 website one week later) 112

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Game results 113

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Misidentifying Legitimate Sites 114

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Falling for Phishing 115

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Comparing Control with Game Control group performance Pre test score: 70.9% Post test score:67.1% The effect is not due to simply showing the quiz. (p<0.0001, N = 4674) (2 sample t test on (Score_post – Score_pre)) 116

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Signal detection theory to measure learning Users are learning well in the game d’_pre = 1.49, d’_post = 2.46 (p<0.001). The improvement is not due to becoming more suspicious, in fact the reverse it true. C’_pre = , C’_post = (p<0.001) 117

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Intervention #2 – Comic Strip

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Intervention #2 – Comic Strip

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Little Knowledge of Phishing Only about half knew meaning of the term “phishing” “Something to do with the band Phish, I take it.”

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Little Attention Paid to URLs Only 55% of participants said they had ever noticed an unexpected or strange-looking URL Most did not consider them to be suspicious

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Some Knowledge of Scams 55% of participants reported being cautious when asks for sensitive financial info But very few reported being suspicious of asking for passwords Knowledge of financial phish reduced likelihood of falling for these scams But did not transfer to other scams, such as an amazon.com password phish

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Embedded Training Evaluation #1 Lab study comparing our prototypes to standard security notices Group A – eBay, PayPal notices Group B – Diagram that explains phishing Group C – Comic strip that tells a story 10 participants in each condition (30 total) Screened so we only have novices Go through 19 s, 4 phishing attacks scattered throughout, 2 training s too Role play as Bobby Smith at Cognix Inc

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Embedded Training Results

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Embedded Training Results Existing practice of security notices is ineffective Diagram intervention somewhat better Though people still fell for final phish Comic strip intervention worked best Statistically significant Combination of less text, graphics, story?

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images Evaluation #2 New questions: Have to fall for phishing to be effective? How well do people retain knowledge? Roughly same experiment as before Role play as Bobby Smith at Cognix Inc, go thru 16 s Embedded condition means have to fall for our Non-embedded means we just send the comic strip Also had people come back after 1 week Kumaraguru, P., Rhee, Y., Sheng, S., Hasan, S., Acquisti, A., Cranor, L. F., and Hong, J. Getting users to pay attention to anti-phishing education: evaluation of retention and transfer. In eCrime ’07: Proceedings of the anti-phishing working groups 2nd annual eCrime researchers summit

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images A Science of Warnings See the warning? Understand? Believe it? Motivated? Planning on refining this model for computer warnings

Steve Sheng C MU U sable P rivacy and S ecurity Laboratory You can add your name and/or URL to the footer if you want. If your figures overlap the CUPS logo, then suppress the background images APWG