Presentation is loading. Please wait.

Presentation is loading. Please wait.

How does computer know what is spam and what is ham?

Published by Modified over 9 years ago

Similar presentations

Presentation on theme: "How does computer know what is spam and what is ham?"— Presentation transcript:

1 How does computer know what is spam and what is ham?

2 Attempt 1: (define (spam? email)‏ (cond ( (email from known sender) False)‏ ( (email contains “viagra”) True)‏ ( (email begins with “Dear Mr/Mrs.”) True)‏ ( (email contains URL) True)‏ ( (email contains attachment) True)‏ (...

3 Problem: (email contain URL) is an indication, NOT a PROOF Attempt 1: (define (spam? email)‏ (cond ( (email from known sender) False)‏ ( (email contains “viagra”) True)‏ ( (email begins with “Dear Mr/Mrs.”) True)‏ ( (email contains URL) True)‏ ( (email contains attachment) True)‏ (...

4 Features: Score: email from known sender -50 email contains "viagra" 75 email begins with "Dear Mr/Mrs." 70 email contains URL 10 email contains attachment 5......... If Total Sum > 100, classify as spam.

5 Features: Score: email from known sender -50 email contains "viagra" 75 email begins with "Dear Mr/Mrs." 70 email contains URL 10 email contains attachment 5......... If Total Sum > 100, classify as spam. Problems: - How to determine the score? - How to combine the score?

6 Key Idea: Learn which features are important through examples Training Set: lots of emails with correct labels (both spam and ham)

7 The Naive Bayes Algorithm: Step 1. Gather Statistics inside Training Set:

8 The Naive Bayes Algorithm: Step 1. Gather Statistics inside Training Set: - Count percentage of spams in Training Set: P(spam)‏ - Count percentage of hams in Training Set: P(ham)‏ - For every feature F_1, F_2, F_3... : = Count percentage of spams with feature F_i : P(F_i | spam)‏ = Count percentage of hams with feature F_i : P(F_i | ham)‏

9 The Naive Bayes Algorithm: Say, F_1 = email contains “viagra” F_2 = email begins with “Dear Mr/Mrs.”

10 The Naive Bayes Algorithm: Say, F_1 = email contains “viagra” F_2 = email begins with “Dear Mr/Mrs.” From Training Set, we discovered: P(spam) = 0.85 P(ham) = 0.15 P(F_1 | spam) = 0.2 P(NOT F_1 | spam) = 0.8 P(F_1 | ham) = 0.001 P(NOT F_1 | ham) 0.999 P(F_2 | spam) = 0.99 P(NOT F_2 | spam) = 0.01 P(F_2 | ham) = 0.0001 P(NOT F_2 | ham) = 0.9999

11

12 The Naive Bayes Algorithm: Step 1. Gather Statistics inside Training Set: - Count percentage of spams in Training Set: P(spam)‏ - Count percentage of hams in Training Set: P(ham)‏ - For every feature F_1, F_2, F_3... : = Count percentage of spams with feature F_i : P(F_i | spam)‏ = Count percentage of hams with feature F_i : P(F_i | ham)‏ Step 2. On a new Instance: - Find what features the new instance has - Use Bayes Rule to compute probability - Take the most probable label

13 Example: Optical Character Recognition GOAL: recognize scanned hand-written numbers..................................++++++......................##############++............+++++##########+..................+.+++++##+........................+##........................+##+.......................+##+........................+#+.........................##+........................+#+........................+##+........................##+........................###+.......................+##+.......................+##+.......................+###+.......................+##.......................................................+#........................+###.......................+####+......................+######+...................+###+####+..................+##..+####..................+#+...+##+..................+#+...###+..................+##+++####+..................#####++##+..................+###+..+##+..................+++....+#+.........................+##..........................+#+.........................+##+.........................+#+.........................+##+.........................+#+..........................+#+..........................#+................................

14 Instance – scanned image of hand-written number Labels – 1,2,3,4,5,6,7,8,9 Features – (for project)‏ every 2x2 pixel squares

15 Instance – scanned image of hand-written number Labels – 1,2,3,4,5,6,7,8,9 Features – (for project)‏ every 2x2 pixel squares........................................+#........................+###.......................+####+......................+######+...................+###+####+..................+##..+####..................+#+...+##+..................+#+...###+..................+##+++####+..................#####++##+..................+###+..+##+..................+++....+#+.........................+##..........................+#+.........................+##+.........................+#+.........................+##+.........................+#+..........................+#+..........................#+................................

16 Instance – scanned image of hand-written number Labels – 1,2,3,4,5,6,7,8,9 Features – (for project)‏ every 2x2 pixel squares........................................+#........................+###.......................+####+......................+######+...................+###+####+..................+##..+####..................+#+...+##+..................+#+...###+..................+##+++####+..................#####++##+..................+###+..+##+..................+++....+#+.........................+##..........................+#+.........................+##+.........................+#+.........................+##+.........................+#+..........................+#+..........................#+................................

17 Instance – scanned image of hand-written number Labels – 1,2,3,4,5,6,7,8,9 Features – (for project)‏ every 2x2 pixel squares........................................+#........................+###.......................+####+......................+######+...................+###+####+..................+##..+####..................+#+...+##+..................+#+...###+..................+##+++####+..................#####++##+..................+###+..+##+..................+++....+#+.........................+##..........................+#+.........................+##+.........................+#+.........................+##+.........................+#+..........................+#+..........................#+................................

18 Instance – scanned image of hand-written number Labels – 1,2,3,4,5,6,7,8,9 Features – (for project)‏ every 2x2 pixel squares........................................+#........................+###.......................+####+......................+######+...................+###+####+..................+##..+####..................+#+...+##+..................+#+...###+..................+##+++####+..................#####++##+..................+###+..+##+..................+++....+#+.........................+##..........................+#+.........................+##+.........................+#+.........................+##+.........................+#+..........................+#+..........................#+................................

19 Instance – scanned image of hand-written number Labels – 1,2,3,4,5,6,7,8,9 Features – (for project)‏ every 2x2 pixel squares........................................+#........................+###.......................+####+......................+######+...................+###+####+..................+##..+####..................+#+...+##+..................+#+...###+..................+##+++####+..................#####++##+..................+###+..+##+..................+++....+#+.........................+##..........................+#+.........................+##+.........................+#+.........................+##+.........................+#+..........................+#+..........................#+................................

20 Instance – scanned image of hand-written number Labels – 1,2,3,4,5,6,7,8,9 Features – (for project)‏ every 2x2 pixel squares Steps. - Turn image-file into a stream of Images (Abstract Data Type)‏ (done for you)‏

21 Instance – scanned image of hand-written number Labels – 1,2,3,4,5,6,7,8,9 Features – (for project)‏ every 2x2 pixel squares Steps. - Turn image-file into a stream of Images (Abstract Data Type)‏ (done for you)‏ - Gather feature statistics from Training File (mostly done for you)‏

22 Instance – scanned image of hand-written number Labels – 1,2,3,4,5,6,7,8,9 Features – (for project)‏ every 2x2 pixel squares Steps. - Turn image-file into a stream of Images (Abstract Data Type)‏ (done for you)‏ - Gather feature statistics from Training File (mostly done for you)‏ - Implement Bayes' Rule (mostly your own work)‏

23 Instance – scanned image of hand-written number Labels – 1,2,3,4,5,6,7,8,9 Features – (for project)‏ every 2x2 pixel squares Steps. - Turn image-file into a stream of Images (Abstract Data Type)‏ (done for you)‏ - Gather feature statistics from Training File (mostly done for you)‏ - Implement Bayes' Rule (mostly your own work)‏ - Evaluate your OCR by guessing labels on Validation File (mostly done for you)‏

Download ppt "How does computer know what is spam and what is ham?"

Similar presentations

Document Filtering Dr. Frank McCown Intro to Web Science Harding University This work is licensed under a Creative Commons Attribution-NonCommercial- ShareAlike.

Document Filtering Dr. Frank McCown Intro to Web Science Harding University This work is licensed under a Creative Commons Attribution-NonCommercial- ShareAlike.
Announcements •Homework due Tuesday. •Office hours Monday 1-2 instead of Wed. 2-3.

Announcements •Homework due Tuesday. •Office hours Monday 1-2 instead of Wed. 2-3.
Classification Classification Examples

Classification Classification Examples
Rapid Object Detection using a Boosted Cascade of Simple Features Paul Viola, Michael Jones Conference on Computer Vision and Pattern Recognition 2001.

Rapid Object Detection using a Boosted Cascade of Simple Features Paul Viola, Michael Jones Conference on Computer Vision and Pattern Recognition 2001.
The 10 Commandments for Java Programmers VII: Thou shalt study thy libraries with care and strive not to reinvent them without cause, that thy code may.

The 10 Commandments for Java Programmers VII: Thou shalt study thy libraries with care and strive not to reinvent them without cause, that thy code may.
電腦視覺 Computer and Robot Vision I Chapter2: Binary Machine Vision: Thresholding and Segmentation Instructor: Shih-Shinh Huang 1.

電腦視覺 Computer and Robot Vision I Chapter2: Binary Machine Vision: Thresholding and Segmentation Instructor: Shih-Shinh Huang 1.
CSC 380 Algorithm Project Presentation Spam Detection Algorithms Kyle McCombs Bridget Kelly.

CSC 380 Algorithm Project Presentation Spam Detection Algorithms Kyle McCombs Bridget Kelly.
The Viola/Jones Face Detector Prepared with figures taken from “Robust real-time object detection” CRL 2001/01, February 2001.

The Viola/Jones Face Detector Prepared with figures taken from “Robust real-time object detection” CRL 2001/01, February 2001.
Assuming normally distributed data! Naïve Bayes Classifier.

Assuming normally distributed data! Naïve Bayes Classifier.
Bayes Rule How is this rule derived? Using Bayes rule for probabilistic inference: –P(Cause | Evidence): diagnostic probability –P(Evidence | Cause): causal.

Bayes Rule How is this rule derived? Using Bayes rule for probabilistic inference: –P(Cause | Evidence): diagnostic probability –P(Evidence | Cause): causal.
Multi-Class Object Recognition Using Shared SIFT Features

Multi-Class Object Recognition Using Shared SIFT Features
5/30/2006EE 148, Spring 20061 Visual Categorization with Bags of Keypoints Gabriella Csurka Christopher R. Dance Lixin Fan Jutta Willamowski Cedric Bray.

5/30/2006EE 148, Spring Visual Categorization with Bags of Keypoints Gabriella Csurka Christopher R. Dance Lixin Fan Jutta Willamowski Cedric Bray.
ML ALGORITHMS. Algorithm Types Classification (supervised) Given -> A set of classified examples “instances” Produce -> A way of classifying new examples.

ML ALGORITHMS. Algorithm Types Classification (supervised) Given -> A set of classified examples “instances” Produce -> A way of classifying new examples.
CPSC 422, Lecture 18Slide 1 Intelligent Systems (AI-2) Computer Science cpsc422, Lecture 18 Feb, 25, 2015 Slide Sources Raymond J. Mooney University of.

CPSC 422, Lecture 18Slide 1 Intelligent Systems (AI-2) Computer Science cpsc422, Lecture 18 Feb, 25, 2015 Slide Sources Raymond J. Mooney University of.
Naive Bayes model Comp221 tutorial 4 (assignment 1) TA: Zhang Kai.

Naive Bayes model Comp221 tutorial 4 (assignment 1) TA: Zhang Kai.
Introduction to machine learning

Introduction to machine learning
Handwritten Character Recognition using Hidden Markov Models Quantifying the marginal benefit of exploiting correlations between adjacent characters and.

Handwritten Character Recognition using Hidden Markov Models Quantifying the marginal benefit of exploiting correlations between adjacent characters and.
Learning at Low False Positive Rate Scott Wen-tau Yih Joshua Goodman Learning for Messaging and Adversarial Problems Microsoft Research Geoff Hulten Microsoft.

Learning at Low False Positive Rate Scott Wen-tau Yih Joshua Goodman Learning for Messaging and Adversarial Problems Microsoft Research Geoff Hulten Microsoft.
Naïve Bayes Chapter 4, DDS. Introduction Classification Training set  design a model Test set  validate the model Classify data set using the model.

Naïve Bayes Chapter 4, DDS. Introduction Classification Training set  design a model Test set  validate the model Classify data set using the model.
EMAIL SPAM DETECTION USING MACHINE LEARNING Lydia Song, Lauren Steimle, Xiaoxiao Xu.

SPAM DETECTION USING MACHINE LEARNING Lydia Song, Lauren Steimle, Xiaoxiao Xu.

Similar presentations

Ads by Google