1. Data Mining (and machine learning) ROC curves Rule Induction Basics of Text Mining

2. Two classes is a common and special case

3. Medical applications: cancer, or not? Computer Vision applications: landmine, or not? Security applications: terrorist, or not? Biotech applications: gene, or not? …

4. Two classes is a common and special case Medical applications: cancer, or not? Computer Vision applications: landmine, or not? Security applications: terrorist, or not? Biotech applications: gene, or not? … Predicted YPredicted N Actually YTrue PositiveFalse Negative Actually NFalse PositiveTrue Negative

5. Two classes is a common and special case True Positive: these are ideal. E.g. we correctly detect cancer Predicted YPredicted N Actually YTrue PositiveFalse Negative Actually NFalse PositiveTrue Negative

6. Two classes is a common and special case True Positive: these are ideal. E.g. we correctly detect cancer False Positive: to be minimised – cause false alarm – can be better to be safe than sorry, but can be very costly. Predicted YPredicted N Actually YTrue PositiveFalse Negative Actually NFalse PositiveTrue Negative

7. Two classes is a common and special case True Positive: these are ideal. E.g. we correctly detect cancer False Positive: to be minimised – cause false alarm – can be better to be safe than sorry, but can be very costly. False Negative: also to be minimised – miss a landmine / cancer very bad in many applications Predicted YPredicted N Actually YTrue PositiveFalse Negative Actually NFalse PositiveTrue Negative

8. Two classes is a common and special case True Positive: these are ideal. E.g. we correctly detect cancer False Positive: to be minimised – cause false alarm – can be better to be safe than sorry, but can be very costly. False Negative: also to be minimised – miss a landmine / cancer very bad in many applications True Negative?: Predicted YPredicted N Actually YTrue PositiveFalse Negative Actually NFalse PositiveTrue Negative

9. Sensitivity and Specificity: common measures of accuracy in this kind of 2-class tasks Predicted YPredicted N Actually YTrue PositiveFalse Negative Actually NFalse PositiveTrue Negative

10. Sensitivity and Specificity: common measures of accuracy in this kind of 2-class task Sensitivity = TP/(TP+FN) - how much of the real ‘Yes’ cases are detected? How well can it detect the condition? Specificity = TN/(FP+TN) - how much of the real ‘No’ cases are correctly classified? How well can it rule out the condition? Predicted YPredicted N Actually YTrue PositiveFalse Negative Actually NFalse PositiveTrue Negative

11. YES NO

13. Sensitivity: 100% Specificity: 25% YES NO

14. Sensitivity: 93.8% Specificity: 50%

15. YES NO Sensitivity: 81.3% Specificity: 83.3% YES NO

16. Sensitivity: 56.3% Specificity: 100% YES NO

17. Sensitivity: 100% Specificity: 25% YES NO 100% Sensitivity means: detects all cancer cases (or whatever) but possibly with many false positives

18. YES NO Sensitivity: 56.3% Specificity: 100% YES NO 100% Specificity means: misses some cancer cases (or whatever) but no false positives

19. Sensitivity and Specificity: common measures of accuracy in this kind of 2-class tasks Sensitivity = TP/(TP+FN) - how much of the real TRUE cases are detected? How sensitive is the classifier to TRUE cases? A highly sensitive test for cancer: if “NO” then you be sure it’s “NO” Specificity = TN/(TN+FP) - how sensitive is the classifier to the negative cases? A highly specific test for cancer: if “Y” then you be sure it’s “Y”. With many trained classifiers, you can ‘move the line’ in this way. E.g. with NB, we could use a threshold indicating how much higher the log likelihood for Y should be than for N

20. ROC curves David Corne, and Nick Taylor, Heriot-Watt University - dwcorne@gmail.com These slides and related resources: http://www.macs.hw.ac.uk/~dwcorne/Teaching/dmml.html

21. Rule Induction Rules are useful when you want to learn a clear / interpretable classifier, and are less worried about squeezing out as much accuracy as possible There are a number of different ways to ‘learn’ rules or rulesets. Before we go there, what is a rule / ruleset?

22. Rules IF Condition … Then Class Value is …

23. YES NO 543210 543210 0 1 2 3 4 5 6 7 8 9 10 11 12 Rules are Rectangular IF (X>0)&(X 0.5)&(Y<5) THEN YES

24. YES NO 543210 543210 0 1 2 3 4 5 6 7 8 9 10 11 12 Rules are Rectangular IF (X>5)&(X 4.5)&(Y<5.1) THEN NO

25. A Ruleset IF Condition1 … Then Class = A IF Condition2 … Then Class = A IF Condition3 … Then Class = B IF Condition4 … Then Class = C …

26. YES NO 543210 543210 0 1 2 3 4 5 6 7 8 9 10 11 12 What’s wrong with this ruleset? (two things)

27. YES NO 543210 543210 0 1 2 3 4 5 6 7 8 9 10 11 12 What about this ruleset?

28. Two ways to interpret a ruleset:

29. As a Decision List IF Condition1 … Then Class = A ELSE IF Condition2 … Then Class = A ELSE IF Condition3 … Then Class = B ELSE IF Condition4 … Then Class = C … ELSE … predict Background Majority Class

30. Two ways to interpret a ruleset: As an unordered set IF Condition1 … Then Class = A IF Condition2 … Then Class = A IF Condition3 … Then Class = B IF Condition4 … Then Class = C Check each rule and gather votes for each class If no winner, predict background majority class

31. Three broad ways to learn rulesets

32. 1. Just build a decision tree with ID3 (or something else) and you can translate the tree into rules!

33. Three broad ways to learn rulesets 2. Use any good search/optimisation algorithm. Evolutionary (genetic) algorithms are the most common. You will do this coursework 3. This means simply guessing a ruleset at random, and then trying mutations and variants, gradually improving them over time.

34. Three broad ways to learn rulesets 3. A number of ‘old’ AI algorithms exist that still work well, and/or can be engineered to work with an evolutionary algorithm. The basic idea is: iterated coverage

35. YES NO 543210 543210 0 1 2 3 4 5 6 7 8 9 10 11 12 Take each class in turn..

36. YES NO 543210 543210 0 1 2 3 4 5 6 7 8 9 10 11 12 Pick a random member of that class in the training set

37. YES NO 543210 543210 0 1 2 3 4 5 6 7 8 9 10 11 12 Extend it as much as possible without including another class

38. YES NO 543210 543210 0 1 2 3 4 5 6 7 8 9 10 11 12 Extend it as much as possible without including another class

39. YES NO 543210 543210 0 1 2 3 4 5 6 7 8 9 10 11 12 Extend it as much as possible without including another class

40. YES NO 543210 543210 0 1 2 3 4 5 6 7 8 9 10 11 12 Extend it as much as possible without including another class

41. YES NO 543210 543210 0 1 2 3 4 5 6 7 8 9 10 11 12 Next class

42. YES NO 543210 543210 0 1 2 3 4 5 6 7 8 9 10 11 12 Next class

43. YES NO 543210 543210 0 1 2 3 4 5 6 7 8 9 10 11 12 And so on…

44. Text as Data: what and why?

45. 2012 2014 Students’ implementation choices for DMML CW1 “Word Clouds” - word frequency patterns provides useful information

46. Classify sentiment http://www.inside-r.org/howto/mining-twitter-airline-consumer-sentiment Twitter sentiment ACS Index “Word Clouds” - word frequency patterns provides useful information …which can be used to predict a class value / category / signal … in this case the document(s) are “tweets mentioning our airline over past few hours” class value is a satisfaction score, between 0 and 1

47. sentiment map of NYC http://necsi.edu/research/social/new york/sentimentmap/ more info from tweets, this time, a “happiness” score.

48. “similar pages” Based on distances between word frequency patterns

49. Predicting relationship between two people based on their text messages

50. Can you predict class: Desktop, Laptop or LED-TV from word frequencies of product description on amazon ?

51. So, word frequency is important – does this mean that the most frequent words in a text carry the most useful information about its content/category/meaning?

52. Zipf’s law -- text of Moby Dick -- http://searchengineland.com/the-long-tail-of-search-12198

53. http://www.wordfrequency.info/files/entri esWithoutCollocates.txt Rank Word Part of speech Frequency Dispersion 1 the a 22038615 0.98 2 be v 12545825 0.97 3 and c 10741073 0.99 4 of i 10343885 0.97 5 a a 10144200 0.98 6 in i 6996437 0.98 7 to t 6332195 0.98 8 have v 4303955 0.97 9 to i 3856916 0.99 10 it p 3872477 0.96 11 I p 3978265 0.93 12 that c 3430996 0.97 13 for i 3281454 0.98 14 you p 3081151 0.92 15 he p 2909254 0.94 16 with i 2683014 0.99 17 on i 2485306 0.99 18 do v 2573587 0.95 19 say v 1915138 0.95 20 this d 1885366 0.96 21 they p 1865580 0.96 Frequencies of words from http://corpus.byu.edu/coca Corpus of Contemporary American English 450 million words from fiction books, newspapers, magazines, etc… So, 22,038,615/450,000,000 = 4.9% are ‘the’ /

54. http://www.wordfrequency.info/files/entries WithoutCollocates.txt 1000 detailn387500.97 1001 methodn426670.88 1002 signv394180.95 1003 somebodyp416390.90 1004 magazinen391970.96 1005 hoteln396420.95 1006 soldiern395520.95 1007 reflectv406870.92 1008 heavyj391600.96 1009 sexualj418650.89 1010 causen388430.96 1011 bagn400070.93 1012 heatn403630.92 1013 falln388220.96 1014 marriagen385170.96 1015 toughj396000.94

55. http://www.wordfrequency.info/files/entries WithoutCollocates.txt 4986 kneelv54670.86 4987 vacuumn50280.93 4988 selectedj53150.88 4989 dictatev49350.95 4990 stereotypen51090.92 4991 sensorn51540.91 4992 laundryn50630.93 4993 manualn50400.93 4994 pistoln52610.89 4995 navalj49900.94 4996 immigrantj0.97 4996 plaintiffn53120.88 4997 kidv50940.92 4998 middle-classj50250.93 4999 apologyn49720.94 5000 tilli50790.92

56. Zipf’s law -- text of Moby Dick -- http://searchengineland.com/the-long-tail-of-search-12198 the frequency of a specific word in text X is important, but, only if it is not similarly frequent in other texts – in which case it carries little information about X

57. Which leads us to TFIDF We can do any kind of DMML with text, as soon as we convert text into numbers -This is usually done with a “TFIDF” encoding -and almost always done with either TFIDF or a close relation -TFIDF is basically word frequency, but takes into account ‘background frequency’ – so words that are very common have their value reduced.

58. A one-slide text-mining tutorial an essay about sport an article about poltics another article about politics (0.1, 0.2, 0, 0.02...) (0.4, 0, 0.1, 0...) (0.11, 0.3, 0, 0.01..) NOW you can do Clustering, Retrieving similar Documents, Supervised Classification Etc... Vectors based on word frequencies. One key issue is to choose the right set of words (or other features)

59. First, a quick illustration to show why word- frequency vectors are useful

60. How did I get these vectors from these two `documents’? Compilers: lecture 1 This lecture will introduce the concept of lexical analysis, in which the source code is scanned to reveal the basic tokens it contains. For this, we will need the concept of regular expressions (r.e.s). Compilers The Guardian uses several compilers for its daily cryptic crosswords. One of the most frequently used is Araucaria, and one of the most difficult is Bunthorne. 35, 2, 0 26, 2, 2

61. What about these two vectors? Compilers: lecture 1 This lecture will introduce the concept of lexical analysis, in which the source code is scanned to reveal the basic tokens it contains. For this, we will need the concept of regular expressions (r.e.s). Compilers The Guardian uses several compilers for its daily cryptic crosswords. One of the most frequently used is Araucaria, and one of the most difficult is Bunthorne. 0, 0, 0, 1, 1, 1 1, 1, 1, 0, 0, 0

62. From this MASTER WORD LIST (ordered) (Crossword, Cryptic, Difficult, Expression, Lexical, Token) If a document contains `crossword’, it gets a 1 in position 1 of the vector, otherwise 0. If it contains `lexical’, it gets a 1 in position 5, otherwise 0, and so on. How similar would be the vectors for two docs about crossword compilers?.

63. Turning a document into a vector We start with a template for the vector, which needs a master list of terms. A term can be a word, or a number, or anything that appears frequently in documents. There are almost 200,000 words in English – it would take much too long to process documents vectors of that length. Commonly, vectors are made from a small number (50—1000) of most frequently-occurring words. However, the master list usually does not include words from a stoplist, Which contains words such as the, and, there, which, etc … why?

64. The TFIDF Encoding (Term Frequency x Inverse Document Frequency) A term is a word, or some other frequently occuring item Given some term i, and a document j, the term count is the number of times that term i occurs in document j Given a collection of k terms and a set D of documents, the term frequency, is: … considering only the terms of interest, this is the proportion of document j that is made up from term i.

65. The TFIDF Encoding (Term Frequency x Inverse Document Frequency) A term is a word, or some other frequently occuring item Given some term i, and a document j, the term count is the number of times that term i occurs in document j Given a collection of k terms and a set D of documents, the term frequency, is: frequency of this word in this doc total number of words in this doc … considering only the terms of interest, this is the proportion of document j that is made up from term i.

66. Some made-up data for illustration: TF vectors moneyinterestEUdesignerwear CATEGORY 0.030.040.0800.01Economics 0.020.060.0900Economics 0.04 0.020.010.02Fashion 00.0300.010.02Fashion

67. Term frequency is a measure of the importance of this term in this document Inverse document frequency (which we see next) is a measure of the discriminatory value of the term in the collection of documents we are looking at.. It is a measure of the rarity of this word in this document collection E.g. high term frequency for “money” means that money is an important word in a specific document. But high document frequency (low inverse document frequency) for “money”, given a particular set of documents, means that money does not carry much useful information, since it is in many of the documents.

68. Inverse document frequency of term i is: where D is a master collection of documents – and C are the subset of D that contain term i at least once. E.g. if we are trying to learn a classifier of news articles into ‘sport’, ‘economics’, etc… D might be a set of 100,000 news articles Often, we simply replace idf with ‘background frequences’ obtained From a corpus such as the CCA corpus

69. TFIDF encoding of a document So, given: - a background collection of documents (e.g. 100,000 random web pages, all the articles we can find about cancer 100 student essays submitted as coursework …) - a specific ordered list (possibly large) of terms We can encode any document as a vector of TFIDF numbers, where the ith entry in the vector for document j is:

70. Some made-up data for illustration: now they are TFIDF vectors, and some have reduced more than others MoneyinterestEUdesignerwear CATEGORY 0.00750.040.0700.01Economics 0.0050.060.0800Economics 0.010.040.01 0.02Fashion 00.0300.010.02Fashion

71. Vector representation of documents underpins: Many areas of automated document analysis Such as: automated classification of documents Clustering and organising document collections Building maps of the web, and of different web communities Understanding the interactions between different scientific communities, which in turn will lead to helping with automated WWW-based scientific discovery.

72. Example / recent work of my PhD student Hamouda Chantar

73. Three datasets / classification / main issue: Feature Selection DatasetArticles in Train / Test categ ories Distinct words in training set Al-Jazeera News 1200 / 3005 5,329 Alwatan821 / 3524 12,282 Akhbar - Alkhaleej 1365 / 3434 8,913

74. Hamouda’s work Focus on automated classification of an article (e.g. Finance, Economics, Sport, Culture,...) Emphasis on Feature Selection – which words or other features should constitute the vectors, to enable accurate classification?

75. Example categories: this is the Akhbar-Alkhaleej dataset CategoryTrainTestTotal International News 22858286 Local news576144720 Sport34386429 Economy21855273 Total13653431708

76. We look at 3 pre-classified datasets Akhbar-Alkhaleej: 5690 Arabic news documents gathered evenly from the online newspaper "Akhbar-Alkhaleej" Alwatan: 20,291 Arabic news documents gathered from online newspaper "Alwatan” Al-jazeera-News:1500 documents from the Al-Jazeera news site.

77. is.gd/arabdata

78. We look at 3 classification methods (when evaluating feature subsets on the test set) C4.5: well-known decision tree classifier, we use weka’s implementation, “J48” Naive Bayes: It’s Naive, and it’s Bayes SVM: with a linear kernel

79. Results: Alwatan dataset

80. Results on Al Jazeera dataset

81. Results: Akhbar-Alkhaleej dataset

82. tara