WEB FORUM MINING BASED ON USER SATISFACTION PAGE 1 WEB FORUM MINING BASED ON USER SATISFACTION By: Suresh Pokharel Information and Communications Technologies Asian Institute of Technology Committee: Dr. Sumanta Guha (Chairperson) Prof. Phan Minh Dung Assoc. Prof. Tapio J. Erke May 2010

Introduction Problem Statement Objectives Methodology Implementation Results and Discussion Conclusion and Future Work Demonstration WEB FORUM MINING BASED ON USER SATISFACTION PAGE 2 Agendas Introduction

 Internet-Forum or Message Board  Online Discussion Site  Asynchronous  People participating in an Internet forum may cultivate social bonds and interest groups for a topic may form from the discussions WEB FORUM MINING BASED ON USER SATISFACTION PAGE 3

Figure 1: Organization of Threads WEB FORUM MINING BASED ON USER SATISFACTION PAGE 4 Introduction

Table 1: An example of a thread WEB FORUM MINING BASED ON USER SATISFACTION PAGE 5 Introduction Title: Software for Ubuntu. Post No.PostUsersCategory 1 I am really new to Linux, where can I find software for Ubuntu? avacomputersQuestion 2 Applications>Add/remove If you are using Ubuntu Firefox, you can also use danielrmtAnswer 3http:// 4 something else I was reading about is a port from OzOs ( called apt:foo devildoc5Answer 5 or the easy way >.> CLICK SYSTEM > PREFERENCES > SYNAPTIC PACKAGE MANAGER there you can search all kinds of games, software! anything you like, thousands to choose from! They download and install easily onto your system Codix121Answer 6Thanks GUys.avacomputersAnswer Questioner Repliers Questioner Questioner Post

Introduction Problem Statement Objectives Methodology Implementation Results and Discussion Conclusion and Future Work Demonstration WEB FORUM MINING BASED ON USER SATISFACTION PAGE 6 Agendas

Problem Statement Which forum may have solution? Lots of Forums…… Ooops WEB FORUM MINING BASED ON USER SATISFACTION PAGE 7 I don’t want to test all forums…

Introduction Problem Statement Objectives Methodology Implementation Results and Discussion Conclusion and Future Work Demonstration WEB FORUM MINING BASED ON USER SATISFACTION PAGE 8 Agendas

Objectives  To categorize a post as a question post or an answer post.  To classify a thread as answered or unanswered based on questioner’s satisfaction and forum features.  To predict a solution post based on interaction and satisfaction of questioner. WEB FORUM MINING BASED ON USER SATISFACTION PAGE 9

Introduction Problem Statement Objectives Methodology Implementation Results and Discussion Conclusion and Future Work Demonstration WEB FORUM MINING BASED ON USER SATISFACTION PAGE 10 Agendas

Methodology: Framework of Study Figure 4: Framework of Study WEB FORUM MINING BASED ON USER SATISFACTION PAGE 11

Figure 5 : Sentence Classification WEB FORUM MINING BASED ON USER SATISFACTION PAGE 12 Methodology: Sentence Classification Example

Label Sequential Patterns (LSPs), p, in the form of  LHS is a sequence, a i is named “item”.  c is a class label (question/non-question) A= has a subsequence B=  A contains B  A LSP p 1 is contained by p 2 if the sequence p 1.LHS is contained by p 2.LHS and p 1.c = p 2.c. Example: t 1 = (,Q) t 2 = (,Q) t 3 = (,NQ) 1 ) LSP p 1 = (, Q) is contained in t 1 and t 2 sup(p 1 ) = 2/3 = 66.7%, conf(p 1 )=(2/3)/(2/3) = 100% 2) LSP p 2 = (, Q) sup(p 2 ) = 2/3 = 66.7%, conf(p 2 )= (2/3)/(3/3) = 66.7% WEB FORUM MINING BASED ON USER SATISFACTION PAGE 13 Methodology: Sentence Classification

Mining LSPs  Word length of sequence : 4  Setting minimum support at 0.01% and minimum confidence at 95% Converting to features  LSP  5W1H word  Auxiliary Verb  Question Mark The corresponding feature being set at 1 if a sentence includes a LSP, question mark, start with 5W1H word, or auxiliary verb. WEB FORUM MINING BASED ON USER SATISFACTION PAGE 14 Methodology: Sentence Classification

Figure 6 : Classification of Thread WEB FORUM MINING BASED ON USER SATISFACTION PAGE 15 Methodology: Thread Classification

 Satisfied Phrase Derive Features :  Unsatisfied Phrase WEB FORUM MINING BASED ON USER SATISFACTION PAGE 16 Methodology: Questioner Post Classification

 Question Present  Presence of More Post WEB FORUM MINING BASED ON USER SATISFACTION PAGE 17 Methodology: Questioner Post Classification Derive Features :

 Satisfied Post Length WEB FORUM MINING BASED ON USER SATISFACTION PAGE 18  Presence of Emoticons  Happy emoticon ( ) : mood of satisfaction  Unhappy emoticon (  ) : mood of un-satisfaction Methodology: Questioner Post Classification Derive Features :

 Original Post WEB FORUM MINING BASED ON USER SATISFACTION PAGE 19 Methodology: Questioner Post Classification Derive Features :

WEB FORUM MINING BASED ON USER SATISFACTION PAGE 20 Figure 7: Classification of Questioner Post Methodology: Questioner Post Classification

 Presence of Quote Find Solution Post  Presence of User Name WEB FORUM MINING BASED ON USER SATISFACTION PAGE 21 Methodology: Predict Solution Posts

 May be between the questioner post QP R R SP Solution Posts WEB FORUM MINING BASED ON USER SATISFACTION PAGE 22 Methodology: Predict Solution Posts

Introduction Problem Statement Objectives Scope and Limitation Methodology Implementation Results and Discussion Conclusion and Future Work Demonstration WEB FORUM MINING BASED ON USER SATISFACTION PAGE 23 Agendas

Dataset  Forum : Ubuntu (  Sentence classification: datasets of 100, 200 and 300 from 3000 sentences  Questioner Post Classification: 250 posts from 79 threads  Manual Evaluation : 100 threads by two team contains 5 person in each team Tools and Language  POS Tag, Tokenization, Sentence Detection : OpenNLP  Classifier : Support Vector Machine (LibSVM, SMO)  Model : SVM is trained using libSVM for classifier model  Language : Java WEB FORUM MINING BASED ON USER SATISFACTION PAGE 24 Implementation

Introduction Problem Statement Objectives Methodology Implementation Results and Discussion Conclusion and Future Work Demonstration WEB FORUM MINING BASED ON USER SATISFACTION PAGE 25 Agendas

FeaturesAccuracyRecallPrecisionF-Measure WH word (5W1H) Question Mark (QM) Auxiliary Verb (Aux) W1H + QM + Aux Labeled Sequential Pattern (LSP)0.94 QM+Aux+LSP+5W1H (LSP+)0.96 Table 3: Accuracy of Sentence Classification by using LSP+ by Class WEB FORUM MINING BASED ON USER SATISFACTION PAGE 26 Result and Discussion : Sentence Classification Comparison  All the results obtained from10 fold cross validation

Table 5: Questioner Post Classification Comparison using Different Features FeaturesPrecisionRecallF-Measure Satisfied Words (SW) 0.78 Unsatisfied Words (UW) 0.72 Question (Ques) More Post (MP) 0.83 Word Count (WC) Happy Emoticon (HE) Unhappy Emoticon (UE) Original Post (OP) Ques+MP0.84 Ques+OP0.86 MP+UE+OP0.88 Ques+MP+OP0.85 SW + UW + Ques SW+UW+Ques+MP+WC+HE+UE+OP0.91 WEB FORUM MINING BASED ON USER SATISFACTION PAGE 27 Result and Discussion : QP Classification Comparison

PerformanceAccuracyRecallPrecisionF-Measure System with Team A System with Team B Average Table 6: Comparison of System Result with Manual Evaluation for thread classification AccuracyRecallPrecisionF-Measure System Accuracy with Team A System Accuracy with Team B Average System Accuracy Table 7: System Accuracy for Prediction of Solution Posts WEB FORUM MINING BASED ON USER SATISFACTION PAGE 28 Result and Discussion : Comparison with Team’s Evaluation

Introduction Problem Statement Objectives Methodology Implementation Results and Discussion Conclusion and Future Work Demonstration WEB FORUM MINING BASED ON USER SATISFACTION PAGE 29 Agendas

Conclusion :  Finding answered threads in web forum is achieved by tracing user satisfaction.  Thread and sentence are classified by deriving different features.  Performance of system is increased when combining different features. WEB FORUM MINING BASED ON USER SATISFACTION PAGE 30 Conclusion and Future Work: Conclusion

Future Work :  It can be used for query based raking of thread.  It can be used for extracting answered sentences with better accuracy.  The performance of system can be increased by incorporating semantics. WEB FORUM MINING BASED ON USER SATISFACTION PAGE 31 Conclusion and Future Work: Future Work