SIGIR 2008 Yandong Liu, Jiang Bian, Eugene Agichtein from Emory & Georgia Tech University

Definition

Features  Question  Question-Answer Relationship  Asker User History  Answerer User History  Category Features  Textual Features

Formally a two-class classification problem but primarily focus on the satisfied class.

 Amazon’s paid rater service  Mechanical Turk

Datasets

Setting  Methods  Human  Heuristic  Baseline  Evaluation Metrics  Precision  Recall  F1  Accuracy  ASP  ASP_SVM  ASP_RandomForest  ASP_C4.5  ASP_Boosting  ASP_NaiveBayes

Result

Selected features

 Online vs. Offline

 Feature Ablation

 Textual Features

 Past Experience

Conclusion  First to quantify and predict asker satisfaction  Shown importance of asker history is this  Our system outperform human assessors