1. Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign

2. Text Categorization An archetypical Multi-Class Classification (MCC) problem  F : X → Y  a document, d ∈ X, a collection of classes Y = {c 1, c 2,..., c N } Sports Health Business … Science C1 C2 C3 … C N 2

3. Motivation: what are we missing? Class labels (Y) contain information which can help classification How can we explore the label space? C1 C2 C3 … C N Sports Health Business … Science 3

4. Car Navigation Command Classification 4 XY

5. Aspect Variables 5 X Y find nearest restaurant Show me where I can eat nearby Find nearest restaurant ActionDetailModifierTopicManner z1z1 z2z2 z3z3 z4z4 z5z5 Null

6. Significance of the Aspect Variables Predicting better aspects implies predicting better class labels Adding constraints to the aspect space Predicting previously unobserved labels 6 If Topic = “restaurant”, then Action ≠ “turn” Observed Label 1. turn on the radio 2. GPS navigation Unobserved Label turn on GPS

7. Outline Car Command Text Categorization Task  Data and aspects  Unobserved labels Constrained Conditional Model (CCM)  Aspects variables to introduce constraints  Objective function  Training and Inference Experimental Results  Standard multiclass classification setting  Predicting Unobserved Labels Conclusion 7

8. x1x1 x6x6 x2x2 x5x5 x4x4 x3x3 x7x7 X Y y1y1 Adding Constraints by Hidden Aspects Intuition: introduce structure on hidden variables

9. Adding Constraints Through Hidden Aspects x1x1 x6x6 x2x2 x5x5 x4x4 x3x3 x7x7 Y Z1 Z2 Z4 Z3 Z5 y1y1 Use constraints to capture the dependencies X

10. Objective Function of CCM 10 Weight Vector for “local” learners Aspect functions Penalty for violating the constraint. How far away is y from a “legal” assignment

11. 11 Training and Inference Learning + Inference (L+I)  Ignore constraints during training Inference Based Training (IBT)  Consider constraints during training References to CCM (aka ILP formulation)  Roth&Yih04, Has been shown useful in the context of many NLP problems: SRL, Summarization; Co-reference; Information Extraction; Transliteration  07; Punyakanok et.al 05,08; Chang et.al 07,08; Clarke&Lapata06,07; Denise&Baldrige07;Goldwasser&Roth'08; Martin,Smith&Xing'09

12. 12 Learning + Inference x1x1 x6x6 x2x2 x5x5 x4x4 x3x3 x7x7 Z1 Z2 Z5 Z4 Z3 f1(x)f1(x) f2(x)f2(x) f3(x)f3(x) f4(x)f4(x) f5(x)f5(x) X Y Learning + Inference (L+I) Learn models independently Learning + Inference (L+I) Learn models independently

13. 13 1111Y’Local Predictions Inference Based Training Example: Perceptron-based Global Learning x1x1 x6x6 x2x2 x5x5 x4x4 x3x3 x7x7 f1(x)f1(x) f2(x)f2(x) f3(x)f3(x) f4(x)f4(x) f5(x)f5(x) X Y 11 YTrue Global Labeling 111 Y’ Apply Constraints:

14. Outline Car Command Text Categorization Task  Data and aspects  Unobserved labels Constrained Conditional Model (CCM)  Aspects variables to introduce constraints  Objective function  Training and Inference Experimental Results  Standard multiclass classification setting  Predicting Unobserved Labels Conclusion 14

15. Car Navigation Command Classification 15 XY

16. Evaluation Metrics Standard Accuracy  The percentage of correctly labeled examples Weighted Aspect-based Metric (WAM)  A weighted Hamming distance computed at the aspect level 16

17. Experiments and Evaluation Standard MCC Setting 17 AlgorithmAccuracy (%)WAM(%) Baseline67.8486.14 MCC(L+I) 71.1889.65 Error Reduction (%) 10.3925.32 AccuracyFless Kappa Human Annotation 75%0.764

18. Experiments and Evaluation Standard MCC Setting 18 AspectsCCMBaselineError Reduction(%) Topic 86.1481.5524.88 Action 88.3182.7232.35 Manner 89.9887.3520.79 Modifier 91.1589.5115.64 Detail 92.6889.5929.68

19. Experiments and Evaluation Predicting Unobserved Labels 19 AlgorithmAccuracy (%)WAM(%) Baseline0.0058.43 MCC(L+I) 28.1670.27 Error Reduction (%) 28.1628.48 Unobserved Label turn on GPS Observed Label 1. turn on the radio 2. GPS navigation

20. Conclusion Summary  Text Categorization with a meaningful, structured label space  A model that exploits the structure by adding hidden aspect variables Adding constraints and reformulating the task as a structure prediction problem Predicting unobserved new labels 20

21. Thank You! AND Questions? 21