Revisiting Output Coding for Sequential Supervised Learning Guohua Hao & Alan Fern School of Electrical Engineering and Computer Science Oregon State University Corvallis, OR, U.S.A. TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AA A A A A A AA

Scalability in CRF Training Linear Chain CRF model Inference in Training  partition function : forward-backward algorithm  Maximizing over label sequences: Viterbi algorithm  Complexity of both: Repeated inference in training  Computationally demanding  Can not scale to large label sets y t-1 y t+1 ytyt X t-1 X t+1 XtXt

Recent Work of Focus Sequential Error Correcting Output Coding (SECOC)  Error Correcting Output Coding (ECOC) ClassCode Word b1b1 b2b2 …bnbn C1C1 10…1 C2C2 00…0 ………...… CmCm 01…1 classifierh1h1 h2h2 hnhn

 Extension to CRF model x t-1 xtxt x t+1 y t-1 ytyt y t+1 y t-1 k ytkytk y t+1 k  Decoding y t-1 1 yt1yt1 y t+1 1 y t-1 n ytnytn y t+1 n

Representational Capacity of SECOC Intuitively, it feels that training each binary CRF independently will not be able to capture rich transition structure Counter-example to independent training Our hypothesis: when the transition structure is critical, independent training will not do as well 1 32 Y = b 1 (Y) = b 1 (Y)* = b 2 (Y) = b 3 (Y) = yb1b1 b2b2 b3b

Our Method—Cascaded SECOC Help capture the transition structure For problems where a transition model is critical, we hope to see cascade training outperform independent training For problem where a observation model is more informative but the sliding window is small. Large sliding window will dominate the effect of cascade training Previous binary predictions

Experimental Results Base CRF training algorithms  Gradient Tree Boosting (GTB)  Voted Perceptron (VP) Methods for comparison  iid-- Non sequential ECOC  i-SECOC--Independent SECOC  c-SECOC (h)--Cascaded SECOC w/ history length h  Beam search Synthetic Data Sets  Generation by HMM  “Transition” Data Set  “Both” Data Set

Nettalk Data Set (134 labels)

Noun Phrase Chunking (NPC) (121 labels) Synthetic Data Sets (40 labels)

Comparing to Beam Search

Summary i-SECOC can perform poorly when explicitly capturing complex transition models is critical c-SECOC can improve accuracy in such situations by using cascade features Performance of c-SECOC can depends strongly on the base CRF algorithm; Algorithms capable of capturing complex (non-linear) feature interactions are preferred When using less powerful base CRF learning algorithms, other approaches (e.g. beam search) can outperform c- SECOC

Future Directions Efficient validation procedure for selecting cascade history length Incremental generation of code words Wide comparison of methods for dealing with large label ses Acknowledgements We thank John Langford for discussion of the counter example to independent SECOC and Thomas Dietterich for his support. This work was supported by NSF grant IIS