Online Learning Algorithms

Outline Online learning Framework Design principles of online learning algorithms (additive updates) Perceptron, Passive-Aggressive and Confidence weighted classification Classification – binary, multi-class and structured prediction Hypothesis averaging and Regularization Multiplicative updates Weighted majority, Winnow, and connections to Gradient Descent(GD) and Exponentiated Gradient Descent (EGD)

Formal setting – Classification Instances Images, Sentences Labels Parse tree, Names Prediction rule Linear prediction rule Loss No. of mistakes

Predictions Continuous predictions : Linear Classifiers Label Confidence Linear Classifiers Prediction : Confidence:

Loss Functions Natural Loss: Real-valued-predictions loss: Zero-One loss: Real-valued-predictions loss: Hinge loss: Exponential loss (Boosting)

Loss Functions Hinge Loss Zero-One Loss 1 1

Online Framework Initialize Classifier Algorithm works in rounds On round the online algorithm : Receives an input instance Outputs a prediction Receives a feedback label Computes loss Updates the prediction rule Goal : Suffer small cumulative loss

Margin Margin of an example with respect to the classifier : Note : The set is separable iff there exists u such that

Geometrical Interpretation Margin <<0 Margin >0 Margin >>0 Margin <0

Hinge Loss

Why Online Learning? Fast Memory efficient - process one example at a time Simple to implement Formal guarantees – Mistake bounds Online to Batch conversions No statistical assumptions Adaptive

Update Rules Online algorithms are based on an update rule which defines from (and possibly other information) Linear Classifiers : find from based on the input Some Update Rules : Perceptron (Rosenblat) ALMA (Gentile) ROMMA (Li & Long) NORMA (Kivinen et. al) MIRA (Crammer & Singer) EG (Littlestown and Warmuth) Bregman Based (Warmuth) CWL (Dredge et. al)

Design Principles of Algorithms If the learner suffers non-zero loss at any round, then we want to balance two goals: Corrective: Change weights enough so that we don’t make this error again (1) Conservative: Don’t change the weights too much (2) How to define too much ?

Design Principles of Algorithms If we use Euclidean distance to measure the change between old and new weights Enforcing (1) and minimizing (2) e.g., Perceptron for squared loss (Windrow-Hoff or Least Mean Squares) Passive-Aggressive algorithms do exactly same except (1) is much stronger – we want to make a correct classification with margin of at least 1 Confidence-Weighted classifiers maintains a distribution over weight vectors (1) is same as passive-aggressive with a probabilistic notion of margin Change is measured by KL divergence between two distributions

Design Principles of Algorithms If we assume all weights are positive we can use (unnormalized) KL divergence to measure the change Multiplicative update or EG algorithm (Kivinen and Warmuth)

The Perceptron Algorithm If No-Mistake Do nothing If Mistake Update Margin after update:

Passive-Aggressive Algorithms

Passive-Aggressive: Motivation Perceptron: No guaranties of margin after the update PA: Enforce a minimal non-zero margin after the update In particular: If the margin is large enough (1), then do nothing If the margin is less then unit, update such that the margin after the update is enforced to be unit

Aggressive Update Step Set to be the solution of the following optimization problem: Closed-form update: (2) (1) where,

Passive-Aggressive Update

Unrealizable Case

Confidence Weighted Classification

Confidence-Weighted Classification: Motivation Many positive reviews with the word best Wbest Later negative review “boring book – best if you want to sleep in seconds” Linear update will reduce both Wbest Wboring But best appeared more than boring How to adjust weights at different rates? Wboring Wbest

Update Rules The weight vector is a linear combination of examples Two rate schedules (among others): Perceptron algorithm, conservative: Passive-aggressive

Distributions in Version Space Mean weight-vector Example

Margin as a Random Variable Signed margin is a Gaussian-distributed variable Thus:

PA-like Update PA: New Update :

Weight Vector (Version) Space Place most of the probability mass in this region

Passive Step Nothing to do, most weight vectors already classify the example correctly

Aggressive Step Mean moved past the mistake line (large margin) The covariance is shirked in the direction of the new example Project the current Gaussian distribution onto the half-space

Extensions: Multi-class and Structured Prediction

Multiclass Representation I k Prototypes New instance Compute Prediction: the class achieving the highest Score Class r 1 -1.08 2 1.66 3 0.37 4 -2.09

Multiclass Representation II Map all input and labels into a joint vector space Score labels by projecting the corresponding feature vector F Estimated volume was a light 2.4 million ounces . = (0 1 1 0 … ) B I O B I I I I O

Multiclass Representation II Predict label with highest score (Inference) Naïve search is expensive if the set of possible labels is large No. of labelings = 3No. of words Estimated volume was a light 2.4 million ounces . B I O B I I I I O Efficient Viterbi decoding for sequences!

Two Representations Weight-vector per class (Representation I) Intuitive Improved algorithms Single weight-vector (Representation II) Generalizes representation I Allows complex interactions between input and output x F(x,4) =

Margin for Multi Class Binary: Multi Class:

Margin for Multi Class But different mistakes cost (aka loss function) differently – so use it! Margin scaled by loss function:

Perceptron Multiclass online algorithm Initialize For Receive an input instance Outputs a prediction Receives a feedback label Computes loss Update the prediction rule

PA Multiclass online algorithm Initialize For Receive an input instance Outputs a prediction Receives a feedback label Computes loss Update the prediction rule

Regularization Key Idea: Popular choices: If an online algorithm works well on a sequence of i.i.d examples, then an ensemble of online hypotheses should generalize well. Popular choices: the averaged hypothesis the majority vote use validation set to make a choice