Feature extraction for change detection Can you detect an abrupt change in this picture? Ludmila I Kuncheva School of Computer Science Bangor University Answer – at the end

Plan 1.Zeno says there is no such thing as change... 2.If change exists, is it a good thing? 3.Context or nothing! 4.Feature extraction for change detection – PCA backwards?

Zeno of Elea (ca. 490–430 BC) “If everything, when it occupies an equal space, is at rest, and if that which is in locomotion is always occupying such a space at any moment, the flying arrow is therefore motionless.” – as recounted by Aristotle, Physics VI:9, 239b5 No motion, no movement, NO CHANGE Zeno’s Paradox of the Arrow

Does change exist? Zeno says ‘no’...

Nonetheless... Change Types Possible applications: fraud detection market analysis medical condition monitoring network traffic control Univariate detectors (Control charts): Shewhart's method CUSUM (CUmulative SUM) SPRT (Wald's Sequential Probability Ratio Test)

2 approaches Use an adaptive algorithm (No need to identify the type of change or detect change explicitly) Detect change (Update/re-train the algorithm if necessary) Labelled data Unlabelled data

Data (all features) Labels are available Classifier Distribution modelling Error rate Change statistic threshold Change/ NO change Classification

Data (all features) Labels are available Labels are NOT available Classifier Distribution modelling Error rate Change statistic threshold Change/ NO change Data (all features) Feature EXTRACTOR Distribution modelling Change statistic threshold Change/ NO change Features multidimensional

Data (all features) Labels are available Labels are NOT available Classifier GMM HMM Parzen windows kernel methods martingales Error rate threshold Change/ NO change Data (all features) Feature EXTRACTOR clustering kernel methods GMM kd-trees Hotelling threshold Change/ NO change Features

A change in the (unconditional) data distribution will: 1.render the classifier useless 2.make no difference to the classification performance 3.improve the classification performance Classification

A change in the (unconditional) data distribution will: 1.render the classifier useless 2.make no difference to the classification performance 3.improve the classification performance Vote, please!

A change in the (unconditional) data distribution will: 1.render the classifier useless 2.make no difference to the classification performance 3.improve the classification performance Vote, please!

Classification No change in the (unconditional) data distribution will: 1.render the classifier useless 2.make no difference to the classification performance 3.improve the classification performance

No change in the (unconditional) data distribution will: 1.render the classifier useless 2.make no difference to the classification performance 3.improve the classification performance Vote, please!

Classifier ensembles Brain-computer interface MathWorks products My scope of interest Literature

Change may or may not cause trouble...

Is there a change ?

mean (moving average) mean – 2std changes Shewhart with threshold 2 sigma Yes!

Is there a change ? No!

Is there a change?

Yes, for the purposes of “Spot the difference”. No, as this is a bee with a flower in the sun.

Is there a change? No!

Is there a change? sin(10x) * randn sin(20x) * randn Yes!

change detection

Change does not exist out of context!

ENTER Feature Extraction

Context: Amplitude variability Feature: AMPLITUDE

Context: Time series patterns in a fixed window. Feature: A PATTERN IN A FIXED WINDOW

Context: Children’s puzzle Feature: PIXEL B/W VALUE Context: Frequency variability Feature: FREQUENCY sin(10x) * randnsin(20x) * randn

Suppose that CONTEXT is not available. Principal Component Analysis (PCA) captures data variability. Then why not use PCA here? Labels are NOT available Data (all features) Feature EXTRACTOR Distribution modelling Change statistic threshold Change/ NO change Features

PCA intuition: The components corresponding to the largest eigen values are more important

But is this the case for change detection? Distributions are similar (small sensitivity to change) Distributions are different (large sensitivity to change) PC1 PC2 Holds for “blind”: Translation Rotation Variance change... Kuncheva L.I. and W.J. Faithfull, PCA feature extraction for change detection in multidimensional unlabelled data, IEEE Transactions on Neural Networks and Learning Systems, 25(1), 2014, 69-80

Some experiments: 1.Take a data set with n features 2.Sample randomly “windows” W1 and W2 with K objects in each window. 3.Calculate PCA from W1. Choose a proportion of explained variance and use the remaining (low- variance) components. 4.Generate a random integer k between 1 and n 4(a)Shuffle VALUES Choose randomly k features. For each chosen feature, shuffle randomly the values for this feature in window W2. 4(b)Shuffle FEATURES Choose randomly k features. Randomly permute the respective columns in window W2. 5.Transform W2 using the calculated PC and keep the low-variance components. 6.Calculate the CHANGE DETECTION CRITERION between W1 and W2. Store as NEGATIVE INSTANCE (no change).

Some experiments: 1.Take a data set with n features 2.Sample randomly “windows” W1 and W2 with K objects in each window. 3.Calculate PCA from W1. Choose a proportion of explained variance and use the remaining (low- variance) components. 4.Generate a random integer k between 1 and n 4(a)Shuffle VALUES Choose randomly k features. For each chosen feature, shuffle randomly the values for this feature in window W2. 4(b)Shuffle FEATURES Choose randomly k features. Randomly permute the respective columns in window W2. 5.Transform W2 using the calculated PC and keep the low-variance components. 6.Calculate the CHANGE DETECTION CRITERION between W1 and W2. Store as POSITIVE INSTANCE (change).

Run 100 times for POS and 100 for NEG to get the ROC curve for a given data set. Run 100 times for POS and 100 for NEG without applying PCA to get the ROC curve for a given data set. Use the Area Under the Curve (AUC), however disputed this might have become recently... Larger AUC corresponds to better change detection

VALUE shuffle

FEATURE shuffle

PCA - use the least relevant components!?

Conclusion 1.Change detection may be harmful, beneficial or indifferent to classification performance 2.Change does not exist out of context, therefore GENERIC algorithms for change detection are somewhat pointless... 3.Feature extraction for change detection may not follow conventional intuition.

Can you detect an abrupt change in this picture? Remember my little puzzle?