TEMPORAL EVENT CLUSTERING FOR DIGITAL PHOTO COLLECTIONS Matthew Cooper, Jonathan Foote, Andreas Girgensohn, and Lynn Wilcox ACM Multimedia ACM Transactions on Multimedia Computing, Communications and Application

OUTLINE Introduction Feature extraction Clustering techniques Supervised event clustering Unsupervised event clustering Clustering goodness criteria Experimental result Conclusion

I NTRODUCTION Users navigate their photos Temporal order Visual content Associate time and content with the notion of a specific “event” Photos associated with an event often exhibit little coherence in terms of either low-level image features or visual similarity photographs from the same event are taken in relatively close proximity in time

B ASIC CONCEPTS --- E VENT Events are naturally associated with specific times and places. Birthday party Vacation Wedding

B ASIC CONCEPTS --- EXIF & CBIR Exchangeable Image File (EXIF): Time, Location, Focal length, Flash, etc. => Season, place, weather, indoor/outdoor,etc Content-based Image Retrieval (CBIR): Color, Texture, Shape, etc. => Face & Fingerprint Recognition,etc Metadata

FEATURE EXTRACTION EXIF headers are processed to extract the timestamp The N photos in the collection are then ordered in time so the resulting timestamps, {t n :n = 1,..., N},satisfy t 1 ≤ t 2 ≤ … ≤ t N Time difference between indices (photos) is nonuniform t 1 t 2 t 3 t 4 t 5 t 6 ….. t

FEATURE EXTRACTION Computing similarity matrices S K temporal similarity matrix

FEATURE EXTRACTION Computing similarity matrix low-frequency discrete cosine transform (DCT) coefficients from each photo using the cosine distance measure content-based similarity matrix

FEATURE EXTRACTION computing novelty scores K=1000K=10000K= peaks in the novelty scores = cluster boundaries between contiguous groups of similar photos

CLUSTERING TECHNIQUES Supervised event clustering Based on LVQ Unsupervised event clustering Scale-space analysis of the raw timestamp data Temporal Similarity Analysis Combining Time and Content-Based Similarity

Supervised event clustering Let K take M values : K ≡ {K 1,..., K M } Define the M × N matrix N(j,i) = ν K j (i), where Based on LVQ (Learning Vector Quantization) [Kohonen 1989] LVQ codebook discriminates between the two classes “event boundary” and “event interior.” The codebook vectors for each class are used for nearest- neighbor classification of the novelty features for each photo in the test set.

Supervised event clustering In the training phase, a codebook is calculated using an iterative procedure Each step Nearest codebook vector to each training sample is determined shifted toward or away the training sample If Nx and Mc are in the same class If Nx and Mc aren’t in the same class

Supervised event clustering ALGORITHM 1 (LVQ-BASED PHOTO CLUSTERING). (1) Calculate novelty features from labeled sorted training data for each scale K : (i) compute the similarity matrix S K (ii) compute the novelty score ν K (2) Train LVQ using the iterative procedure (3) Calculate novelty features for the testing data for each K (i) compute the similarity matrix S K (ii) compute the novelty score ν K (4) Classify each test sample’s novelty features N i using the LVQ codebook and the nearest-neighbor rule.

U NSUPERVISED EVENT CLUSTERING scale-space analysis operate on the raw timestamps T 0 = [t 1,..., t N ] so that T 0 (i) = t i ALGORITHM 2 (SCALE-SPACE PHOTO CLUSTERING). (1) Extract timestamp data from photo collection: {t 1,..., t N }. (2) For each σ in descending order: (i) compute T σ (ii) detect peaks in T σ, tracing peaks from larger to smaller scales (decreasing σ).

UNSUPERVISED EVENT CLUSTERING Temporal Similarity Analysis Locate peaks at each scale by analysis of the first difference of each novelty scores ν K, proceeding from coarse scale to fine (decreasing K) To build a hierarchical set of event boundaries, we include boundaries detected at coarse scales in the boundary lists for all finer scales. checkerboard kernel used to compute the novelty features

UNSUPERVISED EVENT CLUSTERING Combining Time and Content-Based Similarity constructed a content-based matrix S C using low- frequency DCT features and the cosine distance if |t i -t j | > 48h others if |t i -t j | > 48h others

CLUSTERING GOODNESS CRITERIA Peak detection at each scale K results in a hierarchical set of candidate boundaries Subset must be selected to define the final event clusters Three different automatic approaches Similarity-Based Confidence Score Boundary Selection via Dynamic Programming BIC-Based Boundary Selection

Similarity-Based Confidence Score Detected boundaries at each level K, B K = {b 1,..., b nK }, indexed by photo: B K ⊂ {1,..., N} average intracluster similarity between the photos within each cluster average intercluster similarity between photos in adjacent clusters

Boundary Selection via Dynamic Programming Reduced complexity Begin with the set of peaks detected from the novelty features at all scales Cost of the cluster between photos b i and b j

Boundary Selection via Dynamic Programming Optimal partitions with m boundaries based on the optimal partition with m−1 boundaries First, optimal partitions are computed with two clusters E F (j,m) is the optimal partition of the photos with cardinality m

Boundary Selection via Dynamic Programming Number of clusters increases, the total cost of the partition decreases monotonically Selecting the optimal number of clusters, M ∗, based on the total partition cost

BIC-Based Boundary Selection This method is based on the Bayes information criterion (BIC) [Schwarz 1978] Assumption timestamps within an event are distributed normally around the event mean log-likelihood of the two segment model Log-likelihood of the single segment model and the penalty term λ is 2,since we describe each segment using the sample mean μ,and variance, σ 2

BIC-BASED BOUNDARY SELECTION Employ the hierarchical coarse-to-fine approach At each scale, we test only the newly detected boundaries (undetected at coarser scales) Add the boundaries for which the left side exceeds the right side

ALGORITHM 3 (SIMILARITY- BASED PHOTO CLUSTERING) (1) Extract and sort photo timestamps, {t1,..., tn}. (2) For each K in decreasing order (i) compute the similarity matrix S k (ii) compute the novelty score ν K (iii) detect peaks in ν K (iv) form event boundary list using event boundaries from previous iterations and newly detected peaks (3) Determine a final boundary subset of collected boundaries over all scales considered according to one of the methods : (a) the confidence score (b) the DP boundary selection approach (c) the BIC boundary selection approach

EXPERIMENTAL RESULT Run Times for Different Size Photo Collections The times are in seconds No Conf. indicates times for Steps 1 and 2 BIC peak selection (BIC) Dynamic programming peak selection (DP) similarity-based peak selection (Conf.) Doubling the number of photos(N),the time for the segmentation step(No Conf.) increases linearly, while including the confidence measure (Conf.) incurs a polynomial cost.

EXPERIMENTAL RESULT Compare the event clustering performance of eleven systems on two separate photo collections Collection I consists of 1036 photos taken over 15 months Collection II consists of 413 photos taken over 13 months The first four algorithms in the table are “hand-tuned” to maximize performance. The remaining algorithms are fully automatic.

EXPERIMENTAL RESULT Precision indicates the proportion of falsely labeled boundaries: Recall measures the proportion of true boundaries detected: The F-score is a composite of precision and recall:

EXPERIMENTAL RESULT

The adaptive-thresholding algorithms exhibited high recall and low precision on both test sets, even with manual tuning Scale-space and the two similarity-based approaches demonstrated more consistent performance and traded off precision and recall more evenly

CONCLUSION Employed the automatic temporal similarity- based method Does not rely on preset thresholds or restrictive assumptions As photo collections with location information become available, we hope to extend our system to combine temporal similarity, content-based similarity, and location-based similarity. The automatic methods’ performance exceeded that of manually tuned alternatives in our testing, and have been well received by users of our photo management application.