1. Word Spotting DTW

2. Word Spot DTW Introduction The Basic Idea Pruning DTW
Matching Words With DTW
Experimental Results
Summary

3. Introduction
Libraries contain an enormous amount of hand-written historical documents.
They would like to make it available electronically.
such large collections can only be accessed efficiently if a searchable index exist.
The current state-of-the-art approach is to manually create an index.

4. Introduction – cont.
The quality of historical documents is degraded due to faded ink, stained paper, etc.
Traditional Optical Character Recognition (OCR) techniques that usually recognize words character-by-character, fail.

5. Introduction – cont.

6. The Basic Idea
For handwritten manuscripts written by a single author - the images of multiple instances of the same word are likely to look similar.
Word spotting idea provides an alternative approach to index generation.

7. Word Spotting
Each page in the document collection is segmented into words.
The different instances of a word are clustered together using image matching.
A human can tag the n most interesting clusters for indexing with the appropriate ASCII equivalent.

9. Matching Good matching performance can be achieved by:
A technique that skews, resizes and aligns two candidate words.
Compares the words pixel-by-pixel.
We will use DTW.

10. Pruning
Running a matching algorithm is expensive with growing collection sizes.
Pruning techniques which can discard unlikely matches are used.

11. Pruning Techniques
Pruning of word pairs based on the area and aspect ratio of their bounding boxes.
Require words to have the same number of descenders (strokes below the baseline).
The idea is to require similar pruning statistics.

12. Ascenders
Upper Baseline
Lower
Baseline
Descenders

13. DTW Used to compute a distance between two time series.
A time series is a list of samples taken from a signal ordered by time.
Naive approach: resample one of them and then compare the series sample-by-sample.
does not produce intuitive results, as it compares samples that might not correspond well.

14. DTW
Recovering optimal alignments between sample points in the two time series.
Demonstrates:
time

15. Comparison between Naive & DTW
time
Any distance (Euclidean, Manhattan, …) which aligns the i-th point on one time series with the i-th point on the other will produce a poor similarity score. i
i+2
time
A non-linear (elastic) alignment produces a more intuitive similarity measure, allowing similar shapes to match even if they are out of phase in the time axis.

16. DTW
The DTW-distance between two time series Xi Xm and Yi Yn is D(m,n).
D(i,j)= min {D(i,j-1),D(i-1,j),D(i-1,j-1)} + d(i,j)
d(i,j) varies with the application.
This calculation realizes a local continuity constraint.

17. Warping Function
Time Series A 1 is n m pk
To find the best alignment between A and B one needs to find the path through the grid
P = p1, … , ps , … , pk
ps = (is , js )
which minimizes the total distance between them.
P is called a warping function. js ps
Time Series B 1 p1

18. Time-Normalized Distance Measure
Time Series A
Time-normalized distance between A and B : 1 is n m pk
D(A , B ) =
d(ps): distance between is and js
ws > 0: weighting coefficient. js ps
Best alignment path between A and B :
P0 = (D(A , B )).
Time Series B 1 p1

19. Matching words with DTW

20. Matching words with DTW
The inter-character and intra-character spacing is subject to larger variations.
DTW offers a more flexible way compensate for these variations than linear scaling.
We first normalize the slant and skew angle of candidate images.
From each word, four features per image column are extracted and combined into a single time series.

21. Matching Words With DTW
For each image I with height h and width w, we extract a time series:
X(I) = x1….xw.
xi = f1(I,i),f2(I,i),f3(I,i),f4(I,i).
fk = four extracted features per image column.

22. Matching Words With DTW
In order to run the DTW algorithm on two time series X(I) and Y(J), we define a local distance function:
d(xi,yj ) = ∑ (fk(I,i)-fk(J,j))²
Now, the DTW algorithm can be run to determine a warping path between X and Y:
D(X,Y) = ∑ d(xik,yjk )

23. DTW Features Projection Profiles Word Profiles
Upper word profiles
Lower word profiles
Background/Ink transitions

24. Projection Profile
Projection profile capture the distribution of ink along one dimension in a word image.
A vertical projection profile is computed by summing the intensity values in each image column separately:
PP(I,c) = ∑ (255-I(r,c)) h
r=1

25. (a) original image: slant/skew/baseline-normalized, cleaned.
(b) normalized projection profile.

26. Word Profiles
Word profiles capture part of the outlining shape of a word.
Using upper and lower word profiles.
Going along the upper (lower) boundary of a word’s bounding box.
Recording for each image column the distance to the nearest “ink” pixel in that column.

27. Word Profiles
Due to a number of factors, some image columns may not contain ink pixels.
Therefore, these gaps are closed by linearly interpolating between the two closest points.

28. Upper Boundary

29. Lower Boundary

30. Background/Ink Transitions
A capture of the inner structure of a word is missing.
Records for every image column, the number of transitions from the background to ink pixels:
Determined by threshold.
nbit(I, c).

32. Experimental Results
Data sets and processing
Results

33. Data Sets And Processing
conducted on two test sets of different quality
Acceptable quality (set 1).
Very degraded quality (set 2).
Divide the test to four sets:
15 images in test set 1.
Entire test set 1.
32 images in test set 2.
Entire test set 2.

34. Test Sests

35. Results SC XOR EDM Shape context matching.
The images are aligned to compensate for shear and scale changes and then a difference image is computed.
EDM
Euclidean distance map. Larger regions are weighted more heavily.

36. Results Test set/Algorithm XOR SSD SLH SC EDM DTW A 54.14% 52.66%
42.43%
48.67%
72.61%
73.71% B
n/a
65.34% C
48.11%
49.56%
58.81% D
51.81%

37. Summary & Conclusions
DTW approach perform better than a number of other techniques.
Accuracy.
Speed.
The future work will focus on improvements in speed and accuracy.
Pruning.
Optimizations in DTW.