1. © Leo Burnett

2. 2 “Moe”: April 23 “Moe” : June 6

3. 3  Population size estimation  Behavioral studies  Phenotype analysis

4. 4 Tracking devices:  Require anesthetization  Expensive  Unreliable  Broad coverage all but impossible

5. 5 1.Moe93% 2.Alice5% 3.Bob2% The “ALGORITHM” Database of zebra pictures Input – “the query” Output – “zebra ranking”

6. 6  Face recognition algorithms ▪ Early algorithms too rigid ▪ Modern algorithm could work, but are black boxes  Fingerprint recognition ▪ Generally look for well-defined features ▪ Rarely deal with occlusion, perspective skew, varying distance to camera

7. 7 © Colchester Zoo y x 1.Shape contour tracing 2.Spline function fitting 3.Query and retrieve splines

8. 8

9. 9  How much information is there in the data?

10. 10  How much information is there in the data?

11. 11  Hypothesis: width and spacing of stripes are distinctive when measured finely

12. 12 Zebras are/have:  seldom two-dimensional  frequently obscured  non uniform stripes  high tendency towards pregnancy and violence  afraid of barcode scanners © Barcodeman

13. 13 Build a solution by eliminating the problems!

14. 14 Closer to camera, more pixels for the bodyFurther away, fewer pixels for the body 10 Megapixel camera = 3648 pixels across, 2736 pixels down 1080p HD TV = 1920 pixels across, 1080 pixels down 15” MacBook Pro screen= 1440 pixels across, 900 pixels down

15. 15  Solution: measure widths relative to the previous stripe 52466466638875104

16. 16  Solution: measure widths relative to the previous stripe 524664666388751.38

17. 17  Solution: measure widths relative to the previous stripe 5246646663880.851.38

18. 18  Solution: measure widths relative to the previous stripe 52.81.41.0.951.390.851.38

19. 19

20. 20  Shear transformation flattens small amount of perspective skew [1]. Original imageWith shear transformation

21. 21  Shear transformation flattens small amount of perspective skew [1].  Shear transformation is a special case of affine transformation.  Affine transformation:  Ratios of distances along a line are preserved

22. 22  Solution: measure widths relative to the previous stripe 52.81.41.0.951.390.851.38

23. 23  Solution: measure widths relative to the previous stripe 52.81.41.0.951.390.851.38 0.81.41.00.951.390.851.38 A “strip” of stripes

24. 24 1.41.00.951.390.851.380.7 From original photograph in database: 0.831.390.851.380.7 Zebra occluded from the left side: Missed the rightmost black stripe: 1.41.00.951.390.851.98 Extremely oblique viewing angle: 1.51.11.21.71.011.661.3

25. 25  Solution: Dynamic programming  Align two strips to minimize “errors”  a.k.a. Spell-check, DNA sequence alignment, Needleman-Wunsch algorithm, Smith-Waterman algorithm, edit distance, dynamic time warping, etc.  Stripe-alignment!  occlusion = indel cost  image processing errors = indel + matching cost  stripe distortion = matching cost  strong perspective skew= matching cost  Low alignment “cost” = fewer differences in strips = zebras are very similar

26. 26 1.41.00.951.390.851.380.7 COST = 0 1.41.00.951.390.851.380.7 1.41.00.981.440.851.380.7 1.41.00.951.390.851.380.7 COST = (0.98-0.95) + (1.44-1.39) = 0.08

27. 27 For a new picture (the “query”):  Read a strip off the body at a known location  Align against all the zebra strips in the database, also from the same location  Rank zebras in the database by the alignment cost of their strips

28. 28  One click per zebra  Analogous to a barcode scanner  Handles occlusion, minor perspective skew  Can be applied to any part of the body  Computationally efficient

29. 29  20 zebras, ~6 photos per zebra = 109 pictures. “Transcription” errors

30. 30  Photos were manually identified by Rosemary at Ol’Pejeta Conservancy.  Manually coded stripes along the shoulder  For each photo, rank the closest matches using dynamic programming.  Metric: rank of the correct zebra in the list of closest matches.

31. 31 Proportion of queries at or below rank Average rank = 1.5

32. 32  No love from Zebra #3 – “01_700”  Photo 87 Zebra 3 Flank R PicID 8159 CORRECT_RANK 4  Photo 65 Zebra 3 Flank L PicID 8142 CORRECT_RANK 9  Photo 63 Zebra 3 Flank R PicID 8170 CORRECT_RANK 5  Photo 52 Zebra 3 Flank R PicID 8166 CORRECT_RANK 2 81598142 81708166

33. 33  Worst performance on this picture:

34. 34  Time to search database of 108 pictures:  0.023 seconds-- my ageing 2006 laptop  If the number of stripes on a zebra is O(1), then the time complexity of a single search is linear in the number of photographs.  1,000 pictures~ 0.2 seconds  10,000 pictures~ 3 seconds

35. 35

36. 36 Future work: 1. Build an effective user interface 2. Get field biologists to discover better ways to use it! 3. Run tests on 3,000+ pictures from January Kenya trip 4. Tweak image processing algorithms “So you see! There’s no end to the thing you might know, depending how far beyond Zebra you go.” - Dr. Seuss, Beyond Zebra References: [1] Hutchison and Barrett. Fourier-Mellin registration of line-detained tabular document images. Intl. J. Doc. Analysis 8(2):87-110, 2006. mlahiri@gmail.com