1. Properties of Histograms and their Use for Recognition Stathis Hadjidemetriou, Michael Grossberg, Shree Nayar Department of Computer Science Columbia University New York, NY 10027

2. Motivation Histogramming is a simple operation:

3. Histograms have been used for: –Object recognition [Swain & Ballard 91, Stricker & Orengo 95] –Indexing from visual databases [Bach et al, 96, Niblack et al 93, Zhang et al 95] Histogram advantages: –Efficient –Robust [Chatterjee, 96] Histogram limitation: –Do not represent spatial information Motivation

4. Overview Image transformations that preserve the histogram Image structure through the multiresolution histogram Multiresolution histogram compared with other features

5. Invariance of Histogram with Discontinuous Transformations Cut and rearrange regions Shuffle pixels

6. Invariance of Histogram with Continuous Transformations Rotation Shear

7. What is the complete class of continuous transformations that preserves the histogram?

8. Model for Image Continuous domain Image: Map from continuous domain to intensities

9. Model for Histogram U Histogram count for bin U ≡ Area bounded by level sets U U

10. Vector fields, X, morph images [Spivak, 65] : Continuous Image Transformations

11. Gradient Transformations Original

12. Histograms of Gradient Transformations

13. Condition 1: Histogram Preservation and Local Area Histogram preserved  Local area preserved [Hadjidemetriou et al, 01]  …… 

14. Condition 2: Local Area Preservation and Divergence Divergence is rate of area change per unit area Local area preserved  divergence is zero [Arnold, 89] Small region

15. Fields along isovalue contours of an energy function F Isovalue contours Hamiltonian Fields Flow of incompressible fluids [Arnold, 89] Hamiltonian flow

16. Computing Hamiltonian Fields Gradient of 1.Compute gradient of F2.Rotate gradient pointwise 90 0 Hamiltonian of

17. Transformations preserve histogram of all images  corresponding field is Hamiltonian [Hadjidemetriou et al, CVPR, 00, Hadjidemetriou et al, IJCV, 01] Theorem Condition 3: Divergence and Hamiltonian Fields Divergence of field is zero  Hamiltonian field [Arnold, 89]

18. Examples of Hamiltonian Transformations Linear: Translations, rotations, shears Original

19. Examples of Hamiltonian Transformations

20. Border Preserving Hamiltonian Transformations 0 w h

21. Examples of Windowed Hamiltonian Transformations

22. Identical histograms:

23. Weak Perspective Projection Depth (z) causes scaling [Hadjidemetriou et al, 01] Planar object tilt  causes shearing and scaling

24. The Hamiltonian transformations is the complete class of continuous image transformations that preserves the histogram

25. How can spatial information be embedded into the histogram?

26. Previous work on Features combining the Histogram with Spatial Information Local statistics: −Local histograms [Hsu et al, 95, Smith & Chang, 96, Koenderink and Doorn, 99, Griffin, 97] −Intensity patterns [Haralick,79, Huang et al, 97] One histogram: −Derivative filters [Schiele and Crowley, 00, Mel 97] −Gaussian filter [Lee and Dickinson, 94] Many techniques are ad-hoc or not complete

27. Multiresolution Histogram G (l  2 )

28. Limitations of Histograms Database of synthetic images with identical histograms [Hadjidemetriou et al, 01]

29. Matching with Multiresolution Histograms Match under Gaussian noise of st.dev. 15 graylevels:

30. Matching with Multiresolution Histograms Match under Gaussian noise of st.dev. 15 graylevels:

31. How is Image Structure Encoded in the Multiresolution Histogram? ? Image structure Differences of histograms L Image h( L * G (l)) Multiresolution histogram

32. Histogram Change with Resolution and Spatial Information Bin j: Spatial information Averages of bins: where P j are proportionality factors ill-conditioned well-conditioned

33. Histogram Change with Resolution and Fisher Information Measures = Generalized Fisher information measures of order q [Stam, 59, Plastino et al, 97] ≡ L is the image = D is the image domain Averages

34. Image Structure Through Fisher Information Measures ? Image structure Differences of histograms L Image h( L * G (l)) Multiresolution histogram Fisher information measures (Analysis) P JqJq

35. Shape Boundary and Multiresolution Histogram Superquadrics:  =0.56 Histogram change with l is higher for complex boundary  =1.00  =1.48  =2.00  =6.67

36. Texel Repetition and Multiresolution Histogram Histogram change with l is proportional to number of texels (analytically)

37. Texel Placement and Multiresolution Histogram Std. dev. of perturbation Histogram change with l decreases with randomness

38. Matching Algorithm for Multiresolution Histograms Burt-Adelson image pyramid Cumulative histograms L 1 norm Differences of histograms between consecutive image resolutions Concatenate to form feature vector

39. Histogram Parameters Bin width Smoothing to avoid aliasing Normalization: −Image size −Histogram size 179x179 89x89 44x44 5x5 ……

40. Database of Synthetic Images 108 images with identical histograms [Hadjidemetriou et al, 01]

41. Sensitivity of Matching for Synthetic Images

42. Database of Brodatz Textures 91 images with identical equalized histograms: 13 textures different rotations

43. Match Results for Brodatz Textures Match under Gaussian noise of st.dev. 15 graylevels:

44. Sensitivity of Class Matching for Brodatz Textures

45. Database of CUReT Textures 8,046 images with identical equalized histograms : 61 materials under different illuminations [Dana et al, 99]

46. Match Results for CUReT Textures Match under Gaussian noise of st.dev. 15 graylevels:

47. Match Results for CUReT Textures Match under Gaussian noise of st.dev. 15 graylevels:

48. Sensitivity of Class Matching for CUReT Textures 100 randomly selected images per noise level

49. Embed spatial information into the histogram with the multiresolution histogram

50. How well does the multiresolution histogram perform compared to other image features?

51. Comparison of Multiresolution Histogram with Other Features Multiresolution histogram: −Variable bin width −Histogram smoothing Fourier power spectrum annuli [Bajsky, 73] Gabor features [Farrokhnia & Jain, 91] Daubechies wavelet packets energies [Laine & Fan, 93] Auto-cooccurrence matrix [Haralick, 92] Markov random field parameters [Lee & Lee, 96]

52. Comparison of Effects of Transformations on the Features FeatureTranslationRotation Uniform Scaling 1 Fourier power spectrum annuli invariantrobustequivariant 2 Gabor featuresinvariantsensitiveequivariant 3 Daubechies wavelet energies sensitive 4 Multiresolution histograms invariant equivariant 5 Auto-cooccurrence matrix invariantrobustequivariant 6 Markov random field parameters invariantsensitive

53. Comparison of Class Matching Sensitivity of Features Database of Brodatz textures

54. Comparison of Class Matching Sensitivity of Features Database of CUReT textures 100 randomly selected images per noise level

55. Sensitivity of Features to Matching Feature Gaussian Noise Database size,# classes Illuminati- on Parameter selection Fourier power spectrum annuli sensitive robustvery sensitive Gabor featuresrobust sensitive Daubechies wavelet energies sensitiverobust Multiresolution histogram robust Auto-cooccurrence matrix very sensitive Markov random field parameters very sensitive sensitiveN/A

56. Comparison of Computation Costs of Features 1 Markov random field parameters O(n( 2 -1) 2 -( 2 -1) 3 /3) 2 Gabor features  ( (logn+1)nlogn) 3 Fourier power spectrum annuli O(n 3/2 ) 4 Auto-cooccurrence matrix O(n  ) 5 Wavelet packets energies O(n l) 6 Multiresolution histograms  n  n- number of pixels - window width l- resolution levels Decreasing cost

57. The multiresolution histogram compared to other image features is robust and efficient

58. Summary and Discussion Hamiltonian transformations preserve features based on: −Histogram −Image topology Multiresolution histograms: −Embed spatial information Comparison of multiresolution histograms with other features: −Efficient and robust

59. Recognition of 3D Matte Polyhedral Objects Face histograms: –Magnitude scaled by tilt angle (   ) –Intensity scaled by illumination (a i ) In an object database find [Hadjidemetriou et al, 00] : –Object identity –Pose (   ) –Illumination (a i ) Total histogram: Sum of h(i) of visible faces

60. A Simple Experiment Object 1:Object 2: Object 3: Object 4: ObjectTestsRank=1Rank=2 Total40382

61. Shape Elongation and Multiresolution Histogram Elongation: St. dev. along axes:  x,  y. Gaussian: Sides of base : r x, r y. Pyramid: Elongation: (analytically) Histogram change with l

62. Are all image resolutions equally significant?

63. Resolution Selection with Entropy of Multiresolution Histograms Entropy-resolution plot [Hadjidemetriou et al, ECCV, 02] : –Global –Non-monotonic …. l …………………… … …………

64. Examples of Entropy-Resolution Plots

65. The entropy of the multiresolution histogram can be used to detect significant image resolutions

66. Future Work Histogram preserving fields: −Transformations over limited regions −Sensitivity of features to image transformation Multiresolution histograms: −Color images −Rotational variance with elliptic Gaussians Resolution selection: −Preprocessing step −Non-monotonic features