1. A System for Observing and Recognizing Objects in the Real World J.O. Eklundh, M. Björkman, E. Hayman Computational Vision and Active Perception Lab Royal Institute of Technology (KTH)

2. Vision in the Real World: Attending, Foveating and Recognizing Objects Motivation An autonomous agent moving about in a dynamic indoor environment, performing tasks such as finding, picking up and delivering known objects or classes of objects. What should the vision system of such an agent be capable of?

3. Vision in the Real World: Attending, Foveating and Recognizing Objects Capabilities “ where” what attention segmentation recognition what Should be dealt with jointly. Bootstrapping?

4. Vision in the Real World: Attending, Foveating and Recognizing Objects Recognition and Categorization Feifei et al 03

5. Vision in the Real World: Attending, Foveating and Recognizing Objects A robot looking at a table at 1.5 m Objects subtend only a fraction of the scene and are not centered (no attentional step)

6. Vision in the Real World: Attending, Foveating and Recognizing Objects Approach - themes 3D cues relevant in the scene Motion and stereo used for bootstrapping Integration of multiple cues A system interacting with the environment Fast processes and anytime algorithms desirable

7. Vision in the Real World: Attending, Foveating and Recognizing Objects The f-g-s problem Segmenting 3D objects from the background Computing motion, depth and ego-motion Acquiring appearance models Issues: Combining cues Demonstrate simple algorithms that suffice Monocular as well as binocular cases

8. Vision in the Real World: Attending, Foveating and Recognizing Objects Cue integration for f-g-s First, the dynamic monocular case Problem: classifying pixels as being foreground or background (or into layers) Cues: motion, colour, contrast + prediction (temporal continuity) Inference problem: observations from different spaces to be combined

9. Vision in the Real World: Attending, Foveating and Recognizing Objects Integration Two approaches: - probabilistic: likelihood of observing data, given a model of each layer - voting: each cue decides independently, form weighted combination Algorithm Online initialization of colour + texture models –Use segmentation from motion to train distributions Suppress unreliable cues Sequential algorithm

10. Vision in the Real World: Attending, Foveating and Recognizing Objects Related work Khan & Shah CVPR’01 Triesch & von der Malsburg ICAFGR’00 Spengler & Schiele ICVS’01 Toyama & Horvitz ACCV’00 Kragic & Christensen ICRA’99 Belongie et al ICCV’98 … Similarity to tracking

11. Vision in the Real World: Attending, Foveating and Recognizing Objects Voting The likelihood of observation from cue k at pixel i given model of layer j : Posterior probability of layer j : We set

12. Vision in the Real World: Attending, Foveating and Recognizing Objects Probabilistic fusion Assume independent observations The total likelihood of the observations given the combined model : The posterior estimate of layer membership: An independent opinion pool

13. Vision in the Real World: Attending, Foveating and Recognizing Objects Illustrations Assume a distribution for each layer for each cue f.g. model b.g. model 8 f ( Z | f.g ) = 8 0.4 f ( Z | b.g ) = 0.4 Observation: Z

14. Vision in the Real World: Attending, Foveating and Recognizing Objects Cue combination: Weighted voting Each cue makes independent decision Combine using weighted sum Assuming equal weights: f.g.b.g. Colour8.00.4 Motion0.20.3

15. Vision in the Real World: Attending, Foveating and Recognizing Objects Cue combination: Probabilistic Compute total likelihood of observations for each layer Classify using Bayes’ Rule Assuming uniform priors: f.g.b.g. Colour8.00.4 Motion0.20.3

16. Vision in the Real World: Attending, Foveating and Recognizing Objects Pros and cons Simple! Easy to combine observations from very different spaces Not obvious how cues interact Graded output Mathematically well-founded One cue can easily dominate over others Almost binary output Voting: Probabilistic:

17. Vision in the Real World: Attending, Foveating and Recognizing Objects Training and adaptation Start with segmentation just from motion Use this to train colour and contrast distributions –Use EM algorithm to train Gaussian mixture models Subsequently adapted online –Recompute models from current data –Update model as weighted sum over time window (Raja et al ECCV’98)

18. Vision in the Real World: Attending, Foveating and Recognizing Objects Suppressing unreliable cues Cues unreliable during training No independent motion poor motion segmentation Unreliable in the past probably unreliable now! (Triesch and von der Malsburg ICAFGR’00) Mechanisms: Voting:weights Probabilistic:hyper-priors

19. Vision in the Real World: Attending, Foveating and Recognizing Objects The effect of hyper-priors Orginal pdf’sAfter marginalization

20. Vision in the Real World: Attending, Foveating and Recognizing Objects Results OriginalProbabilisticVoting Degree of membership to foreground

21. Vision in the Real World: Attending, Foveating and Recognizing Objects Cue combination Motion ColourPrediction Texture (contrast) Combined Probabilistic Combined Voting

22. Vision in the Real World: Attending, Foveating and Recognizing Objects Results OriginalForeground

23. Vision in the Real World: Attending, Foveating and Recognizing Objects Results: Probabilistic cue integration OriginalForeground mask

24. Vision in the Real World: Attending, Foveating and Recognizing Objects The cues Motion ColourPrediction Texture (contrast) Combined

25. Vision in the Real World: Attending, Foveating and Recognizing Objects Epipolar geometry Non-retinal info. Image features Disparity map Ego-motion Independent motion Additional retinal info. Regions of interest Fixation point Top-down control Process overview

26. Vision in the Real World: Attending, Foveating and Recognizing Objects Calibration Relative orientations has to be known to relate disparities to depths simplify estimation of disparities xyxy (1+x )  – y r xy  + r + x r 2 z zy =+ 1 z 1 – x t - y t Using corner features and optical flow model Unstable process => be careful We first assume r and r to be zero. zy

27. Vision in the Real World: Attending, Foveating and Recognizing Objects More examples

28. Vision in the Real World: Attending, Foveating and Recognizing Objects Final output

29. Vision in the Real World: Attending, Foveating and Recognizing Objects Many objects of interest static. Harder! Motion included in full system; now only stereo The cues in these examples –Stereo data - exist along contours –Color data/appearance between contours 3D Cues: stereo and motion

30. Vision in the Real World: Attending, Foveating and Recognizing Objects Proposed system structure Technically by a combination of wide field and foveal cameras A wide field for attention, recognition in foveated view Problem: transfer from wide field to foveal view Steps: Divide scene into 3D objects Select objects through attention (e.g.hue and expected size) Fixate (and track) object of interest Recognize objects in foveal view

31. Vision in the Real World: Attending, Foveating and Recognizing Objects Processes Recognition Attention Segmen- tation Hypotheses Knowledge Adaptation Shape and size Knowledge Region of Interest Where What

32. Vision in the Real World: Attending, Foveating and Recognizing Objects Flow of information Left SegmentationGlobal hueSIFT featuresLocal hue Attention Gaze direction Recognition Registration LeftRight FixationCalibration Wide fieldFoveal

33. Vision in the Real World: Attending, Foveating and Recognizing Objects Figure-ground segmentation Disparity map is sliced into layers. Widths are set to that of requested object.

34. Vision in the Real World: Attending, Foveating and Recognizing Objects Figure-ground segmentation Disparities using SAD correlations. Segmentation based on slicing the 3D world. BinoCuesBinoAttn

35. Vision in the Real World: Attending, Foveating and Recognizing Objects Hue based attention Local hue histograms correlated with that of requested object. Fast implementation using rotating sums.

36. Vision in the Real World: Attending, Foveating and Recognizing Objects Saliency peaks Peaks from blob detection of depth slices. Based on Differences of Gaussians. Hue saliency map used for weighting. Random value added before selection.

37. Vision in the Real World: Attending, Foveating and Recognizing Objects Fixation 0 0 c 0 0 d a b e F = The foveal system continuously tries to fixate done using corner features and affine essential matrix Zero disparity filters won’t work

38. Vision in the Real World: Attending, Foveating and Recognizing Objects Foveated segmentation To boost recognition Foveal segmentation based on disparities Rectification using affine fundamental matrix Only search for disparities around zero => Large number of false positives Points clustered in 3D using mean shift

39. Vision in the Real World: Attending, Foveating and Recognizing Objects Foveated segmentation

40. Vision in the Real World: Attending, Foveating and Recognizing Objects Foveated segmentation

41. Vision in the Real World: Attending, Foveating and Recognizing Objects Small object database in real-time experiments Models of SIFT features and hue histograms

42. Vision in the Real World: Attending, Foveating and Recognizing Objects Visual scene search

43. Vision in the Real World: Attending, Foveating and Recognizing Objects Segmentation robustness

44. Vision in the Real World: Attending, Foveating and Recognizing Objects Segmentation robustness

45. Vision in the Real World: Attending, Foveating and Recognizing Objects Effect of occlusions

46. Vision in the Real World: Attending, Foveating and Recognizing Objects Effect of rotations

47. Vision in the Real World: Attending, Foveating and Recognizing Objects Recognition in w-f-o-v

48. Vision in the Real World: Attending, Foveating and Recognizing Objects Recognition after foveation

49. Vision in the Real World: Attending, Foveating and Recognizing Objects Conclusions We have a running system. Objects normally found within three saccades Concern: dependency on corner features Current work: Focus on recognition and categorization More robust foveal segmentation Additional cues e.g. texture Learning and adaptation on all levels