1. Interactive Discovery and Semantic Labeling of Patterns in Spatial Data Thomas Funkhouser, Adam Finkelstein, David Blei, and Christiane Fellbaum Princeton University FODAVA Review Meeting December 2009

2. Motivation Lots of 3D data is available with spatial patterns that reveal semantic information Archeology Climate Simulation Molecular Biology

3. Example: Lidar Scans of Cities Ottawa

4. Goal Discover spatial patterns in 3D data to assist semantic segmentation and labeling

5. Related Work Specific Objects [Chen and Chen 07] Buildings, trees, etc. Markov Random Fields [Anguelov et al. 05] Point labeling Segment, label [Secord and Zakhor 07] [Carlberg et al. 09] [Golovinskiy et al. 09]

6. Current Approach Supervised learning: Training data  Locate  Segment  Describe  Build classifier New data  Locate  Segment  Describe  Apply classifier Training Area

7. Current Approach Supervised learning: Problems  Training data only useful if matches new data  Trainer prescribes semantic classes  Trainer must label enough training data to cover all possible new data Training Area

8. Problems I Raw data may be difficult to segment automatically into semantic objects

9. Problems II Local spatial patterns may not be descriptive enough to assign semantic labels What is this?

10. Problems III Spatial patterns/features for objects of same type may be different in different data sets

11. Problems IV Semantic objects of interest may be different for different users What areas of city have too few street lights? What is spacing between fire hydrants? Where should trees be planted? Where could a terrorist could hide a bomb? Where do people park?

12. Another Possible Approach Active learning: Off-line  Locate  Segment  Describe On-line  System builds classifier by requesting user labels

13. Another Possible Approach Active learning: Problems  Computer drives training process – tries to learn user’s semantic model  Usually classes are pre-specified  Discrete sequence of visual recognition tasks – if jump from example to example

14. Our Approach User-driven learning: Off-line  Locate  Segment  Describe On-line  User interactively adjusts segments and labels on data  System builds clusters, classifiers, and provides visual feedback

15. Our Approach User-driven learning: Advantages:  System can guide user towards most useful input with visualization  User drives process – can focus on what he/she cares about  User can create/remove classes during labeling process  Continuous visual recognition, easier since camera is controlled by user

16. Main Challenge User-driven learning: Integrate off-line analysis with unsupervised learning with interactively updated probabilistic inference model while providing interactive visual feedback User Class labels

17. First Steps

18. Outline Introduction User-driven learning Specific research issues Segmentation Shape description Pattern discovery Visualization Wrap up

19. Outline Introduction User-driven learning Specific research issues  Segmentation Shape description Pattern discovery Visualization Wrap up

20. Segmentation Problem: Clustering points into semantic objects

21. Segmentation Current approach: Hierarchical clustering to find candidate clusters Min-cut separation of foreground from clutter SegmentationProximity graphInput data

22. Segmentation Current approach: Hierarchical clustering to find candidate clusters Min-cut separation of foreground from clutter

23. Segmentation Current results:

24. Segmentation Current results:

25. Segmentation Future challenges: 1)Provide interactive, adaptive segmentation tools 2)Integrate segmentation with recognition 3)Integrate segmentation with inference

26. Outline Introduction User-driven learning Specific research issues Segmentation  Shape description Pattern discovery Visualization Wrap up

27. Shape Description Problem: Describe cluster of points by a feature vector that discriminates its semantic class

28. Shape Description Current approach: Shape features (volume, eccentricity, …) Shape descriptors (spin images, shape contexts, …) Contextual cues (distances to other objects) Spin Image

29. Shape Description Current results: Training AreaTesting Area

30. Shape Description Current results:

31. Shape Description Proposed approach: Data-adaptive dictionaries -- adaptable filters designed to discriminate user selected object types Blue = positive, Red = negative Feature vector... -0.90.81.00.8 Adaptable filters Query Shape

32. Shape Description Future challenges: Computational representation for adaptable descriptors Efficient adaptation of descriptors as user labels examples Interactive user guidance in refinement of descriptors

33. Outline Introduction User-driven learning Specific research issues Segmentation Shape description  Pattern discovery Visualization Wrap up

34. Pattern Discovery Goal: Recognize spatial patterns and use them to segment and label clusters of points

35. Pattern Discovery Current approach: Learn probabilistic representation of symmetries and use it to predict labels Input dataMarked lampposts Symmetry transform (probabilistic model of translational symmetry)

36. Pattern Discovery Current results: Adding probabilistic symmetry as a feature helps recognition (by a little) Symmetry

37. Pattern Discovery Future challenges: 1)Better representations for symmetries and spatial relationships 2)Integrate symmetries and spatial relationships into probabilistic inference model 3)Interactive specification and visualization of symmetries and spatial patterns Symmetry transform (probabilistic model of translational symmetry)

38. Outline Introduction User-driven learning Specific research issues Segmentation Shape description Pattern discovery  Visualization Wrap up

39. Visualization Goals: Provide interactive displays to help user understand …  Input data  Specified segments and labels  Inferred segments and labels  Value of further input  Computational models

40. Visualization Proposed approach: Multiple views  3D space  Feature space  Symmetry space Symmetry Space 3D Space Feature Space

41. Visualization Future challenges: Provide methods to …  Integrate multiple views  Represent uncertainty  Guide user input  Reduce clutter

42. Other Applications Molecular Biology Paleontology Climate Archeology

43. Wrap Up Goal: Segment and label patterns in 3D data Approach: user-driven learning User interactively guides segmentation and labeling System learns model and provides visual feedback Research challenges: user-driven … Segmentation Shape description Pattern discovery Inference Visualization

44. Acknowledgments Students: Former: Alex Golovinskiy Current: Aleksey Boyko, Vladimir Kim Other funding sources: Former: AFRL, URGENT Current: NSF, Google