1. Fast, Multiscale Image Segmentation: From Pixels to Semantics Ronen Basri The Weizmann Institute of Science Joint work with Achi Brandt, Meirav Galun, Eitan Sharon

4. Camouflage

5. Camouflage Malik et al.’s “Normalized cuts”

11. Our Results

12. Segmentation by Weighted Aggregation A multiscale algorithm: Optimizes a global measure Returns a full hierarchy of segments Linear complexity Combines multiscale measurements: –Texture –Boundary integrity

13. The Pixel Graph Couplings (weights) reflect intensity similarity Low contrast – strong coupling High contrast – weak coupling

14. Normalized-cut Measure Minimize:

15. Saliency Measure Minimize:

16. Multiscale Computation of Ncuts Our objective is to rapidly find the segments (0-1 partitions) that optimize  For single-node cuts we simply evaluate  For multiple-node cuts we perform “soft contraction” using coarsening procedures from algebraic multigrid solvers of PDEs.

17. Coarsening the Graph Suppose we can define a sparse mapping such that for all minimal states

18. Coarse Energy Then P T WP, P T LP define a new (smaller) graph

19. Recursive Coarsening

20. Representative subset

21. Recursive Coarsening For a salient segment :, sparse interpolation matrix

22. Weighted Aggregation aggregate

23. Hierarchical Graph  Pyramid of graphs  Soft relations between levels  Segments emerge as salient nodes at some level of the pyramid

24. Importance of Soft Relations

25. Physical Motivation Our algorithm is motivated by algebraic multigrid solutions to heat or electric networks u - temperature/potential a(x, y) – conductivity At steady state largest temperature differences are along the cuts AMG coarsening is independent of f

26. Determine the Boundaries 1 0 0 1,0,0,…,0 P

27. Hierarchy in SWA

28. Texture Examples

29. Filters (From Malik and Perona) Oriented filters Center- surround

30. A Chicken and Egg Problem Problem: Coarse measurements mix neighboring statistics Solution: Support of measurements is determined as the segmentation process proceeds Hey, I was here first

31. Texture Aggregation Aggregates assumed to capture texture elements Compare neighboring aggregates according to the following statistics: –Multiscale brightness measures –Multiscale shape measures –Filter responses Use statistics to modify couplings

32. Recursive Computation of Measures Given some measure of aggregates at a certain level (e.g., orientation) At every coarser level we take a weighted sum of this measure from previous level The result can be used to compute the average, variance or histogram of the measure Complexity is linear

33. Use Averages to Modify the Graph

34. Adaptive vs. Rigid Measurements Averaging Our algorithm - SWA Original Geometric

35. Adaptive vs. Rigid Measurements Interpolation Geometric Original Our algorithm - SWA

36. Adaptive vs. Rigid Measurements

41. Texture Aggregation Fine (homogeneous) Coarse (heterogeneous)

42. Multiscale Variance Vector

44. Variance: Avoid Mixing aggregationSliding window

45. Leopard

46. More Leopards…

47. And More…

48. Birds

49. More Animals

50. Boat

51. Malik’s Ncuts

52. Key Differences Optimize a global measure (like Malik’s Ncuts) Hierarchy with soft relations (unlike agglomerative/graph contraction) Combine texture measurements while avoiding the “chicken and egg problem”

53. Complexity Every level contains about half the nodes of the previous level: Total #nodes 2 X #pixels All connections are local, cleaning small weights Top-down sharpening: constant number of levels Linear complexity Implementation: 5 seconds for 400x400

54. Average intensity Texture Shape Representation

55. Relevance to Biology? Layers of retinotopic maps Feed-forward (fine-to-coarse) progression Feedback refinement Edges determine both boundaries and textures Segmentation determined by combinations of cues Coarse nodes represent abstract shapes and properties

56. Matching (with Chen Brestel)

57. More…

58. Data: Filippi 30 slices, 180x220 in 3 minutes MRI Data

59. MS Lesion Detection Tagged Our results Data: Filippi

60. Tagged Our results

61. Data: Filippi Tagged Our results

62. Data: Filippi Tagged Our results

63. 2D Segmentation Data: Filippi

64. 3D Segmentation

65. Cell Movement

66. Summary image segments Shape properties Leopard