1. An Oscillatory Correlation Approach to Scene Segmentation DeLiang Wang The Ohio State University

2. IPAM'022 Outline of Presentation l Introduction l Scene segmentation problem l Oscillatory Correlation Theory l LEGION network l Oscillatory Correlation Approach to Image Segmentation l Circuit Implementation of LEGION l Summary

3. IPAM'023 Scene Analysis Problem

4. IPAM'024 Simplified Scenario

5. IPAM'025 Temporal Correlation Theory Feature binding is a fundamental problemn l In neuroscience l In perception Temporal correlation as a representation l Extra degree of freedom l A plausible mechanism See von der Malsburg’81, Milner’74, Abeles’82

6. IPAM'026 Neurophysiological Evidence l Gray & Singer (1989)

7. IPAM'027 Neurophysiological Evidence - continued l Gray et al. (1989)

8. IPAM'028 Oscillatory Correlation Theory

9. IPAM'029 Computational Requirements for Oscillatory Correlation Need to synchronize locally coupled oscillator population l Extensive literature in theoretical physics and applied mathematics Need to desynchronize different populations, when facing multiple objects l Critically, the above functions must be achieved very rapidly

10. IPAM'0210 Three Possible Ways to Reach Synchrony l Comparator model l Globally connected l Locally connected

11. IPAM'0211 LEGION Architecture l LEGION - Locally Excitatory Globally Inhibitory Oscillator Network (Terman & Wang’95)

12. IPAM'0212 Single Relaxation Oscillator With stimulus Without stimulus Typical x trace (membrane potential)

13. IPAM'0213 Model of a Relaxation Oscillator l Model definition l Coupling between oscillators l Where N(i) is the set of neighboring oscillators that connect to oscillator i

14. IPAM'0214 l Global inhibitor Model of a Relaxation Oscillator - continued

15. IPAM'0215 Somers & Kopell’93 Fast Threshold Modulation for Two Oscillators LB RB E LB E RB

16. IPAM'0216 Selective Gating for Two Oscillators RB LB LB I

17. IPAM'0217 Summary of Analytical Results Definitions: A pattern is a connected region, and a block a subset of oscillators stimulated by a given pattern. The following results are established for  > 0 sufficiently small l Theorem 1. (Synchronization). The parameters of the system can be chosen so that all of the oscillators in a block synchronize. Moreover, the rate of synchronization is exponential

18. IPAM'0218 Summary of Analytical Results - continued Theorem 2. (Multiple patterns) If at the beginning all the oscillators of the same block synchronize with each other and the distance between any two oscillators belonging to two different blocks is greater than some constant, then: l Synchronization within each block is maintained l The ordering of the activation among different blocks is fixed l Given a certain period of time, at least one block is in its active phase l At most one block is in its active phase at any time

19. IPAM'0219 Summary of Analytical Results - continued Theorem 3. (Desynchronization) If at the beginning all the oscillators of the system lie not too far away from each other, then the condition of Theorem 2 will be satisfied after some time. Moreover, the time it takes to satisfy the condition is no greater than N cycles, where N is the number of patterns. l The entire mechanism is called Selective Gating (Terman & Wang’95)

20. IPAM'0220 LEGION Example Input image Successive snapshots

21. IPAM'0221 LEGION Example - Temporal Traces

22. IPAM'0222 LEGION Example: Segmentation Capacity Input image

23. IPAM'0223 Oscillatory Correlation Approach to Image Segmentation l Feature extraction first takes place l An image feature can be pixel intensity, depth, local image patch, texture element, optic flow, etc. l Connection weights between two neighboring oscillators are set to be proportional to feature similarity l Global inhibitor controls granularity of segmentation l Larger inhibition results in more and smaller regions l Segments pop out from LEGION in time

24. IPAM'0224 Image Segmentation Example Input imageSegmentation result

25. IPAM'0225 Image Segmentation Example Input imageSegmentation result

26. IPAM'0226 3D MRI Image Segmentation Left: LEGION results Right: Manual results

27. IPAM'0227 3D MRI Image Segmentation: a 2D View Left: Input Middle: Segmentation results Right:Further segmentation into white and gray matter

28. IPAM'0228 Aerial Image Analysis Extraction of hydrographic objects

29. IPAM'0229 Texture Segmentation Upper: input; Lower: segmentation result

30. IPAM'0230 Motion Segmentation

31. IPAM'0231 Circuit Implementation of LEGION l Circuit implementation of LEGION using PWM/PPM l PWM: Pulse-width modulation PPM: Pulse-phase modulation l By Ando et al. (2000) Pulse modulation

32. IPAM'0232 Circuit Implementation of a Single Oscillator Circuit of a single relaxation oscillator SPICE simulation result

33. IPAM'0233 Circuit Implementation of a LEGION Network Connection circuit Global inhibitor circuit

34. IPAM'0234 Circuit Implementation of LEGION - continued Input image l Application to gray-level image segmentation Output at two different times

35. IPAM'0235 LEGION on a Chip (Cosp 2000) The chip area is 6.7 mm 2 (Core 3 mm 2 ) and implements a 16x16 LEGION network

36. IPAM'0236 Back to Biology Gray & McCormick’96

37. IPAM'0237 Summary l Mathematical analysis of the selective gating mechanism and LEGION dynamics l Both synchronous oscillations and the structure of the model are neurally plausible l Oscillatory correlation approach to scene segmentation l VLSI implementation A neural theory for perceptual organization

38. IPAM'0238 Collaborators l David Terman l Erdogan Cesmeli l Ke Chen l Xiuwen Liu l Naeem Shareef