1. Speeding Up MRF Optimization using Graph Cuts for Computer Vision
Vibhav Vineet
Adviser: Prof. P. J. Narayanan

2. Labelling Problem Extracting Foreground Pixels
Disparity map calculation
Image Denoising
Extracting Foreground Object
Left Tsukuba Image
Noisy House Image
Denoised Image
Disparity Map
Flower Image
Pixel Labeling: Assigning a label to each pixel in image.
Image Segmentation: Involves Separating Foreground layer from background layer
Pixel Labeling: Assigning a label to each pixel in image.
Stereo Correspondence: Involves Calculation of depth map using left and right images
Pixel Labeling: Assigning a label to each pixel in image.
Image Denoising: Involves assigning denoised intensity value to each pixel in image.

3. Labelling Problem - MAP Estimation
To find the best possible configuration.
But the complexity increases
With the number of variables/pixels
With the number of labels in the label set
Using joint probability or conditional probabilities to evaluate the best possible configuration
Very hard with the limited computation and memory power
Energy minimization method
MAP – MRF equiivalence
Methods provide approximate solution at a moderate times
Generally, in computer vision an energy function involve unary cost and pairwise interactions between variables.

4. Labelling Problem Image-Graph Equivalence
Total Cost = Unary Cost + PairWise Cost
Different Labeling, different Cost
Energy E ( X ) = Unary Potential + PairWise Potential
Labeling Problem: Find a Labeling ( X ) with Minimum Cost or Energy Value
Image-Graph Equivalence
Graph G( V, E )
PairWise Cost ( Per Edge Cost )
Cost of Assignment for “same label” is low.
Cost of Assignment for “different labels” is high.
Image-Graph Equivalence
Unary Cost ( Per Vertex Cost)
Graph G( V, E )
Cost of Assignment for “fg” is low.
Cost of Assignment for “bg” is high.

5. MAP-MRF Formulation MAP(X) Min Energy(X*)
MAP estimation of a configuration X is equivalent
to the minimum energy defined over the configuration
Energy = Data Term + Smoothness Term
Graph Cuts in Computer Vision
Image
Energy
Function
Graph
Construction
st-MinCut

6. Image-Graph Equivalence
Graph Construction
Vertex per pixel s t
Image-Graph Equivalence
Background Pixels
Foreground Pixels
Add n-edges
Image
Add t-edges
Graph G(V,E)
Graph Constructed for vision problems
Grid graphs
Low connectivity
Connectivity is limited to 4, 8, or 27

7. The st-Mincut Problem Given a Graph G(V,E,W) and two vertices s and t.
Partition G into two disjoint components containing s and t respectively such that sum of edge weights from s to t is minimum s
Mincut t

8. Computing the st-Mincut
Solve the dual Maximum Flow problem
Two approaches
Edmond Karp’s Augmenting path method
Goldberg’s Push-Relabel method
st-Mincut
Max Flow
Dual
In every network, the maximum flow equals
the cost of st-mincut

9. Edmond Karp Method Initialize flow in G to 0
Find a shortest path from s to t.
Augment the path with minimum possible flow
Repeat until there exists a path from s to t
Initialize flow in G to 0
Find a shortest path from s to t.
Augment the path with minimum possible flow
Repeat until there exists a path from s to t
Initialize flow in G to 0
Find a shortest path from s to t.
Augment the path with minimum possible flow
Repeat until there exists a path from s to t
Initialize flow in G to 0
Find a shortest path from s to t.
Augment the path with minimum possible flow
Repeat until there exists a path from s to t
Initialize flow in G to 0
Find a shortest path from s to t.
Augment the path with minimum possible flow
Repeat until there exists a path from s to t
Initialize flow in G to 0
Find a shortest path from s to t.
Augment the path with minimum possible flow
Repeat until there exists a path from s to t
Initialize flow in G to 0
Find a shortest path from s to t.
Augment the path with minimum possible flow
Repeat until there exists a path from s to t s t
100 40 7 3 4 6 11 13 14 s t
100 40 7 3 4 6 11 13 14 s t
100 37 7 1 6 11 13 14 3 s t
100 37 7 1 6 11 13 14 3 s t 93 37 1 6 11 13 7 3 s t 93 37 1 6 11 13 7 3 s t 87 37 1 17 7 3 6
Current Flow: 10
Current Flow: 16
Current Flow: 10
Current Flow: 0
Current Flow: 0
Current Flow: 3
Current Flow: 3
Flow <= Edge Capacity
Edge Capcity must be positive

10. Edmond Karp Method Initialize flow in G to 0
Find a shortest path from s to t.
Augment the path with minimum possible flow
Repeat until there exists a path from s to t s 87 37 6 7 3 7 17 1 1
Current Flow: 16 t
Flow <= Edge Capacity
Edge Capcity must be positive

11. Goldberg’s Push-Relabel Algorithm
Initialize excess flow and heights in G
Perform an applicable Push or Relabel operation
Repeat until there exists an applicable push or relabel operation
Initialize excess flow and heights in G
Perform an applicable Push or Relabel operation
Repeat until there exists an applicable push or relabel operation
Initialize excess flow and heights in G
Perform an applicable Push or Relabel operation
Repeat until there exists an applicable push or relabel operation
Initialize excess flow and heights in G
Perform an applicable Push or Relabel operation
Repeat until there exists an applicable push or relabel operation
Initialize excess flow and heights in G
Perform an applicable Push or Relabel operation
Repeat until there exists an applicable push or relabel operation
Initialize excess flow and heights in G
Perform an applicable Push or Relabel operation
Repeat until there exists an applicable push or relabel operation
Initialize excess flow and heights in G
Perform an applicable Push or Relabel operation
Repeat until there exists an applicable push or relabel operation
Initialize excess flow and heights in G
Perform an applicable Push or Relabel operation
Repeat until there exists an applicable push or relabel operation
Initialize excess flow and heights in G
Perform an applicable Push or Relabel operation
Repeat until there exists an applicable push or relabel operation
Initialize excess flow and heights in G
Perform an applicable Push or Relabel operation
Repeat until there exists an applicable push or relabel operation
Initialize excess flow and heights in G
Perform an applicable Push or Relabel operation
Repeat until there exists an applicable push or relabel operation
Initialize excess flow and heights in G
Perform an applicable Push or Relabel operation
Repeat until there exists an applicable push or relabel operation
Initialize excess flow and heights in G
Perform an applicable Push or Relabel operation
Repeat until there exists an applicable push or relabel operation
Initialize excess flow and heights in G
Perform an applicable Push or Relabel operation
Repeat until there exists an applicable push or relabel operation
Initialize excess flow and heights in G
Perform an applicable Push or Relabel operation
Repeat until there exists an applicable push or relabel operation s 87 37 7 t 3 1 6 11 13 8
Height s 87 37 7 t 3 1 6 17 8
Height s 87 37 t 7 3 1 6 17
Height s 87 37 7 t 3 1 6 17 8
Height s 87 37 7 t 3 1 6 17 8
Height s 87 37 7 t 3 1 6 11 13 8
Height s 87 37 7 t 3 1 6 11 13 8
Height s 87 37 7 t 3 1 6 11 13 8
Height s 87 37 7 3 6 t 4 11 13 14
Height s
100 40 t 7 3 4 6 11 13 14
Height s
100 40 t 7 3 4 6 11 13 14
Height s
100 40 t 7 3 4 6 11 13 14
Height s 87 37 7 3 6 t 4 11 13 14
Height s 87 37 7 3 6 t 4 11 13 14
Height s 87 37 7 3 6 t 4 11 13 14
Height
Current Flow: 9
Current Flow: 0
Current Flow: 0
Current Flow: 16
Current Flow: 9
Current Flow: 0
Current Flow: 0
Current Flow: 0
Current Flow: 9
Current Flow: 9
Current Flow: 9
Current Flow: 0
Current Flow: 9
Current Flow: 0
Current Flow: 9
Flow <= Edge Capacity
Edge Capcity must be positive

12. Goldberg’s Push-Relabel Algorithm
Initialize excess flow and heights in G
Perform an applicable Push or Relabel operation
Repeat until there exists an applicable push or relabel operation
Height s 6 37 87 7 17 3 t 1 1 6
Current Flow: 16
Flow <= Edge Capacity
Edge Capcity must be positive

13. Motivation Fast Computation Required
Robot navigation, surveillance, video processing etc
Video Processing at real time
You tube and other web-servers
Large images Processing
Even our offshelf cameras take high resolution images
Interactive tools

14. Mapping to CUDA
Image
Grid
Thread per pixel
Image
CUDA Grid
CUDA Block

15. Push-Relabel Algorithm on CUDA
Push is an local operation with each node sending flows to its neighbors.
Relabel is also a local operation, each vertex updates its own height.
Problems faced:
Read After Write consistency
Synchronization of threads

16. Handling Problems using Atomics
Push operations can performed without any read after write inconsistencies
Relabel is a per vertex operation
Employing atomic Capabilities and combining the push and pull kernels
Push Kernel
Relabel Kernel
Lowers Global memory access, empirically faster convergence is observed.

17. The Push Kernel
Load heights from the global memory to the shared memory.
Synchronize threads ensuring the completion of load operation.
Push flows to eligible neighbors atomically.
Update the edge-weights atomically in the residual graph.
Update excess flow atomically in the residual graph.

18. The Relabel Kernel
Load height from the global memory to the shared memory.
Synchronize ensuring the completion of load operation.
Compute the minimum height of all neighbors and set own height to plus one of this.
Write the new height to global memory.

19. Using Shared Memory
Per CUDA block requires (Block_size+2 X Block_size+2) memory to be loaded into the shared memory
Corner pixels need heights from other blocks
Block size + 2
Thread
Block size + 2
Height needed per thread
Block

20. Heuristics on Push and Relabel
On grid graphs Global relabel (BFS based) is an expensive operation
Local relabel perform better empirically
Multiple pushes can be performed before applying a relabel step using
For most general graphs m=3 and k=7 are found to be optimal.
(m*Push + Relabel)*k + Global Relabel

21. Stochastic Cuts MRF consists of simple and difficult pixels.
Simple pixels get their correct labels in few initial iterations
Difficult (few) pixels exchange flows with their neighbors in later iterations
Stochastic Cuts processes pixels based on their activity
Activity is based on change in flows from previous to current iteration. Low activity is observed for simple pixels
Heuristically process simple pixels after a fixed number of iterations

22. Experimental Results

23. Experimental Results Image Size Time CPU (ms) Time Non Atomic
Time Atomic
Stochastic
Sponge
640x480
142 28 16 11
Flower
608x456
188 33 26
Person
140 31 27 20
Synthetic
1Kx1K
655 19 10 7

24. Graph ReparameterizationS t 5 9 4 2 1 S 2 9
Graph Cuts
Graph Cuts 1 2 5 4 t

25. Graph ReparameterizationS t 5 9 4 2 1 S 2 9
Graph Cuts
Graph Cuts 1 2 5 4 t
Graph Reparameterized
Graph Reparameterized
No change in cut S S
2+2 9 4 9 1
Graph Cuts
Graph Cuts 1 2 2
5+2 4 7 4 t t

26. Dynamic Cuts EA SA EB SB Problem Instance 1 Problem Instance 2
Problems instances where they differ slightly.
Solving each independently is computationally expensive
Example: Continuous frames in a video

27. Dynamic Cuts Steps Involved
Edge capacities are updated and reparameterized using
Previous frame edge capacities
Previous frame residual flow
Current frame edge capacities ci
Previous Frame fi
Previous Frame after st-MinCut ri
ci’
Current Frame
Updation Step:
ri’ = ri + ci’ - ci
ri’
Approximate cut using previous frame and its st-MinCut
fi’
Reparameterization Step:
rsi’ = 0
rit’ = cit - fit + fsi – csi’
Final st-MinCut of current frame

28. Dynamic Cuts are parallizable
Updation and Reparameterization are independent and parallizable operations, work locally at every vertex.
st-Mincut is performed using a parallel implementation of Push Relabel algorithm.

29. Dynamic Cuts Empirically
Running time depends on the percentage of weights that changed
On a low resolution video, the dynamic cuts takes about 2 ms compared to 7 ms on the same image for the st-MinCut
Consecutive frames of a video segmented using dynamic cuts

30. The Multilabeling problem
Multi-way cut on any graph is an NP-Hard problem for L > 2
Approximate solutions based on graph cuts
α-Expansion
αβ-Swap

31. The α-Expansion 1: Initialize the MRF with an arbitrary labeling X
2: For each label alpha \in L do
3: Construct the graph based on the current configuration
4: Perform one α-Expansion step (st-cut)
5: Update the configuration if energy decreases
6: End For
7: Repeat steps 2 to 6 till convergence.
Step 2-6 is a cycle and 3-5 is an iteration

32. Incremental α-Expansion
Reusability of flows, as in dynamic MRF
Better initializations for next graph cut
Incremental/Dynamic
Reuse the flows from label to label and and re- cycle flows from cycle to cycle.
Cycle 1
Input
Label1
Label2
Label3
Cycle 2

33. Incremental α-Expansion Results
Tsukuba
Teddy
Penguin
Panorama

34. Incremental α-Expansion Results
Total Timings on Different Datasets

35. Processing on High Detailed Scene
High Resolution Image
High Dynamic Ranges of Colors
Wide View Angles.
Challenges
High Computation Cost
High Memory Requirement
Interaction with high resolution images
Statistics of image sizes available on Google images.
An overwhelming fraction of images are of size 2 to 10 million pixels.
Only 0.6% of fewer images had more than 40 mega pixels.

36. Processing on High Detailed Scene
Define E(x) for coarsest image
Final Result for this level
Define E(x) for next finer level
Final Result at this level
Solve an optimization problem at the coarser level to dynamically update the optimization instance for the next level for better initialization.

37. Pyramid Reparameterization
Construction
Pyramid is Constructed.
Input largest image at the base of the pyramid.
Each pixel coarser level image is mean of 4 pixels at the previous finer level

38. Pyramid Reparameterization
Image (i-1)
Graph G(V,E)
Minimization at (i-1)
Segmented Image at (i-1)
Residual Graph G(V,E)
Upsampling Step
Upsampled Initial Graph
Upsampled Residual Graph

39. Pyramid Reparameterization
Computationally Expensive Graph Cuts
Graph at the current level
Final Residual Graph at the current level

40. Pyramid Reparameterization
Graph Cuts
Upsampled graph of previous level
Reuse of flows
Difference between two graphs
Cheaper solution
Computationally Expensive Graph Cuts
Graph at the current level
Final Residual Graph at the current level

41. Residual Graph Upsampling
Upsampling Rules
Graph Upsampling
Graph Cuts
Residual Graph Upsampling

42. Residual Graph Upsampling
Upsampling Rules
Graph Upsampling
Graph Cuts
Residual Graph Upsampling

43. Residual Graph Upsampling
Upsampling Rules
Graph Upsampling
Graph Cuts
Residual Graph Upsampling

44. Image Segmentation Results
Horse
3.3 MP (2048x1600)

45. Image Segmentation Results

46. Interactive Image Segmentation Tool
User Interaction Important in foreground/background separation
User 1
User 2
Results of a user study on image size for comfortable manipulation for two display sizes.
Average subjective response for six image sizes.
Images that are larger than the display is disfavored users.

47. Pyramid Segmentation System
Display Window
User interacts at the display window of comfortable size
Actual Image
Actual Segmentation goes on in background on other levels
Quick Segment:
Display the segmentation results on this display image
Provides perceptual response to start planning further interactions

48. User Study
Results of User Study on CPU and GPU version of Pyramid Segmentation With GrabCut and Quick Selection
Interaction Time
Response Time
Total Time
Subjective Response of Users

49. Multiresolution alpha-expansion
- Build pyramid of graphs.
Perform alpha-expansion at a lower resolution graph.
Save the initial and final residual graphs for all the labels.
Upsample and reparameterize the previous resolution initial and final graphs and current resolution initial graph.
Perform alpha-expansion at this level.
Repeat this for all the levels in the pyramid.

50. Stereo Correspondence
Image size – 1328 x 1104
Number of Labels (Disparity) –

51. Stereo Correspondence
Optimization Time
Total Time (Optimization time + graph construction time + energy function calculation time)
Running time in seconds for stereo correpondence using Pyramid Cuts on the GPU (G-PyCut), the CPU (C-PyCut) and a single level Graph Cuts(GCuts).
A speed up of 5-6 times on the CPU is observed.

52. Number of Labels (Disparity) – 256
Image Denoising
Image size – 1000 x 1000
Number of Labels (Disparity) – 256

53. Image Denoising
Total Time (Optimization time + graph construction time + energy function calculation time)
Optimization Time
Running time in seconds for stereo correpondence using Pyramid Cuts on the GPU (G-PyCut), the CPU (C-PyCut) and a single level Graph Cuts(GCuts).
A speed up of 5-6 times on the CPU is observed.

54. Future Work Higher order Interactions of variables in MRF
Computationally more challenging
Modelling this on our hierarchical and multiresolution framework
Using multiple GPUs to parallelize the alpha-expansion
Better interactive tools:
Both global and local interactions

55. Conclusion Two methods to optimize basic graph cuts algorithm
Using facilities provided by parallel accelerators like GPU
Modelling graph cuts on hierarchical and dynamic framework for better initialization
Graph Cuts methods proved very instrumental solving many computationally challenging problems
Successes of graph cuts -> Promising future in the realm of energy minimization methods

56. Related Publications
P. J. Narayanan, Vibhav Vineet and Timo Stitch. Fast Graph Cuts on the GPU. GPU Computing Gems (GCG), Volume 1 Dec (Book Chapter).
Vibhav Vineet and P. J. Narayanan. Solving Multi-label MRFs using incremental alpha-expansion move on the GPUs. In Proceeding of Ninth Asian Conference on Computer Vision. (ACCV-2009), China, 2009.
Vibhav Vineet and P. J. Narayanan. CUDA Cuts: Fast Graph Cuts on the GPU. In Proceeding of CVPR workshop on Visual Computer Vision on GPUs (CVGPU-2008), Alaska, USA, 2008.
Vibhav Vineet, Pawan Harish, Suryakant Patidar and P. J. Narayanan. Fast Minimum Spanning Tree for Large Graphs on the GPU. In Proceeding of ACM SIGGRAPH High Performance Graphics (HPG-2009), New Orleans, LA, USA, 2009.
Pawan Harish, Vibhav Vineet and P. J. Narayanan. Large Graph Algorithms for Massively Multithreaded Architectures. IIIT Tech Report, IIIT/TR/2009/74.
CUDA Cuts: Fast Graph Cuts on the GPU. (Software).
Vibhav Vineet, Pawan Harish, Suryakant Patidar and P. J. Narayanan. Fast Minimum Spanning Tree for Large Graphs on the GPU. GPU Computing Gems (GCG). (Book Chapter).

57. Changes made to the thesis
Reviewer1 (Dr. Kishore)
Tables containing experimental results on more standard images (around 60) images added.
Sections on related and background works expanded.
Reviewer 2 (Dr. Srinivasan)
Missed references added to the related work section.
Formal results on speed up for dynamic graph cuts on video segmentation added to the result section.
Figure captions properly referenced with the paper of Kohli and Torr.
Other minor changes made as recommended.

58. Thank You