Jonathan Dinger 1

Traffic footage example 2

 Important step in video analysis  Background subtraction is often used 3

 Uses pixelwise computations  Performance could be better  Better segmentation = better traffic detection 4

 Use interaction between neighboring pixels  Keep objects segmented together  Better segmentations 5

 G = (V,E)  Two vertices in V called source and sink ◦ s and t, respectively  Remaining vertices called M  Vertices in M connected to both s and t (T-links)  Vertices in M connected to neighboring vertices (N-links) 6

 T-links are uni-directional  N-links are bi-directional  Each edge has a weight ◦ also known as a capacity  Each pixel has its own vertex 7

 Path ◦ List of vertices connected by edges  s-t cut ◦ Removal of edges such that all vertices have a path to either the source or the sink, but not both  Flow ◦ Each edge has a capacity ◦ That much flow can be pushed through each edge ◦ Flow through a graph is the cumulative amount of flow going from the source to the sink 8

 Minimum cut/maximum flow ◦ A cut with the smallest cost (weight) ◦ Maximum flow that can be pushed from source to sink (capacity) ◦ By max-flow min-cut theorem, these are equal  Graph cuts ◦ Minimum cut on a graph 9

 Method to find maximum flow  Augmenting path ◦ Path where flow can be increased in all edges between vertices in path  Run search 1.Find augmenting path from source to sink 2.Add more flow to that path 3.Loop back to 1. 10

Step 1.Step 2. Step 3.Step 4. 11

Step 5.Step 6. Step 7.Step 8. 12

 Two separate implementations ◦ Our implementation ◦ Kolmogorov’s implementation v3.01.src.tar.gz 13

 Downsampled images  Cut performed over smaller area  Then upsample, and perform cut over band  Faster than graph cuts  Less detailed than graph cuts 14

15

 Downsampling images loses information  Use Laplacian pyramid to store lost data  Computes difference between image and image gained by downsampling and upsampling again  Add back some of the lost detail ◦ Resegment in areas where detail was lost 16

17

 BGC faster, less accurate  GC slightly slower, more accurate Graph cutBanded graph cut 18

 3 or more labels  α-β swap ◦ Loop through label pairs ◦ Run graph cut on current pair of labels  If current label of a pixel is not one of the pair, do not use pixel in graph cut ◦ Graph cuts will swap some pixels with label α to label β and vice versa 19

 Graph cut on each image  Background image computed per pixel ◦ N is the number of images, x i is the grayscale value of the current pixel, and μ is the average grayscale value over all image frames 20

 Find variance of each pixel  N, x i, and μ are as above  σ 2 is the variance  Threshold the variance  If variance is below the threshold, do not include pixel in graph cut ◦ Assume non-varying pixels are background pixels ◦ Avoid divide-by-zero errors in weights 21

 Exponentials  x is the grayscale value of the current pixel ◦ μ and σ 2 are as above  β is a constant that forces the two functions to be equal at α standard deviations 22

 Absolute differences  x and μ are as above  L is the maximum possible distance between x and μ, so for grayscale images  K is a shift constant that forces f 3 and f 4 to be equal at α standard deviations 23

 Simple Grayscale Differences  x is the grayscale value of the pixel. C m and C n are two grayscale values used as a basis for segmentation 24

 Distance ◦ Euclidean distance between two neighboring pixels with coordinates (x 1, y 1 ) and (x 2, y 2 )  Smoothing (similarity) ◦ x and y are the grayscale values of two neighboring pixels. ◦ is the maximum possible difference between the pixel values ◦ γ is a modifier that defines the amount of smoothing that takes place 25

ImageGraph cutBanded graph cut Augmented BGCOur graph cutMulti-way cut 26

Computing times for cut results in milliseconds (ms) 27

ImageBackgroundVariance Exponential cutAbsolute difference cut 28

ExponentialImageAbsolute difference 29

ExponentialImageAbsolute difference 30

ExponentialImageBackground subtraction 31

ExponentialImageAbsolute difference 32

ExponentialImageAbsolute difference 33

Absolute differenceImageBackground subtraction 34

35 ImageGraph cutBackground Subtraction

 Segmentation performance without smoothing comparable to background subtraction ◦ Background subtraction is faster, easier  Smoothing model ◦ Segments larger pieces of vehicles into one section ◦ Vehicle segmentations more “solid”  Absolute difference T-link weights combined with smoothing N-link weights give best results 36

 Use multi-way cuts to add shadow segmentation  Extend to RGB 37