Stereovision Image Noise John Morris Department of Computer Science, Tamaki Campus The University of Auckland Iolanthe at 14 knots on Cockburn Sound, Perth

2 Correlation methods - summary  Simplest code  Poorest performance  Adaptive windows don’t help much!  Medium speed  Window size is an important factor  Simple hardware realization but  Expensive in resource use  Handle random noise only  Window is just treated as vector of pixels  No spatial information used  Occlusions ignored

3 Dynamic Programming Stereo  Attempts to find the ‘best path’ (sequence of disparity values)  Can recognize occlusions!  Averages noise over a scanline  Essentially local  Always moves ‘forward’ in a scanline  Solution generated by backtracking through predecessor array Doesn’t adjust values in backtrack  Uses the ordering constraint  Readily adapted to  allow for gain and offset changes  Perspective distortion  ‘Stubborn’  Incorporates a penalty for occlusions  Tends to make ‘streaks’ in disparity images..\resources\DP_Gonzalez.pdf..\resources\DP_Gonzalez.pdf  Can be improved by using neighbouring scan lines  Requires fewer scan line buffers than correlation window

4 Dynamic Programming Result Note the horizontal streaks! It’s like a bureaucrat: once a DP algorithm ‘decides’ to adopt a disparity value – it doesn’t want to change its mind! Adding inter-scanline constraints (using a neighbouring scanline) generally improves this!

5 Dynamic Programming Performance  Better matching than correlation methods  ‘Global’ along scan lines  Recognizes occlusions  Uses ordering constraint  Uses spatial information  Time complexity O (n 2 d)  Faster than correlation methods in software  Uses memory (typical of DP algorithms)  For n=10 3, d=10 2, t = c 10 8  c = ~ 30 cycles / pixel on 3GHz machine  Not enough for real time in software!

Dynamic Programming  Basic idea  Given a very large problem  Solve small problems first,  Save solutions to the small problems,  Use them to solve larger problem  Continue until large problem is solved  Dynamic programming uses storage space (for solutions to the small problems) to gain speed 6

Dynamic programming stereo  Small problems are the costs of a ‘path’ to a certain pixel  Costs are based on  Difference between pixel intensities  Cost for an occlusion  Sometimes - Discontinuity cost  Assume that disparity changes (depth changes) are relatively rare  Alternatively – assume most surfaces are smooth 7

Dynamic programming stereo 8 8 Monocular Right Pixels Left scan line Monocular Left pixels Right scan line Start End Optimum path marks matching pixels Diagonal path – smooth surface – no change in disparity Jumps – changes in disparity This path would indicate all matches at zero disparity

Dynamic programming stereo - Costs 9 9 Monocular Left Pixels Left scan line Monocular Right pixels Right scan line Start End DP algorithm calculates cumulative costs for all possible paths – using costs for previous pixel to estimate costs for the current pixel Maximum disparity Compute costs for paths in this region Cost stored in this location represents cost of best path to this point The minimum cost here is the end of the best path COSTS ARRAY

Dynamic programming stereo - Predecessors 10 Monocular Left Pixels Left scan line Monocular Right pixels Right scan line Start End Keep a second array of predecessors Maximum disparity Store here best predecessor for a path ending here The minimum cost here is the end of the best path PREDECESSORS ARRAY

Dynamic Programming 11 Two arrays used: Costs and Predecessors As best cost C dx to reach (d,x) is computed, the best predecessor is stored in  d,x

Dynamic Programming - general 12 When the end of the scanline is reached, the minimum cost is selected and its predecessor used to build the optimum profile (red arrows)

Dynamic programming stereo 13 Monocular Left Pixels Left scan line Monocular Right pixels Right scan line Start End As cost for each pixel at each disparity is determined, save the predecessor Use the predecessors to back track Need w  predecessors – Biggest resource cost! Predecessor array stores disparity of best previous point Maximum disparity Compute costs for matches in this region

14

Symmetric Dynamic Programming Stereo 15 Compute costs for the Cyclopean image centred between O L and O R Use visibility constraints - If disparity changes by n pixels, must be n MR or ML points between the two B points Only need to save state in predecessor array.. not disparity!

Symmetric Dynamic Programming Stereo  Conventional Stereo  Match pixel in L image with pixels to R of it in R image  SDPS  For pixel position, x, in Cyclopaean image, match pixels at x L +d/2 with x R -d/2  Even x positions have even disparity, odd x position have odd disparity  Calculate in two phases: odd and even  Resultant image is 2w pixels wide  Backtrack fills in extra pixels in B regions 16

SDPS  Cost equations  Binocular points  C(x,d,B) = dI(x,d) + min(C(x-1,d-1,ML), C(x-2,d,B), C(x-2,d,MR) ) (1)  Costs to reach monocularly visible states are:  C(x,d,ML) = occ_term + min(C(x-1,d-1,ML), C(x-2,d,B), C(x-2,d,MR) ) (2)  C(x,d,MR) = occ term + min(C(x-1,d+1,MR),C(x-1,d+1,B) ) (3)  Pixel mismatch is absolute difference  dI(x,d) = |g L -g R |  Any other dissimilarity measure could be used 17

SDPS  Occlusion cost is fixed constant  Results are not very sensitive to the actual value  Choose about three times pixel intensity variance. ie signal noise  Predecessor array only stores previous state  ML, B or MR – 2 bits only 18

SDPS transitions 19 This pixel, at disparity d 0, can only have 3 states – ML, B, MR Arrows indicate costs used to evaluate cost to reach each state. Visibility constraints allow only these transitions. MR ML Note half integral x positions – SDPS generates 2w points/line

20 System Architecture Interface LVDS/ CameraLink Corrected Images Depth Map Line Buffers Distortion Removal Image Alignment Host Higher order Interpretation L Camera R Camera Control Signals FPGA PC Stereo Matching Disparity  Depth

SDPS hardware – Gidel FPGA card 21

SDPS hardware – Disparity calculator block 22 Small and compact – easily replicated to provide large disparity range

SDPS hardware – proposed system 23

FPGA hardware  Field Programmable Gate Arrays  Soft hardware  ‘Programmed’ in a high level language – VHDL  ‘Compiled’ programs soft loaded onto FPGA  No cost to change the circuit!  Tradeoffs Flexible  Pipeline or inherent parallelism easily achieved Compare GPU  Slower than state of the art CPUs – n00 MHz vs n GHz  Memory may be a problem 24

25 FPGA Hardware  FPGA = Field Programmable Gate Array  ‘Circuit’ is stored in static RAM cells  Changed as easily as reloading a new program

26 FPGA Hardware  Why is programmability important? or  Why not design a custom ASIC?  Optical systems don’t have the flexibility of a human eye  Lenses fabricated from rigid materials  Not possible to make a ‘one system fits all’ system  Optical configurations must be designed for each application  Field of view  Resolution required  Physical constraints  …  Processing hardware has to be adapted to the optical configuration  If we design an ASIC, it will only work for one application!!

Other correspondence algorithms  Belief propagation  DP Stereo extended in all directions  Labels each pixel with a state – its disparity  Minimizes an energy function  Two terms  Data term – pixel difference  Smoothness term – penalizes changes (in disparity)  Iterates until convergence (or cutoff)  Exchanging messages with neighbouring pixels 27

SDPS Results 28 Corrected Left ImageCorrected Right Image Disparity MapOcclusion Map

Belief propagation  Best general performance  Simple BP being overtaken by segmentation-based algorithms  Memory intensive  Needs to store costs for each label for each pixel in a region  Compare last two pixels in scan line for SDPS  Iterates – so slow 29

30 Algorithm Comparison Symmetric DP stereo Graph cut Symmetric BP SCSM CSM Ground Truth

31 Middlebury Benchmark (MB) Algorithms TsukubaSawtooth alluntexdiscalluntexdisc MB-SBPO CSM SCSM * MB-SBPO – symmetric belief propagation algorithm ( best-performing Middlebury benchmark) Algorithms VenusMap alluntexdiscalldisc MB-SBPO CSM SCSM rank among 40 algorithms