1. MACHINE VISION GROUP GPGPU-based surface inspection from structured white light Miguel Bordallo 1, Karri Niemelä 2, Olli Silvén 1 1 Center for Machine Vision Research - University of Oulu, Finland 2 VTT - Technical Research Center of Finland, Oulu, Finland Jari Hannuksela, Olli Silvén Machine Vision Group, Infotech Oulu Department of Electrical and Information Engineeering University of Oulu, Finland

2. MACHINE VISION GROUP Contents Introduction Automatic Surface Inspection Phase extraction from white structured light Practical problems Measuring Prototype Design and construction GPU as a computing engine Experimental setup Description of the system Algorithms and Implementation Experiments Qualitative results Speed and scalability Summary

3. MACHINE VISION GROUP Motivation Automatic surface inspection used in the industry: –To detect all kinds of surface defects –To measure the overall quality of a produced piece Most convenient inspection method should provide exact 3D information High speed of production lines need: –Fast imaging methods –Lots of computational power Systems must be cost effective: –Standard PCs –Graphics Processing Units (GPUs)

4. MACHINE VISION GROUP GPU as a computing engine All computers and many embedded systems include a GPU Standard PCs and components Cost-effective systems Highly scalable GPU can be treated as an independent entity Graphics Processing Units offer important parallelization capabilities GPUs offer ”many-core” computation Thousands of threads can be executed concurrently. GPU and CPU can be used concurrently If data transfer is small, CPU load remains low (CPU can be used for other tasks) CUDA is a highly optimized and attractive accelerator interface

5. MACHINE VISION GROUP Surface topography from white structured light (SLS) Phase-shifting methods: –Based on fringe pattern projections or structured light –Extensively utilized in topography measurement –Provide for high resolution height measurements on each pixel. The illuminator projects a sine pattern: –On a moving target –In a synchronized manner The camera system obtains suitable input pictures using: –Pulse-like illumination –Synchronized camera subsystem –Certain known rate

6. MACHINE VISION GROUP Phase measurement In practice δ1, δ2, δ3 are not known in beforehand If δ1, δ2, δ3 are known: And the height: The input images are defined by the following:

7. MACHINE VISION GROUP Phase extraction with syntetic images 120dg Phase Shifted patterns Reconstructed Images Phase/Height Comparison

8. MACHINE VISION GROUP Problems and errors Clipping effect: saturation Wrong phase shift (δ1, δ2, δ3) Wrong frequency Combined effect ++ =

9. MACHINE VISION GROUP Problems and errors Clipping effect: saturation Wrong frequency Combined effect Input ++ = + Wrong phase shift (δ1, δ2, δ3)

10. MACHINE VISION GROUP Problems and errors Clipping effect: saturation Wrong frequency Combined effect Input Result ++ = += Wrong phase shift (δ1, δ2, δ3)

11. MACHINE VISION GROUP Prototype design

12. MACHINE VISION GROUP Prototype design VTT prototype: Sine period of 250um –Camera: Basler Scout scA 1600-14gm. 1628x1236 pixels, Area 4.4*4.4um2 –Interface: GiGE, 17 frames per second –Optics: Optosigma Telecentric (TC1236). Pixel size 30 µm –Illuminator: 9 Luxeon K2 Red LEDS + collimating lens. 3 channels Laptop: Lenovo W700 –CPU: Intel Core 2 Extreme QX9300 2.53 GHz –GPU: Nvidia Quadro FX3700 (128 cores) –IDE: Visual Studio. CUDA & C code environments Motor Line Controller: ATMEL microcontroler and PC –Line speed: 0,3 m/s Samples used: –Offline: 10 cents coin, printed electronics (10 µm thick) –Online: MDF-fiberboard

13. MACHINE VISION GROUP Prototype construction

14. MACHINE VISION GROUP Application flow

15. MACHINE VISION GROUP Application flow

16. MACHINE VISION GROUP Application flow

17. MACHINE VISION GROUP Application flow

18. MACHINE VISION GROUP Application flow

19. MACHINE VISION GROUP Application flow

20. MACHINE VISION GROUP Application flow

21. MACHINE VISION GROUP Application flow

22. MACHINE VISION GROUP Input images Full frame size: 1628x1236 pixels, 8 or 10 bpp, grayscale, 17 fps

23. MACHINE VISION GROUP Input images 64x256 correlation area Full frame size: 1628x1236 pixels, 8 or 10 bpp, grayscale, 17 fps

24. MACHINE VISION GROUP Image registration Based on modified phase correlation –Tukey window + FFT-based (+ Gaussian filtering) –Robust to blur (even motion blur) –Robust to image intensity changes –Fast to compute Easy to parallelize –CUDA FFT routines already optimized –Per-pixel operations Identifies corresponding pixels –Subpixel level access as a CUDA texture object Predict initial phase shift for phase computation Fine tune the motor displacements & camera rate 600x300 ROIs

25. MACHINE VISION GROUP Correlation algorithm performance FFTMultiplication & Normalization Correlate 2 images (3 fft + mul./norm.) Correlate 3 full frames ( 5 fft + 2 mul./norm.) Matlab Intel Core2 2.6Ghz CUDA Nvidia Quadro FX1700 CUDA Nvidia Quadro FX3700 Matlab Intel Core2 2.6Ghz CUDA Nvidia Quadro FX1700 CUDA Nvidia Quadro FX3700 Matlab Intel Core2 2.6Ghz CUDA Nvidia Quadro FX1700 CUDA Nvidia Quadro FX3700 Matlab Intel Core2 2.6Ghz CUDA Nvidia Quadro FX1700 CUDA Nvidia Quadro FX3700 64x256 Time /SpeedUp 18 ms3.5 ms 0.9 ms 4 ms 0.3 ms <0.1 ms 62 ms 12 ms 2.9 ms 100 ms 20 ms 4 ms X5X20XX12X>40XX5X22XX5X25X 128x512 Time /SpeedUp 70 ms 13 ms 3 ms 15 ms 1.1 ms 0.2 ms 230 ms 40 ms 6.1 ms 390 ms 65 ms 15 ms X5.5X25XX14X67XX5.8X37.5XX6X30X 256x1024 Time /SpeedUp 275 ms 42 ms 9 ms 58 ms 3.5 ms 0.7 ms 820 ms 120 ms 24 ms 1500 ms 200 ms 41 ms X6.5X30XX17X80XX7X34XX7.5X36X

26. MACHINE VISION GROUP Advanced Phase Shifting Algorithm (APSA) First introduced by Z. Wang in 2004 Iterative algorithm –Initial estimation of phase difference (δ1, δ2, δ3) from correlation and previous frames –Phase of each pixel is computed Using a CUDA 2-dimensional kernel –Average phase of the image is computed By adding together the values of all the pixels Using CUDPP parallel reductions –Average phase is the new phase difference –Iterate until convergent and error < threshold Result is a phase wrapped image –Range between -π and π Wrapped image

27. MACHINE VISION GROUP APSA times AlgorithmMATLAB timeC/CUDA timeSize Mpix/s SpeedUp APSA1: Phase extraction (CUDA) 130,0 ms/iteration10,9 ms/iteration350x82626,5211x APSA2: Average phase (CUDA) 470,0 ms/iteration18,8 ms/iteration350x82615,1124x APSA 10 iterations6200 ms300 ms350x8260,9520x

28. MACHINE VISION GROUP Phase unwrapping and surface fitting Lp Norm algorithm: –Developed in CUDA (Mistry, 2009) –Accurate results –Very high computation times (up to 2.5 seconds) –Not suitable for real-time Sorting by reliability in noncontinuous path: –Fast two dimensional unwrapping –Developed in C for a CPU (Arevalillo 2004) –Sufficient accuracy –Very fast (about 125 ms.) –Can be executed concurrently with the GPU phase extraction Surface fitting computes the closer average plane 600x300 Surface map

29. MACHINE VISION GROUP Display system

30. MACHINE VISION GROUP Automatic calibration system Phase maps measured continuously in real time –The information of the phase extraction process can be used to improve further results and conditions. Synchronizes –Illumination, –Camera capture –Motor speed Input parameters: –Correlation results (adjust motor speed) –Phase average (adjust illumination and camera capture Phase tuning and system calibration improve the results gradually

31. MACHINE VISION GROUP Real-time tests: MDF fibreboard sample

32. MACHINE VISION GROUP Real time tests: 3D representation

33. MACHINE VISION GROUP Printed electronics sample

34. MACHINE VISION GROUP Complete system Image size: 3 ROI of 600x300 Computation time: < 150 ms. Frame rate: > 5 fps. Resolution: 30µm per pixel. Phase unwrapping Copy Images as texture Get input frames N = 1 CPUGPU Get Correlation ROI Get Surface ROI Get pixel phase Get average phase APSA1 Get average phase APSA2 Get phase map Surface fitting Forward correlation values Forward phase average values Perform correlation

35. MACHINE VISION GROUP Complete system Image size: 3 ROI of 600x300 Computation time: < 150 ms. Frame rate: > 5 fps. Resolution: 30µm per pixel. Phase unwrapping Get input frames N = 1 CPUGPU Surface fitting Forward correlation values Forward phase average values Copy Images as texture Get Correlation ROI Get Surface ROI Get pixel phase Get average phase APSA1 Get average phase APSA2 Get phase map Perform correlation

36. MACHINE VISION GROUP Complete system Image size: 3 ROI of 600x300 Computation time: < 150 ms. Frame rate: > 5 fps. Resolution: 30µm per pixel. Phase unwrapping Surface ftting Get input frames N = 1 CPUGPU Calculate wrapped phase Image N = 1

37. MACHINE VISION GROUP Complete system Image size: 3 ROI of 600x300 Computation time: < 150 ms. Frame rate: > 5 fps. Resolution: 30µm per pixel. Phase unwrapping Surface fitting N = 1 Calculate wrapped phase Image N = 1 Get input frames N = 1 CPUGPU

38. MACHINE VISION GROUP Complete system Image size: 3 ROI of 600x300 Computation time: < 150 ms. Frame rate: > 5 fps. Resolution: 30µm per pixel. Phase unwrapping Surface fitting N = 1 Calculate wrapped phase Image N = 1 Get input frames N = 1 Get input frames N = 2 CPUGPU

39. MACHINE VISION GROUP Complete system Image size: 3 ROI of 600x300 Computation time: < 150 ms. Frame rate: > 5 fps. Resolution: 30µm per pixel. Phase unwrapping Surface fitting N = 1 Calculate wrapped phase Image N = 1 Calculate wrapped phase Image N = 2 Get input frames N = 1 Get input frames N = 2 CPUGPU

40. MACHINE VISION GROUP Complete system Image size: 3 ROI of 600x300 Computation time: < 150 ms. Frame rate: > 5 fps. Resolution: 30µm per pixel. Phase unwrapping Surface fitting N = 1 Calculate wrapped phase Image N = 1 Calculate wrapped phase Image N = 2 Phase unwrapping Surface fitting N = 2 Calculate wrapped phase Image N = 3 Get input frames N = 1 Get input frames N = 2 Get input frames N = 3 CPUGPU

41. MACHINE VISION GROUP Complete system Image size: 3 ROI of 600x300 Computation time: < 150 ms. Frame rate: > 5 fps. Resolution: 30µm per pixel. Phase unwrapping Surface fitting N = 1 Calculate wrapped phase Image N = 1 Calculate wrapped phase Image N = 2 Phase unwrapping Surface fitting N = 2 Calculate wrapped phase Image N = 3 Get input frames N = 1 Get input frames N = 2 Get input frames N = 3 CPUGPU

42. MACHINE VISION GROUP Complete system Image size: 3 ROI of 600x300 Computation time: < 150 ms. Frame rate: > 5 fps. Resolution: 30µm per pixel. Phase unwrapping Surface fitting N = 1 Calculate wrapped phase Image N = 1 Calculate wrapped phase Image N = 2 Phase unwrapping Surface fitting N = 2 Calculate wrapped phase Image N = 3 Get input frames N = 1 Get input frames N = 2 Get input frames N = 3 Get input frames N = n Phase unwrapping Surface fitting N = n-1 Calculate wrapped phase Image N = n CPUGPU

43. MACHINE VISION GROUP Complete system Image size: 3 ROI of 600x300 Computation time: < 150 ms. Frame rate: > 5 fps. Resolution: 30µm per pixel. Phase unwrapping Surface fitting N = 1 Calculate wrapped phase Image N = 1 Calculate wrapped phase Image N = 2 Phase unwrapping Surface fitting N = 2 Calculate wrapped phase Image N = 3 Get input frames N = 1 Get input frames N = 2 Get input frames N = 3 Get input frames N = n Phase unwrapping Surface fitting N = n-1 Calculate wrapped phase Image N = n CPUGPU

44. MACHINE VISION GROUP Summary A sine projection technique is a suitable method to optically measure a layer-like surface topography The system could be used in rapid motor lines with proper synchronization An integrated automatic calibration system helps synchronization and increases quality and robustness High accuracy can be achieved with fast imaging methods and intensive computation Time critical algorithms can be executed with GPU-based parallel computing

45. MACHINE VISION GROUP Thank you! Any questions ???