1. Computational Framework for Performance Characterization of 3-D Reconstruction Techniques from Sequence of Images Ahmed Eid and Aly Farag Computer Vision and Image Processing (CVIP) Laboratory University of Louisville, Louisville, KY 40292 URL http://www.cvip.uofl.edu 2004

2. www.cvip.uofl.edu 2 Problem Description 3-D Modeling Techniques Testing Methodologies Definition of Ground Truth Quality Measures Quantification of the performance of 3-D modeling techniques by designing and defining 4 dependent modules: Test setup Ground truth Testing methodology Quality measures Design of Test beds

3. www.cvip.uofl.edu 3 Notations Ground truth set G : a finite set of reference 3D, 2D, or 1D points describing a 3D object or its images Data set M : a finite set of points obtained with a given 3D reconstruction technique from the collected images Comparable set D : a set of data points transformed in order to be compared to the ground truth G using a relevant transformation C Generally, performance characterization may need an additional registration of the transformed set M and G : The registration function T is a mapping: T : G  G’ such that  ( M  T(G) ) 2  min

4. www.cvip.uofl.edu 4 Motivations Objectives Many vision-based applications need accuracy validation. Lack of standard Evaluation methodologies and measures Difficulty of generating ground truth data 1- To introduce a model for performance evaluation systems 2- To build a database consisting of data acquired by our system 3- To provide a comparative study of different 3-D reconstruction techniques. 4- Validation, Evaluation and Design of vision-based systems.

5. www.cvip.uofl.edu 5 3-D reconstruction techniques Stereo based approaches Area basedFeature based Volumetric approaches Shape from silhouettes Voxel coloring Generalized voxel coloring Space carving Stereo Volumetric 3-D Reconstruction: An overview

6. www.cvip.uofl.edu 6 Test Setup  Rotating background scanner head CCD camera scanner base support  a- System components b- Top view  t

7. www.cvip.uofl.edu 7 Test Setup (cont.) 1- Sequence of images (discrete mode) 2- Panoramic images (continuous mode)

8. www.cvip.uofl.edu 8 Camera Calibration Compute the Projection Matrix P 0 at the initial position using calibration pattern. Assumption: the camera rotates around Y-axis by an angle   the projection matrix P k at view k is: where k=0, 2….., I  1 (I is the number of acquired images)

9. www.cvip.uofl.edu 9 for Setup Accuracy The error in the rotation angle is Where,

10. www.cvip.uofl.edu 10 Background Subtraction Acquire a sequence of background images Acquire a sequence of data images Subtraction model: normal p.d. of differences between color signals for the same background pixel –zero mean and a diagonal covariance matrix to be estimated [Elgammal’ 00] Use the color probability density estimate: The pixel is considered a foreground one if P(x d,x b ) < T 1 where T 1 is a chosen threshold. Apply median filter to fill gaps

11. www.cvip.uofl.edu 11 Background Subtraction (cont.) Example T 1 = 0.005 (a) input (b) background (c) Background subtraction (d) After median filtering (e) Restore colors

12. www.cvip.uofl.edu 12 3-D Data Registration Registration Through Silhouettes (RTS) Align ground truth G and the measured data M such that Where d denotes the distance Conventional 3-D registration techniques may fail in the evaluation problem because M may be corrupted set of data which or manual selection of matched points can be used. The proposed RTS method doesn’t use the set M through the registration process however, it uses the silhouettes of the input images

13. www.cvip.uofl.edu 13 RTS Procedure 1- Generate N s of input image silhouettes I M. 2- Generate N s of ground truth silhouettes I G using the projection matrices at same views of input silhouettes and the registration parameters  x,  y,  z, t x, t y, and t z. 3- Compute the error criterion: 4- Go to step 2, stop when is minimum. Minimum error criterion indicates full alignment of input silhouettes and ground truth silhouettes. This results in registration of G and M.

14. www.cvip.uofl.edu 14 Two-step optimization We used the a genetic algorithm (GA) to minimize the error criterion. Since the GA maximizes an objective function, then we use The used cross over rate is p c =0.95 and the mutation rate p m =0.01. Since GA takes long time to converge to exact solution, we get only approximate by GA then use it as initial solution to a local search method. Gene structure To be maximized. We used the gene structure shown below.

15. www.cvip.uofl.edu 15 (a) input images at 0, 90 o (b) Corresponding silhouettes of (a) (c) initial silhouettes of scanner (%10) (d) After 100 iterations of GA (%10) (e) After 150 iterations of simplex (%100) 3-D Registration (cont.)

16. www.cvip.uofl.edu 16 3-D Registration parameters (a) X-translation (c) Z-translation (b) Y-translation (d) X-rotation (e) Y-rotation (f) Z-rotation

17. www.cvip.uofl.edu 17 Registration Visual Results (a) before (b) after (c) standard

18. www.cvip.uofl.edu 18 It assesses the performance in the 3-D domain It decimates the registered 3-D data sets G ’ and M into N m batches. It uses the local centroids and deviations from centroids of corresponding pair of patches to assess the quality. The patch size is determined by: Where N m is a user defined parameter, and i is a positive integer and,, Performance Evaluation Techniques 1- Local Quality Assessment (LQA) Test

19. www.cvip.uofl.edu 19 Let C j d, is the centroid distance of patches M j and G’ j and R j D, is the deviation ratio of patches M j and G’ j Then similarity index Q j for the patches pair j is defined as: Where, Q j has range [0 1], with 1 represents the highest quality

20. www.cvip.uofl.edu 20 Example N m = 64 (a) bar graph of Q (b) histogram of Q in (a)

21. www.cvip.uofl.edu 21 Probability Estimate of Quality Let the pdf of beta distribution is defined as: a=0, b=1 for q  [0 1], and  and  are shaping parameters Where are the maximum likelihood estimators of beta distribution The probability estimate P is defined as:

22. www.cvip.uofl.edu 22 Results Applied to Space Carving with varying number of input mages (a) inputs=12 (c) Difference corresponds to (a) (b) inputs=9 Fattening

23. www.cvip.uofl.edu 23 2- Image Re-projection (IR) Test Performance Evaluation Techniques This test can be provided if the 3-D ground truth is not available It is distinct from the [Szeliski’99] in the way it is not restricted to the stereo approaches and it uses calibrated data so there is no need for prediction procedures. It uses quality measures that is used to assess the quality of real images. 1- Generate the image data set D of by projecting the data set M using projection matrices C at each view. 2- Compare D = C(M) to the original images G (as the ground truth) using a certain quality measure. Test Procedure

24. www.cvip.uofl.edu 24 Quality Measures 1- Signal to Noise Ratio for N x M images: Here and below g(i,j) and d(i,j) are intensities at the pixel (i,j) of the reference and data images, respectively Image Re-projection (IR) Test

25. www.cvip.uofl.edu 25 2- Quality Index Q [Z. Wang’02] where  g and  d are the s.d. of reference and data images, respectively g and d are the means of reference and data images, respectively  gd is the covariance between g and d Q has dynamic range [-1 1] and can be modified to Q m as Quality Measures (cont.) Image Re-projection (IR) Test

26. www.cvip.uofl.edu 26 3- Fuzzy Image Metric (FIM) [J. Li’02] based on differences where K  card(G)=card(D) and 0  g i,d i  1 after normalization where V is the domain, u  card ({. })/K, u(V)  1, and N l (f)={x|f(x)>l } FIM has the dynamic range [0 1], and the related IFIM is Quality Measures (cont.) Image Re-projection (IR) Test

27. www.cvip.uofl.edu 27 Results Applied to Space Carving at different resolutions  =1.25, 1.67, 2.00 and 2.50 mm No. of input images =36 (a) original (b)  =1.25 (c)  =1.67 (e)  =2.5 (d)  =2

28. www.cvip.uofl.edu 28 Applied to Space Carving at different numbers of input images 36, 18, 12, and 9 Results (a) inputs=12 (c) Difference corresponds to (a) (d) Difference corresponds to (b) (b) inputs=9 Fattening

29. www.cvip.uofl.edu 29 Parameter  =1.25 mm  =1.67 mm  =2.00 mm  =2.50 mm initial No. of voxels13997521592974134429511771561 No. of Input Images =36 final No. of voxels81535481903236120063 execution time (min.)126.839.418.57.8 mean (dB)9.1907.6326.4974.727 No. of Input Images =18 final No. of voxels8323349629 33310 20459 execution time (min.)55.721.1 3.9 mean (dB)9.0127.6316.4534.662 No. of Input Images =12 final No. of voxels83076496813330520606 execution time (min.)39.712.86.42.6 mean (dB)9.0007.4026.4454.652 No. of Input Images =9 final No. of voxels87591529063506921519 execution time (min.)26.89.14.31.8 mean (dB)7.0695.6245.2743.774 The effect of number of images on the quality of the output reconstructions and the execution time at different resolutions.

30. www.cvip.uofl.edu 30 Comparative Study It has two views: 1- Competitive view: Contrasts the differences between different approaches based on the proposed testing methodologies and measures. Different stereo techniques will be evaluated among themselves, then compared to the volumetric approaches 2- Cooperative view: Merge different techniques to achieve better quality. This integration depends on the outcome of the competition between algorithms. The LQA is suitable in this application Post-evaluation

31. www.cvip.uofl.edu 31 An Example of stereo and space carving comparison The IR test is applied to outputs of stereo and space carving on the same set of 12 input images. OriginalSpace Carving with true colors Stereo with true colors Space Carving with consistent colorsStereo with consistent colors

32. www.cvip.uofl.edu 32 Results SNR values at each view (12 images) for both space carving and stereo: (a) resolution; (b) resolution + color consistency (a)(b)

33. www.cvip.uofl.edu 33 Results Fuzzy measure Quality index

34. www.cvip.uofl.edu 34 3-D Data Fusion Given two reconstructions  1 and  2 (have some dissimilarities) of an object, then we need to fuse them into a single reconstruction 11 22 1- The proposed method uses the contours generated from intensity images to guide the fusion process. 2- Corresponding contours are then generated from the reconstructions  1 and  2 3- The evaluation methodology is applied. 4- The patches of low quality index are elected for the fusion process The Fusion Procedure

35. www.cvip.uofl.edu 35 The Fusion Procedure (cont.)  11 W0W0 22 11 22 V0V0 3D reconstructions projected contours 5- the corresponding contour segments of each low quality patch are determined 6- Two tests: (a)The closest point (b)The closest contour are applied to the contour segments (from the given reconstructions) to determine which of them is closer to the reference contour segment 7- The surface patch that has closest contour segment to the reference one is included in the output reconstruction

36. www.cvip.uofl.edu 36  11 W0W0 22 For each point, the closest points are calculated as follows: and, 1- Closest Point Test

37. www.cvip.uofl.edu 37 2- Closest Contour Test To determine which of, the average distances are calculated as: and The fusion decision is taken based on which of is minimum

38. www.cvip.uofl.edu 38 Results Given reconstructions (a) by 3-D laser scanner (b) by Space Carving

39. www.cvip.uofl.edu 39 Results (cont.) Silhouette from input image Silhouette from Space Carving Silhouette from scanner Contours from input image and space carving Contours from input image and scanner

40. www.cvip.uofl.edu 40 Results (cont.) Play video Space Carving Play video Laser scanner Play video Fused reconstruction

41. www.cvip.uofl.edu 41 Validation of Reconstructions of the Human Jaw Applications We plan use the evaluation framework to the validate the reconstructions of human jaw as a medical application. Two reconstruction approaches will be validated space carving and shape from shading with range data. Old method of validation used manual measurements from the real jaw then compared them to measurements on the reconstructed model

42. www.cvip.uofl.edu 42 The CardEye System at CVIP lab. Evaluation of the CardEye Trinocular Active Vision System The system employs a sensor planning technique for multiple camera vision systems. This allows the automatic selection of camera parameters (vergence, baseline, zoom and focus) that satisfy different vision constraints: field of view, focus, disparity and overlap. The effect of the above parameters on the output reconstruction will be investigated. We will study the effect of adding the third camera on the robustness of the output data. Applications

43. www.cvip.uofl.edu 43 Conclusions and Future Extensions While 3-D reconstruction approaches are used in many applications there are no widely accepted methodologies of quantifying the performance of these approaches. There is a real difficulty of having dense ground truth data. Most of the evaluation work done so far is dedicated to stereo approaches. In this study, we proposed the followings: 1- A design of a test bed that is able to collect optical data and dense ground truth data. Related problems to the system design are investigated such as: Camera calibration. System accuracy (we presented a closed form to system accuracy). Data segmentation (we proposed a background subtraction technique). Data registration (we proposed an efficient 3-D registration technique using silhouettes (RTS).

44. www.cvip.uofl.edu 44 Conclusions and Future Extensions 2- Two testing methodologies: A 3-D evaluation test, the (LQA) test, that can detect local errors in the 3-D model. Which makes it suitable for error analysis and fusion techniques. A 2-D evaluation test, the (IR) test that uses input images as an implicit ground truth. The test is suitable for investigating the quality of the output for applications of virtual reality. 3- We provided an example of evaluation the performance of stereo and space carving under the same framework. In addition we propose the following extensions: 1- Find a relation between the error distribution in 2-D and 3-D. The will validate the use of images as ground truth. 2- Differentiate between the errors that caused by the 3-D technique itself or that is may be introduced by the testing methodology itself. 3- Study object recognition techniques to find good description of an object. This will give good measured to be used in the LQA test.

45. www.cvip.uofl.edu 45 Conclusions and Future Extensions 4- Provide more comparisons between different reconstruction techniques. 5- Enhance the quality of 3-D reconstructions using data integration techniques. 6- Validate the results provided by human jaw modeling system 7- Evaluate the performance of the Cardeye system. 9- Design a 3-D passive scanner