LYU0503 Document Image Reconstruction on Mobile Using Onboard Camera Supervisor: Professor Michael R.Lyu Group Members: Leung Man Kin, Stephen Ng Ying Kit, Tommy

2 Outline Motivation Objective Symbian Operating System Previous Work Implementation Experimental results Difficulties Conclusion

3 Motivation Most people posses mobile phones Quality of onboard camera improves Processing power increases Mobile phones have many functions –Take photos, Music Jukebox –How to add more values?

4 Objective Reconstruct documents on mobile phones Input: Different parts of document Output: A high resolution document

5 Symbian Operation System A mobile OS from Symbian Ltd. Design for 2G, 2.5G and 3G mobile phones 32-bit preemptive multi-tasking OS Features: –Multi-tasking kernel –Mobile telephony support –Communication –Data management –Etc.

6 Symbian Operating System Symbian phone models –Nokia 6600 –Nokia 6630 –Nokia N90 Symbian SDK provided to develop program It enables programmers to gain more control on phone compared with java environment

7 Symbian Operating System Some testing results using Nokia N90 –It can allow use allocate at most 1M memory manually (e.g. malloc in C). –It can allow us to open Mega pixel gray scale image (using class CFbsBitmap). Why different? –The architecture of Symbian OS is client-server. –malloc in C is for client side. –The class is in the server side.

8 Previous Work Study basic idea of different image alignment and stitching techniques. Implemented & tested on PC –Use SIFT to find matching points –Stitch the image using the matching points Tested some related APIs on Symbian OS –Processing image –Fax API –Etc.

9 Implementation The implementation is divided into 5 parts –Take photos of the document –Lens correction of the photos –Mark matching points of the photos –Stitch the photos –Optimize the photo

10 Take photos of the document 1.Take the whole document –Align the distance between the camera and document to take photo of the whole document.

11 Take photos of the document 2.Take the individual part of the document –The camera is taken closer to the document to take the required image

12 Graphical User Interface To instruct the user to take photo of the individual part of the document –Overlaying is used.

13 Graphical User Interface Overlaying –Output.R = (Picture1.R + Picture2.R) / 2 –Output.G = (Picture1.G + Picture2.G) / 2 –Output.B = (Picture1.B + Picture2.B) / 2 –R = Red –G = Green –B = Blue The process is simple and quick –Complexity is O(Width * Height) –Involved one addition and one shift for each color component for each pixel. –Can be used in Symbian phone real-time

14 Take photos of the document After taking photo of the whole document, divide the whole document into several part like Use Overlay to display both the individual part and the current frame from the camera

15 Take photos of the document Before the overlaying, the size of both image should be the same –Cannot instruct the user to take photo correctly. –Resizing is used. –It would take time if interpolation is needed. Careful use of image size –The image resolution 1600 x 1200 (whole document) –The resolution from the camera 800 x 600 If bigger (the same as the image), the camera would be lagging –The wanted resolution is about 400 x 300. –To reduce 1600 x 1200 to 400 x 300 and 800 x 600 to 400 x 300 Just sub-sample the whole document. No interpolation is needed

16 Take photos of the document User is instructed to take photos correctly.

17 Take photos of the document Pseudo code Take phone of whole document Divide the whole document into four For each sub document While (there is next frame from camera) Sub-sampled sub-document Sub-sampled of the current frame Overlay the above two image and display if (User take photo of sub-document) Store the photo break

18 Lens correction Photos suffer from lens distortion –Barrel type –Pincushion type

19 Lens correction Image transformation –Transform the images using the equations

20 Lens correction

21 Mark matching points manually In order to reconstruct the photos well –Need to find match points between photo –Transform the individual photos according to the whole document The finding and matching feature points using SIFT need much time

22 Mark matching points manually On PC (3.0Ghz), it need about: –finding feature points: 4~5min each image –matching feature points: 4~5 min each pair of image –TOTAL = 45 min The processing power of Symbian phone is much less than the PC. –Nokia N90: RISC 220Mhz processer Mark the points manually would be faster and more reasonable.

23 Mark matching points manually Ask the user to mark the matching points

24 Mark matching points manually Like overlaying, resize is needed. –Only sub-sampling is needed. We use (x, y) coordinates to store for each points. In previous example, we would obtain 16 pairs of (x, y) coordinates. –Each individual image contributes 4 pairs –There are 4 individual images. The coordinates are stored and used to transform the individual photos in the later stage.

25 Mark matching points manually Pseudo code For each sub-document Display left half of the sub-document and corresponding part of whole document Marking two pairs of points Display right half of the sub-document and corresponding part of whole document Marking two pairs of points Store the coordinates into the file.

26 Mark matching points manually Marked Marker

27 Stitch the photos 1. Find transform Parameter Set 1 2. Transform corresponding coordinates using set 1

28 Stitch the photos 3. Find transform parameter set 2 by sub-image2 and transformed coordinates 4. Using set 2 to transform whole sub-image2

29 Stitch the photos 5. Similarly for all other sub-images 6. Stick them together and high resolution image formed!

30 Image Blending Bilinear Interpolation –A weighted mean of nearby pixel values

31 Image Blending Averaging Mask –Mean of pixel values of 8 neighbors

32 Image Blending Threshold –Quantize the pixel values by a threshold value

33 Experimental Results

34 Experimental Results

35 Experimental Results

36 Experimental Results

37 Experimental Results

38 Difficulties Limitation of processing speed –Image processing involves much computation –Processing in phone would take much time than PC. Limitation of memory –PC have much more memory and is allowed to use virtual memory. –Image processing, like image stitching, involves much memory

39 Conclusion We have studied different stitching algorithms We have implemented the stitching algorithm on both PC & Symbian phone Defects are still exist in the resulting image

40 ~ THE END ~