1. Whiteboard Scanning Using Super-resolution Wode Ni Advisor: John MacCormick COMP 491 Final Presentation Dec 18 2015

2. OUTLINE OF THE PRESENTATION Motivation Background – image formulation Superresolution algorithm OpenCV Experiment results Summary of this semester’s work Future work

3. MOTIVATION The “distance whiteboard” scenario: far away from the whiteboard, try to obtain a clear, scanned image of the whiteboard Difficulty: limitation of camera resolution Proposed method: Take a video, or multiple images and apply superresolution algorithm on the set of frames of the video/images and get an image of high resolution. LR image scanned by CamScanner

4. BACKGROUND Mitzel, Dennis, et al. "Video super resolution using duality based tv-l 1 optical flow." Pattern Recognition. Springer Berlin Heidelberg, 2009. 432-441. SymbolDegrading FactorsReasons DDownsamplingThe sensor of digital camera (typically CMOS) BBlurringOptical lens of the camera or atmospheric effect WWarpingMotion of the cameraman eAdditive noiseUsually introduced by high ISO

5. BACKGROUND - continued

6. SUPERRESOLUTION ALGORITHM Motion estimation: optical flow Need to enlarge the motion matrix using interpolation (linear) The algorithm will run through T times, and compute an estimation of HR image at a time. At the end of each iteration, the result is updated.

7. OPENCV Open source Computer Vision Library: now maintained by Itseez As for 3.0.0 version: 29 main modules, which includes superres. Extra modules contributed are also available Built-in data structures, memory management, and hundreds of computer vision algorithms available. Superres module uses methods from both Farsiu 2004 and Mitzel 2009.

8. SUPERRES Name in OpenCVSymbol in Mitzel09 Meaning Temporal Radius AreaNThe number of LR images to be processed. IterationsTThe number of HR estimations to be computed. Temporal Radius Area: does not exist in Mitzel’s paper. The OpenCV version of SR algorithm can output a video! As shown in Status Report I presentation, the effect on running time when increasing these parameters linear, meaning the same amount of work is done on each image and in each iteration. Result of running the algorithm: Demo

9. Running time of SR algorithm with varying parameters

10. SOME NEW RESULTS Computed Sum of Squared Difference(SSD) on SR result on the “synthesized image”. Increasing iterations: First decrease and increase Increasing temporal area: increase and converge Other error metrics?

11. WHAT I DID THIS SEMESTER… Tried to estimate the quality of the output quantitatively Studied the source code and understood more about the details of the implementation Did some black-box analysis on the OpenCV implementation of SR algorithm: running times of different values of the parameters Found existing SR implementations and decided to work with OpenCV Summer: studied basics of Computer Vision, C++, Linear Algebra, and OpenCV.

12. FUTURE WORK Coming back to what motivates the project: if we put the result of SR algorithm in CamScanner, will the quality of the scanned image be improved? How can we make the result even better? What is the computational cost and quality improvement of applying SR? Deeper into OpenCV: do some serious debugging (possibly modifying the source) and investigate the reason behind the weird SSD curve. Additional work: Study more math and try to understand the theoretical basis of SR better. Other comments and thoughts LR scanned image SR scanned image

13. Thank You! References: Dennis Mitzel, Thomas Pock, Thomas Schoenemann, and Daniel Cremers. Video super resolution using duality based tv-l 1 optical flow. In Pattern Recognition, pages 432–441. Springer, 2009. Sina Farsiu, M Dirk Robinson, Michael Elad, and Peyman Milanfar. Fast and robust multiframe super resolution. Image processing, IEEE Transactions on, 13(10):1327–1344, 2004.