Image Scaling Jackie Van Ryzin

Outline Project Description –Definition & Requirements –Solutions (what I did) –Exceptions (what I didn’t get to) –Methodology (how I did it) Demonstration – Zooming fun! Learning & Development Process –Strategies (what works for me) –Knowledge (useful CS concepts) –Extensions (where to from here) –Advice (for those yet to come) Q & A

Project Description Develop algorithms for real-time scaling that maintains quality Study and implement smooth Bresenham nearest neighbor interpolation method Look at ways to improve these methods Develop new algorithms or alter existing Implement a user interface that would allow someone to zoom in and out as they watch a streaming video

Overall Plan Read, research, understand existing algorithms Implement those algorithms using general arrays Create program to read and alter images Combine image program and algorithms Develop interface to test and compare algorithms and variations of existing one Alter programs & interface to scale real-time streaming video in addition to still images

In the beginning… Thiadmer Riemersma was unhappy with: –Quality of fast image scaling algorithms –Slow processing for high quality scaling routines He developed smooth Bresenham algorithm (Dr. Dobb’s Journal, May 2002) Used concept of Bresenham’s line algorithm to scale a horizontal line of pixels

Background in Bresenham Determine which points of an n-dimensional grid to plot to best approximate a straight line between two given points (x 0,y 0 ) and (x 1,y 1 ) For each x between x 0 and x 1, choose row y that most nearly matches fcn value to plot pixel (x,y)

To determine y General line formula is Know column x so solve for row y as Can pre-calculate constant Since slope between 0 & 1 by assumption, after rounding we either use same y or add 1 to it

Tracking Error User error value to track difference between current y value and exact y value for current x As increment x++, increase error by slope When error > 0.5, line is closer to next y value rather than current/previous –y++ and error--

Bresenham for Images Pixels are picked up from discrete positions in source image and placed at discrete locations in the destination image Nearest neighbor sampling –Magnifying: pixels duplicated –Minifying: pixels dropped –Both leave artifacts in destination image More accurate sampling requires reading pixels from fractional positions in source image Techniques exist that incorporate weighted averages – computation-intensive

A “Lightweight” Alternative Compromise between linear interpolation & coarse Bresenham scaling Destination pixel set to: –Value of closest source pixel –Unweighted average of neighbors –Based on proximity to source pixel Original 150% 75%

Smooth Bresenham (cont’d) Merit: –Standard (unweighted) averages = simpler calculations –Increased performance speed – important for real- time applications –Dropped pixels & excessive “jaggies” of fast nearest- neighbor techniques (such as coarse Bresenham) are absent or kept to a minimum Adjust this algorithm for 2D images –Nested for loops –Call horizontal line scaling algorithm –Average scaled lines with each other

How we use this… b (0,0) g (0,0) r (0,0) b (1,0) g (1,0) r (1,0) b (2,0) g (2,0) r (2,0) : : b (0,1) g (0,1) r (0,1) b (1,1) g (1,1) r (1,1) : : b (width-1,height-1) g (width-1,height-1) r (width-1,height-1) Use CV’s split fcn to get 3 single-channel images from a 3-channel image Run algorithm on each before remerging with CV’s merge fcn CAREFUL with averaging! OpenCV Platform Images in IplImage format which has char* pointing to data Data stored linearly as bytes as shown

Zooming! At first could capture keys (used 1 & 2) with cvWaitKey for user to scale up or down Discovered cvTrackBar –Attaches nicely to window –Auto updates the variable (scale factor) –Very user-friendly & visually appealing!

Analysis First scale image using Bresenham, then “undo” the scaling by using Bresenham on scaled image to get back to original size Compare with original image (calculate average difference between pixels of original and “un-scaled” images) Do same with cvResize functions & compare results

I Wish I Could/Would Have... Been better able to convert bytes to ints and back –Would have condensed processing time by calling algorithm only once (on 3-channel image rather than separately on 3 1-channel images) Been able to implement more alterations to the algorithm for comparison and analysis Looked into MIP mapping as mentioned in DDJ –Quick scaling by factor of 2 Used actual calculations to determine speed rather than simply visual

Methods Some mini-lectures/discussions with Dr. Pankratz & Dr. McVey in beginning to understand basic algorithm Test algorithms on arrays of ints first! (it was a while before I tried images) Tried using VB at first because Professor Blahnik provided code for reading from a webcam –Converting algorithm into VB (pointers) –DLL to leave it in C code –Picture box didn’t resize when image did

Methods (cont’d) OpenCV –Started with sample program from developers –Strip it down to minimum with John’s help –Add features and functions in baby steps Always had two versions –Still images using LoadImage –Webcam using Capture Wrote functions in small test program to run on arrays of ints before adding to interfaces

Let’s Scale! Real-time scrolling from webcam –HALLELUJAH!\scroll_webcam\scroll_webcam.dswHALLELUJAH!\scroll_webcam\scroll_webcam.dsw Compare with CV resizing using still image –HALLELUJAH!\diff_cv\diff_cv.dswHALLELUJAH!\diff_cv\diff_cv.dsw How well does it “undo” the scaling? –HALLELUJAH!\rescale_cmp\rescale_cmp.dswHALLELUJAH!\rescale_cmp\rescale_cmp.dsw Does a different threshold matter? –HALLELUJAH!\cmp_threshold\cmp_threshold.dswHALLELUJAH!\cmp_threshold\cmp_threshold.dsw

Where did this stuff come from? Dr. Dobb’s Journal OpenCV manual Dr. Pankratz & Dr. McVey Forums and online tutorials for OpenCV Practice, experimenting, trial-and-error John, Ted, Ryan P

Important CS Concepts Data Structures –Arrays, pointers, structures, chars vs. unsigned chars! –Dynamic memory allocation & de-allocation Machine Org. & Assembly Language –Fast calculations for dividing & masking Programming Languages –Designing abstractly without a language All classes –General learning, experimenting, researching techniques

Where to From Here? Be able to convert to array of ints & back to reduce calls to scaling algorithm Test & compare speed using actual clock calculations Find a better mathematical/computational method for comparing how “nice” a picture looks scaled Alter algorithm in some way to improve quality and/or speed Investigate MIP mapping and using a combination of MIP & Bresenham

For Those Who Follow Meet with DCP and other profs OFTEN and right away Ask questions of anyone and everyone –Someone may have a unique solution you wouldn’t think of –Could make you discover a solution with a simple comment about how chars in C aren’t fun When you get stuck, admit it and ask for help Write your journal! It keeps you accountable and encourages progress. Paper can be useful! Draw images, diagrams, etc. It helps to see it before you program it Baby steps…starting small is easier, and every little milestone is rewarding. It also makes testing & debugging easier.

Q & A Thanks to Dr. Pankratz and Dr. McVey for the guidance and ideas! Thank you to classmates for ideas, answers, and suggestions.