Composition-Guided Image Acquisition Serene Banerjee Ph.D. Defense, April 28 th, Committee Members: Prof. Ross Baldick Prof. Alan C. Bovik Prof. Brian L. Evans (Advisor) Prof. Wilson S. Geisler Prof. Joydeep Ghosh Prof. Robert W. Heath, Jr. Computer Engineering Curriculum Track Dept. of Electrical and Computer Engineering The University of Texas at Austin

4/28/2004 Composition-Guided Image Acquisition 2 “One day Alice came to a fork in the road and saw a Cheshire cat in a tree. ‘Which Road do I take?’ she asked. ‘Where do you want to go?’ was his response. ‘I don’t know,’ Alice answered. ‘Then,’ said the cat, ‘it doesn’t matter.” Lewis Carroll Alice in Wonderland

4/28/2004 Composition-Guided Image Acquisition 3 Outline Introduction Motivation Overview of contributions Summary of previous research for main subject detection Contributions Online main subject detection Aesthetic enhancements, given main subject Blur background objects merging with main subject Conclusions

4/28/2004 Composition-Guided Image Acquisition 4 Motivation Problem: Amateur photographers take unappealing pictures (e.g. personal and business use) Help users take better pictures with digital cameras Main subject cropped Too much background No foreground / background distinction

4/28/2004 Composition-Guided Image Acquisition 5 Enhance Picture Appeal Improving photograph appeal [Savakis, Etz & Loui; 2000] Photographic composition Objective measures People/expression Examples of photographic composition rules Rule-of-thirds Amateur Placement Professional Blur background Amateur Shot Professional Avoid Merger Amateur ShotProfessional

4/28/2004 Composition-Guided Image Acquisition 6 Enhance Acquired Picture Appeal Goal: Provide well-composed alternative pictures during image acquisition in digital still cameras Solution: Framework for in-camera automation of photographic composition rules Acquire picture user intended to take Locate main subject by combining optical and digital image processing on a supplementary picture Apply composition rules to user-intended picture Place main subject according to rule-of-thirds Blur entire background given main subject location Blur background objects that merge with main subject User takes intended picture and framework also returns three alternative pictures

4/28/2004 Composition-Guided Image Acquisition 7 Offline Main Subject Detection Neural network based training [Luo, Etz, Singhal & Gray; ] Cluster multi-level wavelet coefficients [Wang et al.; ] Iterative classification from variance maps [Won, Pyan & Gray; 2002] AlgorithmTraining complexityRuntime complexity Neural networkDifficult to form widely applicable training set High (e.g. feature extraction, grouping) Wavelet-basedNo training requiredHigh (e.g. wavelet, k- means clustering) Variance-basedNo training requiredHigh (e.g. iterations, watershed)

4/28/2004 Composition-Guided Image Acquisition 8 Automating Composition Rules Detect main subject Rule-of- thirds Background blur Mitigate merger Original color image Generated picture with rule-of-thirds Generated picture with blur Generated picture without mergers In-camera online framework Provide alternatives to user during image acquisition One-pass low-complexity algorithms [Banerjee & Evans; ] Independent of scene content and setting Amenable to fixed-point implementation Match processing on digital still cameras Supplementary picture

4/28/2004 Composition-Guided Image Acquisition 9 Digital Still Cameras Converts optical image to electric signal Software control Shutter aperture and speed Focus Zoom White balance Additional hardware could control Camera angle Aspect ratio: landscape or portrait

4/28/2004 Composition-Guided Image Acquisition 10 Outline Introduction  Contributions  Online main subject detection  In-camera segmentation of the main subject  Low-complexity one-pass algorithm  Amenable to implementation in digital still cameras Aesthetic enhancement, given main subject Mitigation of mergers with background objects Conclusions

4/28/2004 Composition-Guided Image Acquisition 11 Online Main Subject Detection Auto-focus main subject Take supplementary picture Open shutter aperture (takes 1s) to blur objects not in focus In-focus edges stronger than out- of-focus edges Process supplementary picture to find main subject mask Enhance in-focus edges Detect strong edges Close boundary Contribution #1 3x3 Highpass filter Detect sharper edges Close boundary Auto-focus filter Open shutter for blur Scene Binary main subject mask Compute intensity Supplementary picture

4/28/2004 Composition-Guided Image Acquisition 12 Supplementary picture has intensity function, I I H and I L are highpass and lowpass versions For background image, contribution from I L is greater Goal: Identify pixels contributing high frequencies I is modeled as mixture of I H and I L Highpass filtering of I enhances main subject edges Main Subject Detection: Formulation Contribution #1 where k  1

4/28/2004 Composition-Guided Image Acquisition 13 Step 1: Enhance In-focus Edges Subtract smoothed image from sharpened one Strong edges in main subject, weak edges in background Σ Supplementary image Lowpass image Highboost image + - Contribution #1 Edge-enhanced image with stronger main subject edges

4/28/2004 Composition-Guided Image Acquisition 14 Step 2: Detect Strong Edges Canny edge detector detects strong edges [Canny; 1986] Selects weak edges only if they are connected to strong edges Laplacian of Gaussian detector [Burt & Adelson; 1983] Selects edges based on zero crossings of second derivative Either detects weak and strong edges or eliminates weak edges from main subject (depends on threshold) Contribution #1 Canny edge detector Laplacian of Gaussian

4/28/2004 Composition-Guided Image Acquisition 15 Step 3: Generate Mask Goal: Generate closed contour from strong edges Gradient vector flow [Xu, Yezzi & Prince; 2001] Balances forces Internal: spline characteristics External: normal of gradient of detected strong edges Outer boundary of detected sharp edges is initial contour Change shape of initial contour, depending on gradient Approximate lower complexity method Select leftmost & rightmost “ON” pixel and make row pixels in between them “ON” Can detect convex regions but fails at concavities Contribution #1

4/28/2004 Composition-Guided Image Acquisition 16 Main Subject Detection Results Supplementary image Step 1: Edge map Step 2: Strong edge detection Step 3a: Gradient of strong edges Step 3b: Gradient vector flow field Step 3c: Initial contour Step 3d: Contour after 5 iterations (not mandatory) Main subject mask Contribution #1

4/28/2004 Composition-Guided Image Acquisition 17 Implementation Complexity Per-pixel complexity for algorithm [Banerjee & Evans; ] Multi-level wavelet based [Wang, Lee, Gray, Wiederhold; ] Variance of multi-level wavelet coefficients: ~2X increase k-means clustering: 2(image size)(no. of iterations)X increase Iterative classification from variance maps [Won et al.; 2002] Iterative maximum a posteriori segmentation: ~3X increase Watershed refinement: 6 passes per pixel Contribution #1 Process/OperationMultiply-accumulatesComparesMemory accesses Pre-filtering (3x3)9 Edge detection925 Close boundary21

4/28/2004 Composition-Guided Image Acquisition 18 Comparison With Previous Methods Original image Proposed algorithm [Banerjee & Evans; ] Wavelet-based [Wang et al.; ] Variance maps [Won, Pyan & Gray; 2002] Contribution #1

4/28/2004 Composition-Guided Image Acquisition 19 Limitations Frequency-based features not applicable if Main subject does not have enough high frequencies Background not blurry enough Could incorporate region-based features Example of an image where the proposed algorithm fails to detect the main subject, the flower Contribution #1

4/28/2004 Composition-Guided Image Acquisition 20 Outline Introduction  Contributions Main subject detection  Aesthetic enhancement, given main subject  Reposition main subject to follow rule-of-thirds  Simulate background blur for motion or clarity Mitigation of mergers with background objects Conclusions

4/28/2004 Composition-Guided Image Acquisition 21 Rule-of-Thirds Better interaction of main subject with image background Center of mass of main subject at 1/3 or 2/3 picture width (or height) from the left (or top) edge Contribution #2 Main subject in center of picture Main subject follows rule-of-thirds Outdoor setting; the flower is main subject

4/28/2004 Composition-Guided Image Acquisition 22 Rule-of-Thirds Algorithm Compute center-of-mass of main subject 2 multiply-accumulates, 1 memory read per pixel 1 division per image Locate closest one-third corner 8 compares per image (4 comparisons of (x,y) points) Shift picture so center-of-mass falls at desired corner Mirror undefined boundary pixels Best case: no change to image Worst case: 1/3 rows/columns need to be shifted Average (main subject in middle): shift 1/6 rows/columns 0 to 2 memory accesses per pixel Contribution #2

4/28/2004 Composition-Guided Image Acquisition 23 Ideal Background Blur Example Contribution #2 Background blur emphasizes main subject, the shell, and aids in constrained image communication Indoor setting; no humans in picture

4/28/2004 Composition-Guided Image Acquisition 24 Simulated Background Blur Possible camera blurs Background blur: shutter aperture Linear blur: subject or camera motion Radial blur: camera rotation Zoom: change in zoom Digital alternatives Original image masked with detected main subject mask Region of interest filtering performed on non-masked pixels Complexity: 9 multiply-accumulates and 4 memory accesses per pixel for convolution with symmetric 3x3 filter Contribution #2

4/28/2004 Composition-Guided Image Acquisition 25 Results (1) Supplementary image with main subject(s) in focus Detected main subject mask Rule-of-Thirds: Main subject repositioned Simulated background blur Contribution #2 Outdoor setting; human main subject

4/28/2004 Composition-Guided Image Acquisition 26 Results (2) Supplementary image with main subject(s) in focus Detected main subject mask Rule-of-Thirds: Main subject repositioned Simulated background blur Contribution #2 Outdoor setting; human main subject

4/28/2004 Composition-Guided Image Acquisition 27 Results (3) Supplementary image with main subject(s) in focus Detected main subject mask Rule-of-Thirds: Main subject repositioned Simulated background blur Contribution #2 Indoor setting; no human subjects

4/28/2004 Composition-Guided Image Acquisition 28 Outline Introduction  Contributions Main subject detection Aesthetic enhancement, given main subject  Mitigation of mergers with background objects  Framework for background analysis and merger detection  Low-complexity one-pass algorithm for merger mitigation Conclusions

4/28/2004 Composition-Guided Image Acquisition 29 Ideal Merger Mitigation Example Contribution #3 Unwanted mergers avoided Background bar merges with gymnast’s hand

4/28/2004 Composition-Guided Image Acquisition 30 Mitigation of Mergers: Overview Goal: Identify background objects merging with main subject In-focus background object Connected to main subject mask Large area relative to image size Merger detection Color segmentation based on hue Identify distracting background object based on distance to main subject and frequency content Blur merging background objects to induce a sense of distance Contribution #3 Merging background objects: trees and bush over right shoulder

4/28/2004 Composition-Guided Image Acquisition 31 Segmentation of Background Objects Hues above histogram average are dominant hues Background is a mixture of dominant hues Thresholds: average of two consecutive dominant hues Contribution #3 Background hues Histogram of background hues and identified objects Thresholds = {87, 151}

4/28/2004 Composition-Guided Image Acquisition 32 Merger Object Detection Define Frequency Inverse Distance Measure for each disjoint background object O i Decreases with nearest distance (d i ) from main subject Increases with high spatial frequency coefficients (ω i H ) Merged object: Object with highest transform value Contribution #3

4/28/2004 Composition-Guided Image Acquisition 33 Measure Selection Linear, division, and exponential forms to combine High frequencies computed with residual in Gaussian pyramid decomposition Euclidean distance measured from main subject mask AttributeLinearDivisionalExponential Computational complexity LowHigh Merged object’s size LargeSmall Contribution #3

4/28/2004 Composition-Guided Image Acquisition 34 Merger Mitigation Results Background tree and bush merging with main subject High frequency and inv. distance values for background Blurred tree and bush appear to be farther away Contribution #3

4/28/2004 Composition-Guided Image Acquisition 35 Per-pixel Implementation Complexity Contribution #3 Process /OperationMultiply- accumulates ComparesMemory accesses RGB to hue364 Histogram and thresholding12 RGB to intensity2 Gaussian pyramid94 Approx. inv. distance measure212 Detect merged object11 Gaussian pyramid reconstruction915 TOTAL For comparison, JPEG compression takes ~60 operations/pixel

4/28/2004 Composition-Guided Image Acquisition 36 System Prototype Generated picture with blur Merger mitigated picture Measure how close rule-of-thirds followed Scene Automate rule-of-thirds Simulate background blur Generated picture with rule-of-thirds Binary main subject mask Intensity Gaussian pyramid Background segmentation Inverse distance transform Grayscale image X Detect merging object Grayscale image Reconstruct color pyramid Color Gaussian pyramid Transform coefficients 3x3 Highpass filter Detect sharper edges Close boundary Auto-focus filter Open shutter for blur Compute intensity Original color image Supplementary image

4/28/2004 Composition-Guided Image Acquisition 37 Conclusion Contributions Combined optical/digital image acquisition Provide online feedback to amateur photographers Low-complexity one-pass method for main subject detection Rule-of-thirds: placement of the main subject on the canvas Simulated background blur: motion and depth-of-field Mitigation of mergers with background objects Deliverables Prototype development for digital still image acquisition Copies of MATLAB code, slides, and papers, available at

4/28/2004 Composition-Guided Image Acquisition 38 Future Work Automate other photographic composition rules Best zoom Available frames, lines of interest, best angle, balanced picture Extension for video acquisition Frame-by-frame basis Compressed domain Digital image stabilization: Subject mask as feature Potential research impact: Video cameras, Surveillance, Image/video retrieval, Constrained image/video communication, Main subject detection for specific applications