1. CS 1699: Intro to Computer Vision Matlab Tutorial Prof. Adriana Kovashka University of Pittsburgh September 3, 2015

2. Plan for Today Course basics refresher Matlab tutorial Overview of Homework 1 (Start on image filtering)

3. Course Info Course website: http://people.cs.pitt.edu/~kovashka/cs1699 Instructor: Adriana Kovashka (kovashka@cs.pitt.edu) – Please use "CS1699" at the beginning of your Subject Office: Sennott Square 5325 Office hours: Tuesday and Thursday, 4pm-5pm Grader: Nils Murrugarra-Llerena (nineil@cs.pitt.edu)

4. Textbooks Computer Vision: Algorithms and Applications by Richard Szeliski Computer Vision: Algorithms and Applications Visual Object Recognition by Kristen Grauman and Bastian Leibe Visual Object Recognition Please do the assigned readings before class

5. Homework Submission We will use CourseWeb Navigate to the CourseWeb page for CS1699, click on "Assignments" and the corresponding HW # Attach a zip file with your written responses and code Name the file as YourFirstName_YourLastName.zip or YourFirstName_YourLastName.tar Homework is due at 11:59pm on the due Grades will appear on CourseWeb

6. Late Policy You get 3 "free" late days, i.e., you can submit homework a total of 3 days late. For example, you can submit one problem set 12 hours late, and another 60 hours late. Once you've used up your free late days, you will incur a penalty of 25% from the total project credit possible for each late day. A late day is anything from 1 minute to 24 hours.

7. Collaboration Policy You will work individually. The work you turn in must be your own work. You can discuss the problem sets with your classmates, but do not look at their code. You cannot use posted solutions, search for code on the internet or use Matlab's implementations of something you are asked to write. When in doubt, ask the instructor! Plagiarism will cause you to fail the class and receive disciplinary penalty.

8. Homework 1 Part I: Matlab exercise – Today’s lecture Part II: Short answers – Lectures on 9/08 and 9/15 Part III: Implementing image pyramids – Lecture on 9/10 Part IV: Implementing seam carving – Today’s lecture

9. Matlab tutorial

10. Image Formation Slide credit: Derek Hoiem

11. Digital camera A digital camera replaces film with a sensor array Each cell in the array is light-sensitive diode that converts photons to electrons http://electronics.howstuffworks.com/digital-camera.htm Slide by Steve Seitz

12. Slide credit: Derek Hoiem Digital images

13. Sample the 2D space on a regular grid Quantize each sample (round to nearest integer) Image thus represented as a matrix of integer values. Adapted from S. Seitz 2D 1D

14. Digital color images Slide credit: Kristen Grauman

15. RG B Color images, RGB color space Digital color images Slide credit: Kristen Grauman

16. Images in Matlab Images represented as a matrix Suppose we have a NxM RGB image called “im” –im(1,1,1) = top-left pixel value in R-channel –im(y, x, b) = y pixels down, x pixels to right in the b th channel –im(N, M, 3) = bottom-right pixel in B-channel imread(filename) returns a uint8 image (values 0 to 255) –Convert to double format (values 0 to 1) with im2double 0.920.930.940.970.620.370.850.970.930.920.99 0.950.890.820.890.560.310.750.920.810.950.91 0.890.720.510.550.510.420.570.410.490.910.92 0.960.950.880.940.560.460.910.870.900.970.95 0.710.81 0.870.570.370.800.880.890.790.85 0.490.620.600.580.500.600.580.500.610.450.33 0.860.840.740.580.510.390.730.920.910.490.74 0.960.670.540.850.480.370.880.900.940.820.93 0.690.490.560.660.430.420.770.730.710.900.99 0.790.730.900.670.330.610.690.790.730.930.97 0.910.940.890.490.410.78 0.770.890.990.93 0.920.930.940.970.620.370.850.970.930.920.99 0.950.890.820.890.560.310.750.920.810.950.91 0.890.720.510.550.510.420.570.410.490.910.92 0.960.950.880.940.560.460.910.870.900.970.95 0.710.81 0.870.570.370.800.880.890.790.85 0.490.620.600.580.500.600.580.500.610.450.33 0.860.840.740.580.510.390.730.920.910.490.74 0.960.670.540.850.480.370.880.900.940.820.93 0.690.490.560.660.430.420.770.730.710.900.99 0.790.730.900.670.330.610.690.790.730.930.97 0.910.940.890.490.410.78 0.770.890.990.93 0.920.930.940.970.620.370.850.970.930.920.99 0.950.890.820.890.560.310.750.920.810.950.91 0.890.720.510.550.510.420.570.410.490.910.92 0.960.950.880.940.560.460.910.870.900.970.95 0.710.81 0.870.570.370.800.880.890.790.85 0.490.620.600.580.500.600.580.500.610.450.33 0.860.840.740.580.510.390.730.920.910.490.74 0.960.670.540.850.480.370.880.900.940.820.93 0.690.490.560.660.430.420.770.730.710.900.99 0.790.730.900.670.330.610.690.790.730.930.97 0.910.940.890.490.410.78 0.770.890.990.93 R G B row column Slide credit: Derek Hoiem

17. Tutorial Part I http://cs.brown.edu/courses/cs143/2011/doc s/matlab-tutorial/ http://cs.brown.edu/courses/cs143/2011/doc s/matlab-tutorial/

18. Tutorial Part II http://www.cs.pitt.edu/~kovashka/cs1699/tut orial.m http://www.cs.pitt.edu/~kovashka/cs1699/tut orial.m

19. Homework 1 Seam carving

20. Seam carving: main idea [Shai & Avidan, SIGGRAPH 2007]

21. Content-aware resizing Traditional resizing Seam carving: main idea [Shai & Avidan, SIGGRAPH 2007]

22. Content-aware resizing Seam carving: main idea Intuition: Preserve the most “interesting” content  Prefer to remove pixels with low gradient energy To reduce or increase size in one dimension, remove irregularly shaped “seams”  Optimal solution via dynamic programming. Kristen Grauman, UT-Austin

23. Want to remove seams where they won’t be very noticeable: –Measure “energy” as gradient magnitude Choose seam based on minimum total energy path across image, subject to 8-connectedness. Seam carving: main idea Kristen Grauman, UT-Austin

24. Let a vertical seam s consist of h positions that form an 8-connected path. Let the cost of a seam be: Optimal seam minimizes this cost: Compute it efficiently with dynamic programming. Seam carving: algorithm s1s1 s2s2 s3s3 s4s4 s5s5 Kristen Grauman, UT-Austin

25. How to identify the minimum cost seam? First, consider a greedy approach: Energy matrix (gradient magnitude) Kristen Grauman, UT-Austin

26. row i-1 Seam carving: algorithm Compute the cumulative minimum energy for all possible connected seams at each entry (i,j): Then, min value in last row of M indicates end of the minimal connected vertical seam. Backtrack up from there, selecting min of 3 above in M. j-1 j row i M matrix: cumulative min energy (for vertical seams) Energy matrix (gradient magnitude) j j+1 Kristen Grauman, UT-Austin

27. Example Energy matrix (gradient magnitude) M matrix (for vertical seams) Kristen Grauman, UT-Austin

28. Example Energy matrix (gradient magnitude) M matrix (for vertical seams) Kristen Grauman, UT-Austin

29. Real image example Original Image Energy Map Blue = low energy Red = high energy Kristen Grauman, UT-Austin

30. Real image example Kristen Grauman, UT-Austin

31. Real image example Kristen Grauman, UT-Austin

32. Other notes on seam carving Analogous procedure for horizontal seams Can also insert seams to increase size of image in either dimension –Duplicate optimal seam, averaged with neighbors Other energy functions may be plugged in –E.g., color-based, interactive,… Can use combination of vertical and horizontal seams Kristen Grauman, UT-Austin

33. Results from Eunho Yang Example results from classes at UT Austin

34. Results from Suyog Jain

35. Seam carving result Original image Results from Martin Becker Conventional resize

36. Seam carving result Conventional resize Original image Results from Martin Becker

37. Results from Jay Hennig Original image (599 by 799) Conventional resize (399 by 599) Seam carving (399 by 599)

38. “Failure cases” with seam carving By Donghyuk Shin

39. “Failure cases” with seam carving By Suyog Jain

40. Next Time Image filtering Reading for Tuesday: Szeliski Sec. 3.1.1, 3.2 Reading for today was: Szeliski Sec. 1.1-1.2