1. 1 Digital Cameras Engineering Math Physics (EMP) Jennifer Rexford http://www.cs.princeton.edu/~jrex

2. 2 Image Transmission Over Wireless Networks Image capture and compression –Inner-workings of a digital camera –Manipulating & transforming a matrix of pixels –Implementing a variant of JPEG compression Wireless networks –Wireless technology –Acoustic waves and electrical signals –Radios Video over wireless networks –Video compression and quality –Transmitting video over wireless –Controlling a car over a radio link

3. 3 Traditional Photography A chemical process, little changed from 1826 Taken in France on a pewter plate … with 8-hour exposure The world's first photograph

4. 4 Image Formation Digital Camera Eye Film

5. 5 Image Formation in a Pinhole Camera Light enters a darkened chamber through pinhole opening and forms an image on the further surface

6. 6 Aperture Hole or opening where light enters –Or, the diameter of that hole or opening Pupil of the human eye –Bright light: 1.5 mm diameter –Average light: 3-4 mm diameter –Dim light: 8 mm diameter Camera –Wider aperture admits more light –Though leads to blurriness in the objects away from point of focus

7. 7 Shutter Speed Time for light to enter camera –Longer times lead to more light –… though blurs moving subjects Exposure –Total light entering the camera –Depends on aperture and shutter speed

8. 8 Digital Photography Digital photography is an electronic process Only widely available in the last ten years Digital cameras now surpass film cameras in sales Polaroid closing its film plants this year…

9. 9 Image Formation in a Digital Camera Array of sensors –Light-sensitive diodes convert photons to electrons –Buckets that collect charge in proportion to light –Each bucket corresponds to a picture element (pixel) +10V + + + + + + Photon   + +             A sensor converts one kind of energy to another

10. 10 CCD: Charge Coupled Device Common sensor array used in digital cameras –Each capacitor accumulates charge in response to light Responds to about 70% of the incident light –In contrast, photographic film captures only about 2% Also widely used in astronomy telescopes CCD sensor

11. 11 Sensor Array: Image Sampling Pixel (Picture Element): single point in a graphic image

12. 12 Sensor Array: Reading Out the Pixels Transfer the charge from one row to the next Transfer charge in the serial register one cell at a time Perform digital to analog conversion one cell at a time Store digital representation Digital-to-analog conversion

13. 13 Sensor Array: Reading Out the Pixels

14. 14 More Pixels Mean More Detail 1280 x 960 640 x 480 1600 x 1400

15. 15 The 320 x 240 hand The 2272 x 1704 hand

16. 16 Representing Color Light receptors in the human eye –Rods: sensitive in low light, mostly at periphery of eye –Cones: only at higher light levels, provide color vision –Different types of cones for red, green, and blue RGB color model –A color is some combination of red, green, and blue –Intensity value for each color  0 for no intensity  1 for high intensity –Examples  Red: 1, 0, 0  Green: 0, 1, 0  Yellow: 1, 1, 0

17. 17 Representing Image as a 3D Matrix In the lab this week… –Matlab experiments with digital images Matrix storing color intensities per pixel –Row: from top to bottom –Column: from left to right –Color: red, green, blue Examples –M(3,2,1): third row, second column, red intensity –M(4,3,2): fourth row, third column, green intensity 1 2 3 21

18. 18 Limited Granularity of Color Three intensities, one per color –Any value between 0 and 1 Storing all possible values take a lot of bits –E.g., storing 0.368491029692069439604504560106 –Can a person really differentiate from 0.36849? Limiting the number of intensity settings –Eight bits for each color –From 00000000 to 11111111 –With 2 8 = 256 values Leading to 24 bits per pixel  Red: 255, 0, 0  Green: 0, 255, 0  Yellow: 255, 255, 0

19. 19 Number of Bits Per Pixel Number of bits per pixel –More bits can represent a wider range of colors –24 bits can capture 2 24 = 16,777,216 colors –Most humans can distinguish around 10 million colors 8 bits / pixel / color 4 bits / pixel / color

20. 20 Separate Sensors Per Color Expensive cameras –A prism to split the light into three colors –Three CCD arrays, one per RGB color

21. 21 Practical Color Sensing: Bayer Grid Place a small color filter over each sensor Each cell captures intensity of a single color More green pixels, since human eye is better at resolving green

22. 22 Practical Color Sensing: Interpolating Challenge: inferring what we can’t see –Estimating pixels we do not know Solution: estimate based on neighboring pixels –E.g., red for non-red cell averaged from red neighbors –E.g., blue for non-blue cell averaged from blue neighbors Estimate “R” and “B” at the “G” cells from neighboring values

23. 23 Interpolation Examples of interpolation Accuracy of interpolation –Good in low-contrast areas (neighbors mostly the same) –Poor with sharp edges (e.g., text) andmakes andmakes andmakes

24. 24 Are More Pixels Always Better? Generally more is better –Better resolution of the picture –Though at some point humans can’t tell the difference But, other factors matter as well –Sensor size –Lens quality –Whether Bayer grid is used Problem with too many pixels –Very small sensors catch fewer photons –Much higher signal-to-noise ratio Plus, more pixels means more storage…

25. 25 Digital Images Require a Lot of Storage Three dimensional object –Width (e.g., 640 pixels) –Height (e.g., 480 pixels) –Bits per pixel (e.g., 24-bit color) Storage is the product –Pixel width * pixel height * bits/pixel –Divided by 8 to convert from bits to bytes Example sizes –640 x 480: 1 Megabyte –800 x 600: 1.5 Megabytes –1600 x 1200: 6 Megabytes

26. 26 Compression Benefits of reducing the size –Consume less storage space and network bandwidth –Reduce the time to load, store, and transmit the image Redundancy in the image –Neighboring pixels often the same, or at least similar –E.g., the blue sky Human perception factors –Human eye is not sensitive to high frequencies

27. 27 Compression Pipeline Sender and receiver must agree –Sender/writer compresses the raw data –Receiver/reader un-compresses the compressed data Example: digital photography compress uncompress compress uncompress

28. 28 Two Kinds of Compression Lossless –Only exploits redundancy in the data –So, the data can be reconstructed exactly –Necessary for most text documents (e.g., legal documents, computer programs, and books) Lossy –Exploits both data redundancy and human perception –So, some of the information is lost forever –Acceptable for digital audio, images, and video

29. 29 Lossless: Huffman Encoding Normal encoding of text –Fixed number of bits for each character ASCII with seven bits for each character –Allows representation of 2 7 =128 characters –Use 97 for ‘a’, 98 for ‘b’, …, 122 for ‘z’ But, some characters occur more often than others –Letter ‘a’ occurs much more often than ‘x’ Idea: assign fewer bits to more-popular symbols –Encode ‘a’ as “000” –Encode ‘x’ as “11010111”

30. 30 Lossless: Huffman Encoding Challenge: generating an efficient encoding –Smaller codes for popular characters –Longer codes for unpopular characters English Text: frequency distribution Morse code

31. 31 Lossless: Run-Length Encoding Sometimes the same symbol repeats –Such as “eeeeeee” or “eeeeetnnnnnn” –That is, a run of “e” symbols or a run of “n” symbols Idea: capture the symbol only once –Count the number of times the symbol occurs –Record the symbol and the number of occurrences Examples –So, “eeeeeee” becomes “@e7” –So, “eeeeetnnnnnn” becomes “@e5t@n6” Useful for fax machines –Lots of white, separate by occasional black

32. 32 Joint Photographic Experts Group Starts with an array of pixels in RGB format –With one number per pixel for each of the three colors –And outputs a smaller file with some loss in quality Exploits both redundancy and human perception –Transforms data to identify parts that humans notice less –More about transforming the data in Wednesday’s class Uncompressed: 167 KBGood quality: 46 KBPoor quality: 9 KB

33. 33 108 KB34 KB 8 KB Joint Photographic Experts Group (JPEG) Lossy compression

34. 34 New Era of Computational Photography Beyond manual editing of photographs –E.g., to crop, lighten/darken, sharpen edges, etc. Post-processing of a single photography –De-blur photos marred by camera shake –Changing the depth of field or vantage point Combining multiple pictures –Creating three-dimensional representations –Stitching together photos for a larger view –Creating a higher-resolution picture of a single scene http://www.news.com/8301-13580_3-9882019-39.html

35. 35 Conclusion Conversion of information –Light (photons) and a optical lens –Charge (electrons) and electronic devices –Bits (0s and 1s) and a digital computer Combines many disciplines –Physics: lenses and light –Electrical engineering: charge coupled device –Computer science: manipulating digital representations –Mathematics: compression algorithms –Psychology/biology: human perception Next class: compression algorithms