1. Colour an algorithmic approach Thomas Bangert thomas.bangert@eecs.qmul.ac.uk PhD Research Proposal

2. Human Visual Sensor Array

3. physical sensor response How the physical sensors respond to light … actually a measure of pigment’s ability to absorb photons

4. virtual sensor response implied sensor response based on perceptual studies

5. …derived from colour matching studies … using 3 primaries (700nm, 546nm, 436nm)

6. R=175 G=200 B=25 What is a colour matching study? Subject is asked to adjust primaries until the colour of the 2 regions appears identical?  to match one region with the other Visual field divided into 2 regions region 1 illuminated by monochromatic light region 2 illuminated by primaries R=200 G=200 B=50

7. R=0 G=25 B=25 y=175 What is being proposed? Subject is asked to adjust primaries until the colour of the 2 regions appears identical?  to match one region with the other 4 primaries rather than 3: RGB + yellow R=0 G=0 B=50 y=200

8. … and using modern LCD technology monochrome LCD with modified backlighting one region lit by single spectrum source the second region lit by 4 primaries

9. Why? (the simple answer) … to resolve the problem of negative primaries  ie. areas where colour matching with RGB fails Amount of red needed to add to monochromatic stimuli to get a match

10. … but really, because the human brain is wired with 4 sensors in mind – organized into 2 opponent channels

11. How would the brain like to see its visual sensor input? Colour information is packed into 2 ‘opponent channels’ (2 signed numbers). Driven by 4 sensors ideally, but otherwise what is available is used.

12. Why is this interesting? July 2006. “What birds see”. Scientific American. how a bird sees colour “… is difficult – impossible in fact – for humans to know”

13. Background to colour

14. sensor array of natural visual systems arrangement is random note: very few blue sensors, none in the centre

15. Sensors we build X Y

16. The naïve approach: Just measure R G B Opposite of what natural visual system do http://www.cvl.iis.u-tokyo.ac.jp/~zhao/database.html

17. Human Perceputal Response to luminance

18. Luminance Sensor Idealized note linear response in relation to wavelength -

19. What does a light stimulus look like? The sensor response is simple integration (summation across spectral range) -

20. How do we code stimuli? When spectral composition is approximately equal sensor response = luminous intensity we assume intensity is equal throughout spectrum -

21. Spatial Opponency A Peculiarity of natural visual systems: Luminance is always measured by taking the difference between two sensor values. Produces: contrast value

22. Moving from Luminance to Colour Primitive visual systems were luminance only Night-vision remains luminance only Evolutionary Path –Monochromacy –Dichromacy(most mammals – eg. the dog) –Tetrachromacy (birds, apes, some monkeys) Vital for evolution: backward compatibility

23. Electro-Magnetic Spectrum Visible Spectrum Visual system must represent light stimuli within this zone.

24. Low resolution – equal distribution is ok High resolution – not! spectral distribution is more complex simple luminous intensity fails to describe stimuli correctly -

25. Given a light stimulus within the visible range: What information do we need to describe the stimulus fully? 1. Luminous Intensity 2. Wavelength If we had a reference luminance we could calculate wavelength (by halves). -

26. modify one sensor pair – shifting spectral sensitivity  reference sensor: Roughly speaking wavelength is: λ + ( R – G ) One sensor can be used as a reference to measure intensity and the second to measure spectral position

27. the ideal light stimulus Monochromatic Light Allows wavelength to be measured relative to a reference.

28. Problem: natural stimuli are often not ideal Light stimulus might not activate reference sensor fully. Light stimulus might not be fully monochromatic.

29. Solution: A 3 rd sensor is used to measure equiluminance. Which is subtracted. Then reference sensor can be normalized This means a 3 rd piece of information: 3. Equiluminance

30. Coding colour With the assumption that a stimuli is monochromatic Any light stimulus (within the spectral range) can be represented exactly by 3 values: luminous intensity wavelength equiluminance Wavelength is coded by taking a difference (or opponent) value of 2 sensors – simplest solution.

31. a 4 sensor opponent design 2 opponent pairs only 1 of each pair can be active min sensor is equiluminance

32. Pigment Absorption Data of human cone sensors Red > Green

33. human colour representation is circular! Which is not a new idea, but not currently in fashion. 540nm 620nm 480nm

34. Dual Opponency with Circularity an ideal model using 2 sensor pairs yellow - blue The Primaries: red - green

35. Defining Colour a working hypothesis ‘Colour’ means a stimulus is represented by 3 values luminous intensity distance between 2 primaries equiluminance The primaries are fixed locations on the spectrum. The distance between primaries is measured.

36. Deliverables for 3 year Research Proposal

37. a 4-colour image standard luminous intensity value dual opponent value equiluminance value

38. method to produce 4 primary colours from 3 sensors

39. technique to create images in 4-colour format 3 sensor raw output of conventional technology may be used algorithm is specific to device used need to be able to translate sensor values to wavelength Canon 400D

40. examine possibility of translating historical image archive to 4-colour format photography film

41. 4-colour display prototype adapt existing technology if there is no direct hardware access – use pixels as sub- pixels as long as each pixel is addressible

42. a diagnostic test to determine colour primaries in humans opponency means colours can be ‘tweaked’ by opposing complement human colour perception varies in the individual individuals with variation outside normal bounds are called ‘colour blind’ ‘colour blindness’ can be ‘cured’ by ‘tweaking’ the primaries

43. a colour matching study to confirm approach Does the 4 primary approach solve the ‘negative’ primary problem? primaries R = 650nm G = 530nm B = 460nm primaries R = 700nm G = 546.1nm B = 435.8nm

44. …using conventional lcd display technology for colour matching Light Source http://www.ccs.neu.edu/home/bchafy/monitor/crtlcd.html

45. … with modified backlighting – 2 light sources light from 4 primaries mono- chromatic light

46. (1) monochromatic light source User selectable

47. (2) 4 primaries (red, green, blue, yellow) high quality white light is already often produced by 4 primaries each primary individually adjustable

48. Apparatus monochrome LCD display spectrophotometer monochromator full spectrum light source 4-primary light source

49. Further work: colour arithmetic Transparency implied objects

50. Why is understanding colour correctly important? Colours are computed, not measured! Very important that colour information is in correct form! Starts with sensor information!

51. Colour is very useful for transparency What is the colour?

52. Why do we need transparency? otherwise we might have trouble with windows

53. … and difficulties with these kinds of tasks

54. Colour is very helpful in deciphering the layers Aim: to reconstruct scenes with transparency

55. visual systems with 4 sensors Birds Reptiles Dinosaurs Therapsids (our dinosaur-like ancestor)  about 60nm between sensors  evenly spaced  frequencies narrowed

56. The Ideal Sensor Equally spaced on spectrum Overlap with linear transition colour channel 1: R - Gcolour channel 2: yellow - B No overlap of opponent pairs

57. spectrum is shifted toward more even spacing Actual Sensor Response Sensor Response calculated from CIE perceptual data 460 530 640 CRT RGB Phosphors spectrum is shifted more towards even spacing HVS Sensor + yellow  almost equal distribution

58. a yellow sensor + a few tweaks makes human vision equivalent to bird vision even spacing 60nm between primary colours response narrowed intermediary colours at half- way points requires more processing, is less accurate, but is equivalent

59. Summary Colour is based on contrast HVS has a circular model of spectrum Colour is a code for where on spectrum 2 colour channels, bi-polar  4 primary colours 2 channels  2-d colour space Simple transform to circular representation Single variable represents all colours Purpose is to allow systematic colour transforms  colour computation

60. References Poynton, C. A. (1995). “Poynton’s Color FAQ”, electronic preprint. http://www.poynton.com/notes/colour_and_gamma/ColorFAQ.html http://www.poynton.com/notes/colour_and_gamma/ColorFAQ.html Bangert, Thomas (2008). “TriangleVision: A Toy Visual System”, ICANN 2008. Goldsmith, Timothy H. (July 2006). “What birds see”. Scientific American: 69–75. Neitz, Jay; Neitz, Maureen. (August 2008). “Colour Vision: The Wonder of Hue”. Current Biology 18(16): R700-r702. Questions?

61. Samples of simple colour transforms

62. Blue- Yellow set to 0

63. Red- Green inverted

64. Blue- Yellow inverted

65. playing with colour

66. is easy

67. these are simple transforms

68. not touched by hand