1. Colour an algorithmic approach Thomas Bangert tb300@eecs.qmul.ac.uk MSc in Computer Sciency by Research. Project Viva

2. understanding how visual system process information Visual system: about 30% of cortex most studied part of brain best understood part of brain

3. Image sensors  Binary sensor array  Luminance sensor array  Multi-Spectral sensor array

4. Where do we start? We first need a model of what light information means. Any visual system starts with a sensor: What kind of information do these sensors produce? Let’s first look at sensors we have designed!

5. Sensors we build X Y

6. The Pixel Sensors element may be:  Binary  Luminance  RGB The fundamental unit of information!

7. The Bitmap 2-d space represented by integer array 0 12 0 1

8. What information is produced? 2-d array of pixels:  Black & White Pixel: –single luminance value, usually 8 bit  Colour Pixel –3 colour values, usually 8-bit

9. Where we need to start: the fundamentals of the sensor ?

10. Human Visual System (HVS) The fundamentals!

11. The Sensor 2 systems: day-sensor & night-sensor To simplify: we ignore night sensor system Cone Sensors very similar to RGB sensors we design for cameras

12. BUT: sensor array is not ordered arrangement is random note: very few blue sensors, none in the centre

13. sensor pre-processing circuitry

14. First Question: What information is sent from sensor array to visual system? Very clear division between sensor & pre-processing (Front of Brain) and visual system (Back of Brain) connected with very limited communication link

15. Receptive Fields All sensors in the retina are organized into receptive fields Two types of receptive field. Why?

16. What does a receptive field look like? In the central fovea it is simply a pair of sensors. Always 2 types: plus-centre minus-centre

17. What do retinal receptive fields do? Produce an opponent value: simply the difference between 2 sensors This means: it is a relative measure, not an absolute measure and no difference = no information to brain

18. Sensor Input Luminance Levels it is usual to code 256 levels of luminance Linear: Y Logarithmic: Y’

19. Receptive Field Function - - - - - - - - - + + + + + + + + + - - - - - - - - - + + + + + + + + + - - - - - - - - - + + + + + + + + + Min Zone Max-Min Function Output is difference between average of center and max/min of surround Max Zone Tip of Triangle

20. Dual Response to gradients Why? Often described as second derivative/zero crossing

21. Abstracted Neurons only produce positive values. Dual +/- produces positive & negative values. Together: called a channel Produces signed values. Co-ordinate

22. Human Sensor Response to monochromatic light stimuli

23. HVS Luminance Sensor Idealized A linear response in relation to wavelength. Under ideal conditions can be used to measure wavelength.

24. Spatially Opponent HVS: Luminance is always measured by taking the difference between two sensor values. Produces: contrast value Which is done twice, to get a signed contrast value

25. Moving from Luminance to Colour Primitive visual systems were in b&w Night-vision remains b&w Evolutionary Path –Monochromacy –Dichromacy(most mammals – eg. the dog) –Trichromacy (birds, apes, some monkeys) Vital for evolution: backwards compatibility

26. Electro-Magnetic Spectrum Visible Spectrum Visual system must represent light stimuli within this zone.

27. Colour Vision Young-Helmholtz Theory Argument: Sensors are RGB therefore Brain is RGB  3 colour model

28. Hering colour opponency model Fact: we never see reddish green or yellowish blue. Therefore: colours must be arranged in opponent pairs: Red  Green Blue  Yellow  4 colour model

29. HVS Colour Sensors response to monochromatic light

30. How to calculate spectral frequency with 2 luminance sensors. Roughly speaking:

31. the ideal light stimulus Monochromatic Light Allows frequency to be measured in relation to reference.

32. Problem: natural light is not ideal Light stimulus might not activate reference sensor fully. Light stimulus might not be fully monochromatic. ie. there might be white mixed in

33. Solution: A 3 rd sensor is used to measure equiluminance. Which is subtracted. Then reference sensor can be normalized

34. Equiluminance & Normalization Also called Saturation and Lightness. Must be removed first – before opponent values calculated. Then opponent value = spectral frequency Values must be preserved – otherwise information is lost.

35. a 4 sensor design 2 opponent pairs only 1 of each pair can be active min sensor is equiluminance

36. What does a colour opponent channel look like? luminance contrast opponent channel each colour opponent channel codes for 2 primary colours Total of 4 primary colours

37. What is Colour? Colour is calculated exactly the same as luminance contrast. The only difference is spectral range of sensors is modified. Colour channels are: R  G B  Y Uncorrected colour values are contrast values. But with white subtracted and normalized: Colour is Wavelength!

38. How many sensors? 4 primary colours require 4 sensors!

39. Human Retina only has 3 sensors! What to do? Because of opponency when R=G, R  G colour channel is 0. Why not pair RG and reuse it as a Yellow sensor! Yellow can be R=G

40. How do we abstract information from sensor array? Luma (Y’) Red-Green (C B ) Blue-Yellow (C R )

41. Luminance + 2 colour values + 2 sensor correction values Chroma Blue Chroma Red + Lightness + Saturation

42. Tri-Phosphor Lighting optimised for perception of ‘white’

43. Primary Colours matched to spectrum

44. Testing Colour Opponent model What we should see What we do see Unfortunately it does not match There is Red in our Blue

45. The strange case of Ultra-Violet Light with frequency of 400nm is ultra-blue Red sensor is at opposite of spectrum & not stimulated. Yet we see ultra-violet – which is Blue + Red …and the more we go into UV the more red

46. Colour Matching Data (CIE 1931) (indirect sensor response) a very odd fact – a virtual sensor response

47. Pigment Absorption Data of human cone sensors Red > Green

48. Therefore: HVS colour representation must be circular! Which is not a new idea, but not currently in fashion. 540nm 620nm 480nm

49. Dual Opponency with Circularity an ideal model using 2 sensor pairs

50. Colour Wheel Goethe & Munsell Colours are represented by a single value: Hue

51. RYB Colour Circle no longer used

52. HSL (Hue + S & L) Circular colour coding Any colour represented by 1 number Allows colour arithmetic R=255 G=0 B=0 R=255 G=255 B=0 R=0 G=255 B=0 R=0 G=255 B=255 R=0 G=0 B=255 R=255 G=0 B=255

53. HSL & HSV Simple & Elegant But it is flawed: –simple transformation of RGB –colours do not match perception Why? Because there are 4 primary colours, not 3!

54. gives us a 2-d colour space CBCB CRCR Colour Information: 2 independent values

55. 2-d space: Cartesian coordinates or polar coordinates Co-ordinate systems

56. … requires 2 independent channels  which give 4 primary colours Yellow added as a primary! Which allows a simple transform to circular representation

57. Opponent Values  Hue A simple transform from 2 opponent values to a single hue value How might HVS do this? we keep 2 colour channels but link them

58. Travelling the Colour Wheel (Hue) One Chroma channel is always at max or min The other Chroma channel is incremented or decremented Rules: if (C B ==Max)C R -- if (C R ==Max)C B ++ if (C R ==Min)C B -- if (C B ==Min)C R ++ +-

59. Colour Wheel Simple rule based system that cycles through the colour wheel Allows arithmetic operations on colour

60. What is Hue? Circular representation of spectrum Its purpose is to provide a Spectrum Value Primary Colours are the extreme ends of the 2 linked colour channels

61. Hue: 2 values or 1 2 linked values allow us to turn colour off. (0,0) is not an allowed hue, used for no colour Simple standard input pixel: –luminance value or –colour value

62. Why do we need arithmetic on colour? Colours are computed, not measured!

63. Colour is very useful for transparency What is the colour?

64. Why do we need transparency? otherwise we might have trouble with windows

65. … and difficulties with these kinds of tasks

66. Colour is very helpful in deciphering the layers Aim: to reconstruct scenes with transparency

67. It all must start with the right kind of sensor: Format of ‘pixel’ as it enters visual area of brain for processing: Luminance Information Optional Colour Information Where on spectrum How colourful

68. visual systems with 4 sensors Birds Reptiles Dinosaurs Therapsids (our dinosaur-like ancestor)  about 60nm between sensors  evenly spaced  frequencies narrowed

69. The Ideal Sensor Equally spaced on spectrum Overlap with linear transition colour channel 1: R - Gcolour channel 2: yellow - B No overlap of opponent pairs

70. spectrum is shifted toward more even spacing Actual Sensor Response Sensor Response calculated from CIE perceptual data 460 530 640 CRT RGB Phosphors spectrum is shifted more towards even spacing HVS Sensor + yellow  almost equal distribution

71. a yellow sensor + a few tweaks makes human vision equivalent to bird vision even spacing 60nm between primary colours response narrowed intermediary colours at half- way points requires more processing, is less accurate, but is equivalent

72. How do we get a yellow sensor? we re-use red & green sensors & but only when they are equal (R==G) This implies dividing by a measure of equality

73. Existing Circular Colour Systems: Munsell colour wheel with 5 primary colours 100 years old quite close

74. Existing Circular Colour Systems: CIE L*a*b* & CIE L*C*h L*a*b* is a colour opponent space L*C*h is the transform to circular 4 primary colours Red = 0° Yellow = 90 ° Green = 180 ° Blue = 270 °

75. Summary Colour is based on contrast HVS has a circular model of spectrum Colour is a code for where on spectrum 2 colour channels, bi-polar  4 primary colours 2 channels  2-d colour space Simple transform to circular representation Single variable represents all colours Purpose is to allow systematic colour transforms  colour computation

76. References Poynton, C. A. (1995). “Poynton’s Color FAQ”, electronic preprint. http://www.poynton.com/notes/colour_and_gamma/ColorFAQ.html http://www.poynton.com/notes/colour_and_gamma/ColorFAQ.html Bangert, Thomas (2008). “TriangleVision: A Toy Visual System”, ICANN 2008. Goldsmith, Timothy H. (July 2006). “What birds see”. Scientific American: 69–75. Neitz, Jay; Neitz, Maureen. (August 2008). “Colour Vision: The Wonder of Hue”. Current Biology 18(16): R700-r702. Questions?

77. The problem with Yellow Colour: an algorithmic approach Thomas Bangert tb300@eecs.qmul.ac.uk Thomas Bangert MSc in Computer Sciency by Research. Project Viva

78. Colour channels are pure Opponency means colour pairs are pure with respect to themselves. It follows that a pure colour is achieved only when the other opponent channel is 0. Reddest red only when B-RG is 0 Bluest blue only when R-G is 0 and inversely

79. RGB is pure Red is reddest when G & B = 0 etc. XYZ and LMS are not pure. Sensors of visual system have a broad spectral response. They do not have a pure colour response. Retinal processing produces pure colour channels from noisy and ambiguous data. RGB Red: R=255, G=0, B=0 Green: R=0, G=255, B=0 Blue: R=0, G=0, B=255

80. YUV & YC B C R Transforms JPEG 2000 allows reversible simplification Transform usually expressed in matrix form JPEG without anything odd like ‘headroom’ note: no negative numbers for JPEG, C+=128

81. Lets try some JPEG numbers: not trivial ‘leakage’ Should be 127 Cyan

82. The Problem: Colours channels are not pure. They should be! R 255 0 0 G 0 255 0 B 0 0 255 Magenta 111 127 127 Cyan 195 127 -128 RG 255 255 0 Cyan 0 255 255

83. YUV/YC R C B simplified A large number of transforms exist, most variations of YUV. Minor tweaks of transform from XYZ can lead to quite large differences. All of which work fine perceptually (meaning neurons are not that precise). Why not simplify?

84. Chroma Blue If there was a yellow sensor We use R=G instead: which is (R+G)/2 but we want a value only when R=G

85. Yellow: the Chroma Blue correction factor The less equal R and G are, the less yellow there should be. So: Simply divide R by G to determine how close they are. The more equal they are the more active the ‘yellow’ sensor is.

86. Transform back to RGB Fully Reversible Calculate R and G first, then Blue correction factor.

87. Samples of simple colour transforms

88. Blue- Yellow set to 0

89. Red- Green inverted

90. Blue- Yellow inverted

91. playing with colour

92. is easy

93. these are simple transforms

94. not touched by hand

95. YUV Summary Two simple tweaks allow us to correct conversion between RGB and YUV/YC R C B. Also allows conversion to be simplified.