1. ESE 250 – S'12 Kod & DeHon 1 ESE250: Digital Audio Basics Week 4 February 2, 2012 Time-Frequency

2. 2 Course Map Numbers correspond to course weeks 2,5 6 11 13 12 Today ESE 250 – S'12 Kod & DeHon

3. 3 Teaser: Musical Representation With this compact notation  Could communicate a sound to pianist  Much more compact than 44KHz time-sample amplitudes (fewer bits to represent)  Represent frequencies

4. ESE 250 – S'12 Kod & DeHon 4 Week 4: Time-Frequency There are other ways to represent  Frequency representation particularly efficient http://en.wikipedia.org/wiki/File:Lead_Sheet.png In this lecture we will learn that the frequency domain entails representing time-sampled signals using a conveniently rotated coordinate system

5. ESE 250 – S'12 Kod & DeHon 5 Prelude: Harmonic Analysis Fourier Transform ( FT )  Fourier (& other 19 th Century Mathematicians)  discovered that (real) signals  can always (if they are smooth enough)  be expressed as the sum of harmonics Defn: “Harmonics” (Fourier Series)  collections of periodic signals (e.g., cos, sin)  whose frequencies are related by integer multiples  arranged in order of increasing frequency  summed in a linear combination  whose coefficients provide  an alternative representation the job of this lecture is to replace this signals- analysis perspective with a symbols- synthesis perspective

6. ESE 250 – S'12 Kod & DeHon 6 A Sampled (Real) Signal Sample Data:Sampled Signal:

7. ESE 250 – S'12 Kod & DeHon 7 Reconstructing the Sampled Signal Exact Reconstruction  May be possible  Under the right assumptions  Given the right model This example  A “harmonic” signal  Sampled in time  Can be reconstructed o exactly o from the time-sampled values o given knowledge of the harmonics: Cos[1t]/ p (5/2) p (5/2) ¢ Sin[2t]/ p (5/2) + = { Cos[0t], Sin[1t], Cos[1t], Sin[2t], Cos[2t], Sin[3t], Cos[3t] } p (5/2) ¢

8. ESE 250 – S'12 Kod & DeHon 8 Reconstructing the Sampled Signal Exact Reconstruction  May be possible  Under the right assumptions  Given the right model This example  A “harmonic” signal  Sampled in time  Can be reconstructed o exactly o from the time-sampled values o given knowledge of the harmonics: p (5/2) ¢ Cos[1t]/ p (5/2) p (5/2) ¢ Sin[2t]/ p (5/2) + = { Cos[0t], Sin[1t], Cos[1t], Sin[2t], Cos[2t], Sin[3t], Cos[3t] }

9. ESE 250 – S'12 Kod & DeHon 9 Sequence of Analysis Given  Fundamental frequency: f = 1/2   Sampling Rate: n s = 5  Measured Data: Compute  “basis” functions  coefficients Reconstruct  exact function  from linear combination of o “basis elements” (known) o coefficients (computed) {r (-4  /5), r (-2  /5), r (02  /5), r (2  /5), r (4  /5) } h 0 (t) = Cos[0t] / p 5 h 1s (t) = Sin[1t] / p (5/2) h 1c (t) = Cos[1t]/ p (5/2) h 2s (t) = Sin[2t] / p (5/2) h 2c (t) = Cos[2t]/ p (5/2) 0 p (5/2) 0 r(t) = Cos[t] + Sin[2t] = 0 ¢ h 0 (t) + 0 ¢ h 1s (t) + p (5/2) ¢ h 1c (t) + p (5/2) ¢ h 2s (t) + 0 ¢ h 2c (t)

10. ESE 250 – S'12 Kod & DeHon 10 Fourier Analysis Time-ValuesFrequency-Amplitudes FT SampledSampled Q u a n ti z e d DFT (“closed form”) (computation)

11. ESE 250 – S'12 Kod & DeHon 11 Reconstruction vs Approximation Previous Example  received function was “in the span” of the harmonics  reconstruction achieves exact match at all times More General Case  received function is “close” to the “span”  reconstruction achieves exact match  only at the sampled times  get successively better approximation at all times o by taking successively more samples o and using successively higher harmonics

12. ESE 250 – S'12 Kod & DeHon 12 Another Sampled (Real) Signal t v Sample Data:Sampled Signal:

13. ESE 250 – S'12 Kod & DeHon 13 Approximate Reconstruction  is always achievable  and more relevant  to our problem Example  A roughly “harmonic” signal  Sampled in time  Can be approximated o “arbitrarily” closely o from the time-sampled values o using any “good” set of harmonics Approximating the Sampled Signal { Cos[0t], Sin[1t], Cos[1t], Sin[2t], Cos[2t], Sin[3t], Cos[3t] }

14. ESE 250 – S'12 Kod & DeHon 14 Approximate Reconstruction (Successively Thinner Green Dashed Curves Denote Successively Fewer Harmonic Components) Sum up the (black) harmonics using the (green) coefficients:

15. ESE 250 – S'12 Kod & DeHon 15 More Harmonics are Better 7 Samples; 7 Harmonics 11 Samples 15 Samples; 15 Harmonics ; 11 Harmonics

16. ESE 250 – S'12 Kod & DeHon 16 Usually Computed, Not “Solved” 7 Samples; 7 Harmonics 11 Samples 15 Samples; 15 Harmonics ; 11 Harmonics DFT the “spectrum” is often plotted as a function of frequency

17. ESE 250 – S'12 Kod & DeHon 17 Yet Another Sampled (Real) Signal t v Measured Data: Sampled Signal:

18. ESE 250 – S'12 Kod & DeHon 18 Approximate Reconstruction  although always achievable  may require a lot of samples  to get good performance  from “poorly chosen” harmonics Different “bases”  match different “data”  better or worse (sometimes time is better than frequency) Some Signals Dislike Some Harmonics 15 Samples & Harmonics 21 Samples & Harmonics 31 Samples & Harmonics

19. ESE 250 – S'12 Kod & DeHon 19 Choice of Basis What is a “harmonic”?  we could have used periodic “pulse trains” o previous signal would be reconstructed exactly o with one or two pulse-train harmonics  but “sound-like” signals o would typically require a very large number o of “pulse-train” harmonics Fourier Theory (and generalizations)  permits very broad choice of harmonics  such choices amount to the selection of a model Today’s Lecture  interprets the choice of harmonics o as a selection of coordinate reference frame o in the space of received (sampled,quantized) data  lends (geometric) insight to high-dimensional phenomena  introduces arsenal of linear algebraic computation  encourages “learning” data-driven models

20. ESE 250 – S'12 Kod & DeHon 20 Intuitive Concept Inventory 11 Samples; Q = FT(q) 11 Harmonics Time Domain Frequency Domain r (received signal) (sampling) qQ

21. ESE 250 – S'12 Kod & DeHon 21 Intuitive Concept Inventory 11 Samples; Q = DFT(q) 11 Harmonics Time Domain Frequency Domain Floating Point r (received signal) Sampling & Quantization qQ this week’s idea Perceptual coding

22. ESE 250 – S'12 Kod & DeHon 22 Where Are We Heading After Today? Week 2  Received signal is o discrete-time-stamped o quantized  q = PCM[ r ] = quant L [Sample T s [r] ] Week 3  Quantized Signal is Coded  c =code[ q ] Week 4  Sampled signal o not coded directly o but rather, “ Float ” -‘ed o then linearly transformed o into frequency domain  Q = DFT[ q ] [Painter & Spanias. Proc.IEEE, 88(4):451–512, 2000] q SampleCode Store/ Transmit DecodeProduce r(t)r(t)p(t)p(t) Generic Digital Signal Processor q c c Q Psychoacoustic Audio Coder

23. ESE 250 – S'12 Kod & DeHon 23 Interlude: Audio Communications Close Encounters

24. ESE 250 – S'12 Kod & DeHon 24 Technical Concept Inventory Floating Point Quantization  a symbolic representation  admitting a mimic of continuous arithmetic Vectors  sampled signals are points  in a (high dimensional) vector space Linear Algebra  the “Swiss Army Knife” of high dimensions  provides a logical, geometric, and computational  toolset for manipulating vectors Change of Basis  DFT is a high dimensional rotation  in the vector space of time-sampled signals

25. ESE 250 – S'12 Kod & DeHon 25 Technical Concept Inventory Floating Point Quantization  a symbolic representation  admitting a mimic of continuous arithmetic Vectors  sampled signals are points  in a (high dimensional) vector space Linear Algebra  the “Swiss Army Knife” of high dimensions  provides a logical, geometric, and computational  toolset for manipulating vectors Change of Basis  DFT is a high dimensional rotation  in the vector space of time-sampled signals

26. ESE 250 – S'12 Kod & DeHon 26 r(t)r(t) q1q1 q2q2 q3q3 q4q4 q5q5 Float-Quantized Symbols Act “Real” q = PCM[ r(t) ] = Float (b,p,E) [Sample T s [r(t)] ]  eliminates continuous time dependence  discretizes continuous values o cannot represent an uncountable collection of functions o with a countable (of course, in fact, finite!) set of “symbols” Floating Point Representation and Computer Arithmetic  Choose: Base ( b ), Precision ( p ), Magnitude ( E ) o q = b e ¢ [d 0 + d 1 ¢ b -1 + … + d p-1 ¢ b -( p -1) ] o - E · e · E o 0 < d i < b  Non-uniform quantization o b p different “mantissas” o 2E different exponents o ~ Log 2 [2E] + Log 2 [b p ] bits  Associated Flop Arithmetic op 2 { +, -, *, /} [ { Sqrt, Mod, Flint} ) Flop(x,y) = Float[ op(x,y) ]  Archetypal Computation: Inner product o x = (x 1,.., x n ), y = (y 1, …, y n ) o h x,y i = x 1 ¢ y 1 + x 2 ¢ y 2 + … + x n ¢ y n Crucially important operation for signal processing applications ! [Widrow, et al., IEEE TIM’96]

27. ESE 250 – S'12 Kod & DeHon 27 Technical Concept Inventory Floating Point Quantization  a symbolic representation  admitting a mimic of continuous arithmetic Vectors  sampled signals are points  in a (high dimensional) vector space Linear Algebra  the “Swiss Army Knife” of high dimensions  provides a logical, geometric, and computational  toolset for manipulating vectors Change of Basis  DFT is a high dimensional rotation  in the vector space of time-sampled signals

28. ESE 250 – S'12 Kod & DeHon 28 Sampled received signal Is a discrete sequence of time- stamped floats q = (q 1, q 2, … q n s ) = Float ( r(T 0 +T s ), r(T 0 + 2T s ), …., r(T 0 + n s T s ) )  of “real” (i.e. Float ’ed) values  at each of the n s time-stamps Think of each of the time-stamps  as an “axis”  of “real” (float) values Time Functions are Vectors r(t)r(t) q1q1 q2q2 q3q3 q4q4 q5q5

29. ESE 250 – S'12 Kod & DeHon 29 Time Functions are Vectors Think of each of the time- stamps as an “axis” of “real” (float) values E.g., for three time stamps, n s = 3,  we can record the values  arrange each axis located perpendicular  to the other two in space  mark their values  and interpret them as a vector

30. ESE 250 – S'12 Kod & DeHon 30 Think of each of the time- stamps as an “axis” of “real” (float) values  E.g., for two time stamps, n s = 2, o we can draw both axes o on “graph paper”  … for a greater number of time stamps … o we can “imagine” arranging each axis o in a mutually perpendicular direction o in space of appropriately high dimension t = - 6.28 t = 2.5 q1q1 q2q2 q b1b1 b2b2 Time Functions are Vectors

31. ESE 250 – S'12 Kod & DeHon 31 Technical Concept Inventory Floating Point Quantization  a symbolic representation  admitting a mimic of continuous arithmetic Vectors  sampled signals are points  in a (high dimensional) vector space Linear Algebra  the “Swiss Army Knife” of high dimensions  provides a logical, geometric, and computational  toolset for manipulating vectors Change of Basis  DFT is a high dimensional rotation  in the vector space of time-sampled signals

32. ESE 250 – S'12 Kod & DeHon 32 Linear Algebra: “Swiss Army Knife” We cannot “see” in high dimensions Linear Algebra enables us in high dimensions to  reason precisely  think geometrically  compute Essential Ideas  Basis expansion  Change of basis  Ingredients o Orthonormality o Inner Product h ¢, ¢ i t = - 6.28 t = 2.5 q1q1 r(t)r(t) q1q1 q2q2 B T = { b 1, b 2 } = { (1,0), (1,0)} q2q2 q = (q 1, q 2 ) = (0.8, - 0.9) = 0.8 ¢ (1,0) – 0.9 ¢ (1,0) = 0.8 ¢ b 1 + (– 0.9) ¢ b 2 = h q,b 1 i¢ b 1 + h q,b 2 i ¢ b 2 = q 1 ¢ b 1 + q 2 ¢ b 2 q b1b1 b2b2 where h x,y i = x 1 y 1 + x 2 y 2 h q,b 1 i = 0.8 ¢ 1 + (-0.9) ¢ 0 = 0.8 h q,b 2 i = 0.8 ¢ 0 + (-0.9) ¢ 1 = - 0.9 (computational definition):

33. ESE 250 – S'12 Kod & DeHon 33 Linear Algebra: “Swiss Army Knife” Orthonormal Basis  set of unit length vectors  each “perpendicular” to all the others  total number given by dimension of the space Inner Product  (scaled) cosine of relative angle  scales unit length t = - 6.28 t = 2.5 q1q1 q2q2 q b1b1 b2b2 q 1 = h q,b 1 i = Length(q ) ¢ Cos [ Å (q,b 1 )] Å(q,b1)Å(q,b1) Å(q,b2)Å(q,b2) q 2 = h q,b 2 i = Length(q ) ¢ Cos [ Å (q,b 2 )] Generally: h r, s i = Length(r) ¢ Length(s) ¢ Cos [ Å (r,s)] ) h r, r i = Length(r) 2 geometric re-interpretation of computational definition : h x,y i = x 1 y 1 + x 2 y 2

34. ESE 250 – S'12 Kod & DeHon 34 Technical Concept Inventory Floating Point Quantization  a symbolic representation  admitting a mimic of continuous arithmetic Vectors  sampled signals are points  in a (high dimensional) vector space Linear Algebra  the “Swiss Army Knife” of high dimensions  provides a logical, geometric, and computational  toolset for manipulating vectors Change of Basis  DFT is a high dimensional rotation  in the vector space of time-sampled signals

35. ESE 250 – S'12 Kod & DeHon 35 Change of Coordinates [Google Maps] Vs. Independence Hall 500 Chestnut St.

36. ESE 250 – S'12 Kod & DeHon 36 Why Change Basis ? Efficiency  data sets often lie along  lower dimensional subspaces  Of high dimensional data space Decoupling  receiver model may “prefer”  a specific basis

37. ESE 250 – S'12 Kod & DeHon 37 Linear Algebra: Change of Basis Goal  Re-express q  In terms of B H Notation  use new symbol, Q  denoting different computational representation  even though vector is geometrically unchanged Check: “good” basis?  both unit length?  mutually perpendicular vectors? Further geometric Interpretation  if old basis is orthonormal  then new basis is also  if and only if it is o A “rotation” o Away from the old B H = { H 1, H 2 } = { ( 1/ p 2, 1/ p 2 ), (- 1/ p 2, 1/ p 2 )} Q H1H1 H2H2 Length(H 1 ) 2 = h H 1, H 1 i = 1/ p (2 ¢ 2) + 1/ p (2 ¢ 2) = ½ + ½ = 1 Length(H 2 ) 2 = h H 2, H 2 i = 1/ p (2 ¢ 2) + 1/ p (2 ¢ 2) = ½ + ½ = 1 h H 1, H 2 i = h 1 1 h 2 1 + h 1 2 h 2 2 = - 1/ p 2 ¢ 2 + 1/ p 2 ¢ 2 = 0 t = - 6.28 t = 2.5 b2b2 b1b1

38. ESE 250 – S'12 Kod & DeHon 38 Linear Algebra: Change of Basis Goal  Re-express q = ( q 1, q 2 ) o specified by coordinate representation o in terms of the old basis, B T  As Q= [Q 1, Q 2 ] o Specified by coordinate representation o In terms of rotate basis, B H Idea:  recall geometric meaning  of q = ( q 1, q 2 ) o scale b 1 by q 1 = h b 1, q i o scale b 2 by q 2 = h b 2, q i o form the resultant vector Compute Q= [Q 1, Q 2 ]  using same geometric idea  reveals how to obtain [Q 1, Q 2 ] o scale H 1 by Q 1 = h q,H 1 i o scale H 2 by Q 2 = h q,H 2 i o form the resultant vector q = ( q 1, q 2 ) = q 1 ¢ b 1 + q 2 ¢ b 2 = h q, b 1 i¢ b 1 + h q, b 2 i ¢ b 2 ) Q 1 = h q, H 1 i = h (0.8, - 0.9), ( 1/ p 2, 1/ p 2 ) i = (0.8/1.1 - 0.9 /1.1) ¼ - 0.11 Q = [Q 1, Q 2 ] = h Q,H 1 i¢ H 1 + h Q,H 2 i ¢ H 2 = h q,H 1 i¢ H 1 + h q,H 2 i ¢ H 2 ) Q 2 = h q, H 2 i = h (0.8, - 0.9), (- 1/ p 2, 1/ p 2 ) i = - (0.8/1.1 + 0.9 /1.1) ¼ - 1.6 - Q 2 t = - 6.28 t = 2.5 -Q 1 Q H1H1 b2b2 b1b1 H2H2

39. ESE 250 – S'12 Kod & DeHon 39 Generalize to n s = 3 Samples h 0 (t) = Cos[0t]/ p 3 h 1 (t) = 2 Sin[t]/ p 3 h 2 (t) = 2 Cos[t]/ p 3 H 0 = Float[ h 0 (-2  /3), h 0 (0  /3), h 0 (2  /3) ] H 1 = Float[ h 1 (-2  /3), h 1 (0  /3), h 1 (2  /3) ] H 2 = Float[ h 2 (-2  /3), h 2 (-0  /3), h 2 (2  /3) ] The 3-sample DFT : take inner products of sampled signal with each harmonic

40. ESE 250 – S'12 Kod & DeHon 40 Generalize to n s = 3 Samples h 0 (t) = Cos[0t]/ p 3 h 1 (t) = 2 Sin[t]/ p 3 h 2 (t) = 2 Cos[t]/ p 3

41. ESE 250 – S'12 Kod & DeHon 41 11 Samples; Q = DFT(q) 11 Harmonics Time Domain Frequency Domain Floating Point r (received signal) Sampling & Quantization qQ this week’s idea Perceptual coding Generalize to Arbitrary Samples

42. ESE 250 – S'12 Kod & DeHon 42 … for more understanding…. Courses  ESE 325 !  (Math 240) ) Math 312 !!! Reading  Quantization B. Widrow, I. Kollar, and M. C. Liu. Statistical theory of quantization. IEEE Transactions on Instrumentation and Measurement, 45(2):353– 361, 1996.  Floating Point D. Goldberg. What every computer scientist should know about floating-point arithmetic. ACM Computing Surveys, 23(1), 1991.  Linear Algebra for Frequency Transformations o G. Strang. The discrete cosine transform. SIAM Review, 41(1):135– 147, 1999

43. ESE 250 – S'12 Kod & DeHon 43 ESE250: Digital Audio Basics End Week 4 Lecture Time-Frequency