1. BMI2 SS08 – Class 3 “Image Processing 1” Slide 1 Biomedical Imaging 2 Class 3,4 – Regularization; Time Series Analysis (Pt. 1); Image Post-processing (Pt. 1) 02/05/08

2. BMI2 SS08 – Class 3 “Image Processing 1” Slide 2 Well-Posedness, Ill-Posedness Definition due to Hadamard, 1915: Given the mapping A: X→Y, the equation Ax = y is well- posed if –(Existence) For every y in Y, there is an x in X such that Ax = y. –(Uniqueness) If Ax 1 = Ax 2, then x 1 = x 2. –(Stability) A -1 is continuous x = A -1 y A -1 (y + dy) = x + dx Ax = y is ill-posed if it is not well-posed

3. BMI2 SS08 – Class 3 “Image Processing 1” Slide 3 Well- and Ill-conditioned Problems Overdetermined linear systems (more equations than unknowns) are ill-posed, strictly speaking –No exact Solution exists! –Existence is imposed by using least-squares solution Underdetermined linear systems (fewer equations than unknowns) are ill-posed, strictly speaking –Infinitely many Solutions exist! –Uniqueness is imposed by using minimum-norm solution Can a discrete linear system be unstable?

4. BMI2 SS08 – Class 3 “Image Processing 1” Slide 4 Well- and Ill-conditioned Problems Can a discrete linear system be unstable? –Strictly speaking, no! A -1 (x 0 +  x) = A -1 x 0 + A -1 ∙  x All elements of A -1 and of  x are finite Therefore, all elements of A -1 ∙  x must be finite However, it certainly can be true that ||A -1 ∙  x||/||A -1 x 0 || >> ||  x||/||x 0 || –Small change in input → large change in output –Such a system is called ill-conditioned

5. BMI2 SS08 – Class 3 “Image Processing 1” Slide 5 Example of Ill-conditioning 4×4 Hilbert matrix: This is a full-rank, non-singular matrix, and so it has a well-defined inverse:

6. BMI2 SS08 – Class 3 “Image Processing 1” Slide 6 Example of Ill-conditioning Positive and negative products cancel in exactly the right manner: But what happens if we change any element by even a small amount?

7. BMI2 SS08 – Class 3 “Image Processing 1” Slide 7 Example of Ill-conditioning

8. BMI2 SS08 – Class 3 “Image Processing 1” Slide 8 Example of Ill-conditioning

9. BMI2 SS08 – Class 3 “Image Processing 1” Slide 9 Example of Ill-conditioning

10. BMI2 SS08 – Class 3 “Image Processing 1” Slide 10 Example of Ill-conditioning Our “image reconstruction” operator is unbiased But it has high variance

11. BMI2 SS08 – Class 3 “Image Processing 1” Slide 11 Tradeoff Between Bias and Variance The numerical values in y are eventually represented as gray levels or colors in an image: -4 -180 60 140 As long as the color pattern makes an interpretable image, do you care if the numerical values are exactly right? That is, are you willing to give up accuracy to gain precision (i.e., decrease variance by increasing bias)?

12. BMI2 SS08 – Class 3 “Image Processing 1” Slide 12 Regularization For overdetermined systems, we define the pseudo- inverse A + = (A T A) -1 A T. For underdetermined systems, we define the pseudo- inverse A + = A T (AA T ) -1. For the Hilbert matrix, both of the preceding reduce to the true inverse: –(A T A) -1 A T = [A -1 (A T ) -1 ]A T = A -1 [(A T ) -1 A T ] = A -1 –A T (AA T ) -1 = A T [(A T ) -1 A -1 ] = [A T (A T ) -1 ]A -1 = A -1 Now we introduce one additional term: –A + = (A T A + αI) -1 A T –A + = A T (AA T + αI) -1 Regularization term Regularization parameter

13. BMI2 SS08 – Class 3 “Image Processing 1” Slide 13 What Does Regularization Do? Now we introduce one additional term: –A + = (A T A + αI) -1 A T, A + = A T (AA T + αI) -1 This particular variety is called Tikhonov regularization –Imposes continuity on the computed y That is, limits the spatial scale on which solution can change (long-pass filter) –Could replace the I in the regularization term with a discrete 1 st, 2 nd, etc., derivative operator Then continuity would be imposed on the corresponding derivative of the solution

14. BMI2 SS08 – Class 3 “Image Processing 1” Slide 14 Something To Watch Out for Two things that can go wrong when Tik. Reg. is used: –α is too small (under-regularized case): noise continues to wreak havoc –α is too large (over-regularized case): ability to capture spatial variations of interest is lost Is there an algorithm guaranteed to produce the optimal α? –Alas, no (when is life ever that easy?) –Special cases; Monte Carlo simulations; trial-and- error

15. BMI2 SS08 – Class 3 “Image Processing 1” Slide 15 Regularized Hilbert Matrix Inverse

16. BMI2 SS08 – Class 3 “Image Processing 1” Slide 16 Impact of Noisy Data Unregularized solution (i.e., α = 0): Conclusion: Under-regularized!

17. BMI2 SS08 – Class 3 “Image Processing 1” Slide 17 Impact of Noisy Data Unregularized solution (i.e., α = 0): Conclusion: Over-regularized!

18. BMI2 SS08 – Class 3 “Image Processing 1” Slide 18 Impact of Noisy Data Unregularized solution (i.e., α = 0): Conclusion: Getting close?

19. BMI2 SS08 – Class 3 “Image Processing 1” Slide 19 Final Choice of α Parameter

20. BMI2 SS08 – Class 3 “Image Processing 1” Slide 20 Other Types of Regularization Truncated singular value decomposition Discrete-cosine transform Statistical (Bayesian) –Requires knowledge of the solution and noise covariances Iterative –Steepest descent –Conjugate-gradient descent –Richardson-Lucy –Landweber

21. BMI2 SS08 – Class 3 “Image Processing 1” Slide 21 Time Series Analysis… Definitions The branch of quantitative forecasting in which data for one variable are examined for patterns of trend, seasonality, and cycle. nces.ed.gov/programs/projections/appendix_D.asp nces.ed.gov/programs/projections/appendix_D.asp Analysis of any variable classified by time, in which the values of the variable are functions of the time periods. www.indiainfoline.com/bisc/matt.htmlwww.indiainfoline.com/bisc/matt.html An analysis conducted on people observed over multiple time periods. www.rwjf.org/reports/npreports/hcrig.html www.rwjf.org/reports/npreports/hcrig.html A type of forecast in which data relating to past demand are used to predict future demand. highered.mcgraw- hill.com/sites/0072506369/student_view0/chapter12/glossary.htmlhighered.mcgraw- hill.com/sites/0072506369/student_view0/chapter12/glossary.html In statistics and signal processing, a time series is a sequence of data points, measured typically at successive times, spaced apart at uniform time intervals. Time series analysis comprises methods that attempt to understand such time series, often either to understand the underlying theory of the data points (where did they come from? what generated them?), or to make forecasts (predictions). en.wikipedia.org/wiki/Time_series_analysisen.wikipedia.org/wiki/Time_series_analysis

22. BMI2 SS08 – Class 3 “Image Processing 1” Slide 22 Time Series Analysis… Varieties Frequency (spectral) analysis –Fourier transform: amplitude and phase –Power spectrum; power spectral density Auto-spectral density –Cross-spectral density –Coherence Correlation Analysis –Cross-correlation function Cross-covariance Correlation coefficient function –Autocorrelation function –Cross-spectral density Auto-spectral density

23. BMI2 SS08 – Class 3 “Image Processing 1” Slide 23 Time Series Analysis… Varieties Time-frequency analysis –Short-time Fourier transform –Wavelet analysis Descriptive Statistics –Mean / median; standard deviation / variance / range –Short-time mean, standard deviation, etc. Forecasting / Prediction –Autoregressive (AR) –Moving Average (MA) –Autoregressive moving average (ARMA) –Autoregressive integrated moving average (ARIMA) Random walk, random trend Exponential weighted moving average

24. BMI2 SS08 – Class 3 “Image Processing 1” Slide 24 Time Series Analysis… Varieties Signal separation –Data-driven [blind source separation (BSS), signal source separation (SSS)] Principal component analysis (PCA) Independent component analysis (ICA) Extended spatial decomposition, extended temporal decomposition Canonical correlation analysis (CCA) Singular-value decomposition (SVD) an essential ingredient of all –Model-based General linear model (GLM) Analysis of variance (ANOVA, ANCOVA, MANOVA, MANCOVA) –e.g., Statistical Parametric Mapping, BrainVoyager, AFNI

25. BMI2 SS08 – Class 3 “Image Processing 1” Slide 25 A “Family Secret” of Time Series Analysis… Scary-looking formulas, such as –Are useful and important to learn at some stage, but not really essential for understanding how all these methods work All the math you really need to know, for understanding, is –How to add: 3 + 5 = 8, 2 - 7 = 2 + (-7) = -5 –How to multiply: 3 × 5 = 15, 2 × (-7) = -14 Multiplication distributes over addition u × (v 1 + v 2 + v 3 + …) = u×v 1 + u×v 2 + u×v 3 + … –Pythagorean theorem: a 2 + b 2 = c 2 a b c

26. BMI2 SS08 – Class 3 “Image Processing 1” Slide 26 A “Family Secret” of Time Series Analysis… A most fundamental mathematical operation for time series analysis: The x i time series is measurement or image data. The y i time series depends on what type of analysis we’re doing: Fourier analysis: y i is a sinusoidal function Correlation analysis: y i is a second data or image time series Wavelet or short-time FT: non-zero y i values are concentrated in a small range of i, while most of the y i s are 0. GLM: y i is an ideal, or model, time series that we expect some of the x i time series to resemble

27. BMI2 SS08 – Class 3 “Image Processing 1” Slide 27 Example: Fourier Analysis

28. BMI2 SS08 – Class 3 “Image Processing 1” Slide 28 Example: Fourier Analysis

29. BMI2 SS08 – Class 3 “Image Processing 1” Slide 29 Example: Fourier Analysis

30. BMI2 SS08 – Class 3 “Image Processing 1” Slide 30 Example: Fourier Analysis

31. BMI2 SS08 – Class 3 “Image Processing 1” Slide 31 Example: Fourier Analysis = Σ × sin10πt

32. BMI2 SS08 – Class 3 “Image Processing 1” Slide 32 Example: Fourier Analysis

33. BMI2 SS08 – Class 3 “Image Processing 1” Slide 33 Example: Fourier Analysis

34. BMI2 SS08 – Class 3 “Image Processing 1” Slide 34 Example: Fourier Analysis

35. BMI2 SS08 – Class 3 “Image Processing 1” Slide 35 Example: Fourier Analysis = ΣΣ

36. BMI2 SS08 – Class 3 “Image Processing 1” Slide 36 Example: Fourier Analysis

37. BMI2 SS08 – Class 3 “Image Processing 1” Slide 37 Another “Family Secret” of Time Series Analysis… The second operation that is fundamental to myriad forms of time- series analysis is singular value decomposition (SVD) Variations of SVD underlie: –Principal component analysis (PCA) –Independent component analysis (ICA) –Canonical correlation analysis (CCA) –Extended spatial/temporal decorrelation

38. BMI2 SS08 – Class 3 “Image Processing 1” Slide 38 Significance of angle between x and b Given an arbitrary N×N matrix A and N×1 vector x: ordinarily, b = Ax is different from x in both magnitude and direction. x b However, for any A there will always be some particular directions such that b will be parallel to x (i.e., b is a simple scalar multiple of x, or Ax = λx) if x lies in one of these directions. An x that satisfies Ax = λx is an eigenvector, and λ is the corresponding eigenvalue.

39. BMI2 SS08 – Class 3 “Image Processing 1” Slide 39 Homogeneous Linear System: Ax = 0 Recall definition of eigenvectors and eigenvalues: Ax = λx, x  0. Then Ax - λx = Ax - λIx = (A - λI)x = 0. That is, the eigenvalues are those specific values of λ for which the matrix A - λI is singular, and the eigenvectors are the corresponding nullspaces.

40. BMI2 SS08 – Class 3 “Image Processing 1” Slide 40 Significance of angle between x and b

41. BMI2 SS08 – Class 3 “Image Processing 1” Slide 41 Significance of eigenvalues and eigenvectors An N×N A always has N eigenvalues. If A is symmetric, and λ 1 and λ 2 are two distinct eigenvalues, the corresponding eigenvectors x 1 and x 2 are necessarily orthogonal. If λ 1 = λ 2, we can always subtract off x 1 ’s projection onto x 2 from x 1 (Gram-Schmidt orthogonalization). If A is not symmetric, then its eigenvectors generally are not mutually orthogonal. But recall that the matrices AA T and A T A are always symmetric. The square roots of the eigenvalues of AA T or A T A are the singular values of A. The eigenvectors of AA T or A T A are the singular vectors of A. Computation of the eigenvalues and eigenvectors of AA T and A T A underlies a very useful linear algebraic technique called singular value decomposition (SVD). SVD is the method that allows us to, among other things, tackle the one case we have not yet seen an explicit example of: finding the “solution” of a linear system when A is not of full rank.

42. BMI2 SS08 – Class 3 “Image Processing 1” Slide 42 Significance of eigenvalues and eigenvectors M is symmetric, so M T M = MM T An orthogonal matrix is very easy to invert: X -1 = X T A diagonal matrix is very easy to invert: just reciprocate each diagonal element

43. BMI2 SS08 – Class 3 “Image Processing 1” Slide 43 Significance of eigenvalues and eigenvectors Eigenvalues of A: 1.5002, 0.16914, 0.0067383, 9.6702×10 -5 Eigenvalues of A -1 : 0.66657, 5.9122, 148.41, 10341 Any arbitrary vector x is equal to a sum of the eigenvectors of A -1 : x = av 1 + bv 2 + cv 3 + dv 4, for some numbers a, b, c, d. So A -1 x = aA -1 v 1 + bA -1 v 2 + cA -1 v 3 + dA -1 v 4 = 0.66657av 1 + 5.9122bv 2 + 148.41cv 3 + 10341dv 4

44. BMI2 SS08 – Class 3 “Image Processing 1” Slide 44 Significance of eigenvalues and eigenvectors

45. BMI2 SS08 – Class 3 “Image Processing 1” Slide 45 Significance of eigenvalues and eigenvectors

46. BMI2 SS08 – Class 3 “Image Processing 1” Slide 46 Regularization Redux

47. BMI2 SS08 – Class 3 “Image Processing 1” Slide 47 Regularization Redux

48. BMI2 SS08 – Class 3 “Image Processing 1” Slide 48 What happens if we try to use Gaussian elimination to solve Ax = b, but A is singular? After second round of elimination: There is no Solution! These two equations are inconsistent. Gaussian Elimination Redux But there is a pseudoinverse, A +, which we can find by using SVD:

49. BMI2 SS08 – Class 3 “Image Processing 1” Slide 49 How do we compute A + ?

50. BMI2 SS08 – Class 3 “Image Processing 1” Slide 50 As indicated, for this case AA + ≠ I and A + A ≠ I: Gaussian Elimination Redux What is the pseudoinverse “solution,” and what is its significance?

51. BMI2 SS08 – Class 3 “Image Processing 1” Slide 51 We are not surprised that b + ≠ b, because we already knew that the original system has no Solution. Gaussian Elimination Redux That is, that no linear combination of the columns of A is equal to b. However, the “solution” x + gives us that linear combination of columns of A which is closest to b, in the sense of minimizing the distance between Ax and b.

52. BMI2 SS08 – Class 3 “Image Processing 1” Slide 52 Example: Image Time-series Analysis via PCA

53. BMI2 SS08 – Class 3 “Image Processing 1” Slide 53 METHODS: Target Medium Quasiperiodic Chaotic (Hénon attractor) Stochastic Chaotic (Hénon attractor) Indicated dynamics were imposed on the inclusions’ μ a, which ranged from 0.048 cm -1 to 0.072 cm -1 over time. The remainder of the target had a constant μ a of 0.06 cm -1, and the entire target had constant μ s = 10 cm -1. Black dots denote source/detector locations. 8 cm 0.6 cm

54. BMI2 SS08 – Class 3 “Image Processing 1” Slide 54 METHODS: Dynamics Models y1(t)y1(t) y3(t)y3(t) y2(t)y2(t) y4(t)y4(t) QuasiperiodicChaos 1 Chaos 2 Uniform Stochastic

55. BMI2 SS08 – Class 3 “Image Processing 1” Slide 55 RESULTS: Statistics of Image Time Series

56. BMI2 SS08 – Class 3 “Image Processing 1” Slide 56 RESULTS: Image Time Series PCA Spatial and temporal parts of the first five principal components of noise-free image time series. Essentially 100% of all variability is captured in the first four PCs, each of which is a mixture of the model functions.

57. BMI2 SS08 – Class 3 “Image Processing 1” Slide 57 RESULTS: Image Time Series MSA MS algorithm yields essentially perfect “unmixing” of the four modeled functions, in both the spatial and temporal dimensions.

58. BMI2 SS08 – Class 3 “Image Processing 1” Slide 58 RESULTS: Detector Time Series PCA-MSA Source Detector Application of PCA and MSA directly to the detector time series yields four “unmixed” sets of detector data that capture essentially all of the variability and directly correspond to the four model functions.

59. BMI2 SS08 – Class 3 “Image Processing 1” Slide 59 RESULTS: Images Reconstructed from Detector MS components Note that most ICA algorithms would be unable to distinguish these two, as they have identical histograms

60. BMI2 SS08 – Class 3 “Image Processing 1” Slide 60 RESULTS: Image Time Series GLM Linear Model Coefficients t-statistic Maps Model Functions

61. BMI2 SS08 – Class 3 “Image Processing 1” Slide 61 Noise Study Modeled N/S ratio increases with increasing angle (distance) between source and detector, in agreement with usual experimental or clinical experience.

62. BMI2 SS08 – Class 3 “Image Processing 1” Slide 62 RESULTS: Images from Detector MS components, 5% noise The same qualitative result is obtained at the 3.2% noise level; the deep inclusions merge into a single object, while the peripheral pairs remain largely isolable. When the noise level is  10%, all four dynamic model functions are overwhelmed by it.

63. BMI2 SS08 – Class 3 “Image Processing 1” Slide 63 RESULTS: Image Time Series GLM, 3.2% Noise Linear Model Coefficients t-statistic Maps Significance Level Maps

64. BMI2 SS08 – Class 3 “Image Processing 1” Slide 64 RESULTS: Image Time Series GLM, 50% Noise Linear Model Coefficients t-statistic Maps Significance Level Maps