Object Orie’d Data Analysis, Last Time Gene Cell Cycle Data Microarrays and HDLSS visualization DWD bias adjustment NCI 60 Data Today: Detailed (math ’ cal) look at PCA

Last Time: Checked Data Combo, using DWD Dir ’ ns

DWD Views of NCI 60 Data Interesting Question: Which clusters are really there? Issues: DWD great at finding dir’ns of separation And will do so even if no real structure Is this happening here? Or: which clusters are important? What does “important” mean?

Real Clusters in NCI 60 Data Simple Visual Approach: Randomly relabel data (Cancer Types) Recompute DWD dir’ns & visualization Get heuristic impression from this Deeper Approach Formal Hypothesis Testing (Done later)

Random Relabelling #1

Random Relabelling #2

Random Relabelling #3

Random Relabelling #4

Revisit Real Data

Revisit Real Data (Cont.) Heuristic Results: Strong Clust’s Weak Clust’s Not Clust’s Melanoma C N S NSCLC Leukemia Ovarian Breast Renal Colon

Rediscovery – Renaming of PCA Statistics: Principal Component Analysis (PCA) Social Sciences: Factor Analysis (PCA is a subset) Probability / Electrical Eng: Karhunen – Loeve expansion Applied Mathematics: Proper Orthogonal Decomposition (POD) Geo-Sciences: Empirical Orthogonal Functions (EOF)

An Interesting Historical Note The 1 st (?) application of PCA to Functional Data Analysis: Rao, C. R. (1958) Some statistical methods for comparison of growth curves, Biometrics, 14, st Paper with “Curves as Data” viewpoint

Detailed Look at PCA Three important (and interesting) viewpoints: 1. Mathematics 2. Numerics 3. Statistics 1 st : Review linear alg. and multivar. prob.

Review of Linear Algebra Vector Space: set of “vectors”,, and “scalars” (coefficients), “closed” under “linear combination” ( in space) e.g., “ dim Euclid’n space”

Review of Linear Algebra (Cont.) Subspace: subset that is again a vector space i.e. closed under linear combination e.g. lines through the origin e.g. planes through the origin e.g. subsp. “generated by” a set of vector (all linear combos of them = = containing hyperplane through origin)

Review of Linear Algebra (Cont.) Basis of subspace: set of vectors that: span, i.e. everything is a lin. com. of them are linearly indep’t, i.e. lin. Com. is unique e.g. “unit vector basis” e.g.

Review of Linear Algebra (Cont.) Basis Matrix, of subspace of Given a basis,, create matrix of columns: Then “linear combo” is a matrix multiplicat’n: where Check sizes:

Review of Linear Algebra (Cont.) Aside on matrix multiplication: (linear transformat’n) For matrices, Define the “matrix product” (“inner products” of columns with rows) (composition of linear transformations) Often useful to check sizes:

Review of Linear Algebra (Cont.) Matrix trace: For a square matrix Define Trace commutes with matrix multiplication:

Review of Linear Algebra (Cont.) Dimension of subspace (a notion of “size”): number of elements in a basis (unique) (use basis above) e.g. dim of a line is 1 e.g. dim of a plane is 2 dimension is “degrees of freedom”

Review of Linear Algebra (Cont.) Norm of a vector: in, Idea: “length” of the vector Note: strange properties for high, e.g. “length of diagonal of unit cube” = “length normalized vector”: (has length one, thus on surf. of unit sphere) get “distance” as:

Review of Linear Algebra (Cont.) Inner (dot, scalar) product: for vectors and, related to norm, via measures “angle between and ” as: key to “orthogonality”, i.e. “perpendicul’ty”: if and only if

Review of Linear Algebra (Cont.) Orthonormal basis : All ortho to each other, i.e., for All have length 1, i.e., for “Spectral Representation”: where check: Matrix notation: where i.e. is called “transform (e.g. Fourier, wavelet) of ”

Review of Linear Algebra (Cont.) Parseval identity, for in subsp. gen’d by o. n. basis : Pythagorean theorem “Decomposition of Energy” ANOVA - sums of squares Transform,, has same length as, i.e. “rotation in ”

Review of Linear Algebra (Cont.) Next time: add part about Gram- Schmidt Ortho-normalization

Review of Linear Algebra (Cont.) Projection of a vector onto a subspace : Idea: member of that is closest to (i.e. “approx’n”) Find that solves: (“least squa’s”) For inner product (Hilbert) space: exists and is unique General solution in : for basis matrix So “proj’n operator” is “matrix mult’n”: (thus projection is another linear operation) (note same operation underlies least squares)

Review of Linear Algebra (Cont.) Projection using orthonormal basis : Basis matrix is “orthonormal”: So = Recon(Coeffs of “in dir’n”) For “orthogonal complement”,, and Parseval inequality:

Review of Linear Algebra (Cont.) (Real) Unitary Matrices: with Orthonormal basis matrix (so all of above applies) Follows that (since have full rank, so exists …) Lin. trans. (mult. by ) is like “rotation” of But also includes “mirror images”

Review of Linear Algebra (Cont.) Singular Value Decomposition (SVD): For a matrix Find a diagonal matrix, with entries called singular values And unitary (rotation) matrices, (recall ) so that

Review of Linear Algebra (Cont.) Intuition behind Singular Value Decomposition: For a “linear transf’n” (via matrix multi’n) First rotate Second rescale coordinate axes (by ) Third rotate again i.e. have diagonalized the transformation

Review of Linear Algebra (Cont.) SVD Compact Representation: Useful Labeling: Singular Values in Increasing Order Note: singular values = 0 can be omitted Let = # of positive singular values Then: Where are truncations of

Review of Linear Algebra (Cont.) Eigenvalue Decomposition: For a (symmetric) square matrix Find a diagonal matrix And an orthonormal matrix (i.e. ) So that:, i.e.

Review of Linear Algebra (Cont.) Eigenvalue Decomposition (cont.): Relation to Singular Value Decomposition (looks similar?): Eigenvalue decomposition “harder” Since needs Price is eigenvalue decomp’n is generally complex Except for square and symmetric Then eigenvalue decomp. is real valued Thus is the sing’r value decomp. with:

Review of Linear Algebra (Cont.) Computation of Singular Value and Eigenvalue Decompositions: Details too complex to spend time here A “primitive” of good software packages Eigenvalues are unique Columns of are called “eigenvectors” Eigenvectors are “ -stretched” by :

Review of Linear Algebra (Cont.) Eigenvalue Decomp. solves matrix problems: Inversion: Square Root: is positive (nonn’ve, i.e. semi) definite all

Review of Multivariate Probability Given a “random vector”, A “center” of the distribution is the mean vector, A “measure of spread” is the covariance matrix:

Review of Multivar. Prob. (Cont.) Covariance matrix: Noneg’ve Definite (since all varia’s are 0) Provides “elliptical summary of distribution” Calculated via “outer product”:

Review of Multivar. Prob. (Cont.) Empirical versions: Given a random sample, Estimate the theoretical mean, with the sample mean:

Review of Multivar. Prob. (Cont.) Empirical versions (cont.) And estimate the “theoretical cov.”, with the “sample cov.”: Normalizations: gives unbiasedness gives MLE in Gaussian case

Review of Multivar. Prob. (Cont.) Outer product representation:, where:

PCA as an Optimization Problem Find “direction of greatest variability”:

PCA as Optimization (Cont.) Find “direction of greatest variability”: Given a “direction vector”, (i.e. ) Projection of in the direction : Variability in the direction :

PCA as Optimization (Cont.) Variability in the direction : i.e. (proportional to) a quadratic form in the covariance matrix Simple solution comes from the eigenvalue representation of : where is orthonormal, &

PCA as Optimization (Cont.) Variability in the direction : But = “ transform of ” = “ rotated into coordinates”, and the diagonalized quadratic form becomes

PCA as Optimization (Cont.) Now since is an orthonormal basis matrix, and So the rotation gives a distribution of the (unit) energy of over the eigen- directions And is max’d (over ), by putting all energy in the “largest direction”, i.e., where “eigenvalues are ordered”,

PCA as Optimization (Cont.) Notes: Solution is unique when Else have sol’ns in subsp. gen’d by 1st s Projecting onto subspace to, gives as next direction Continue through,…, Replace by to get theoretical PCA Estimated by the empirical version

Iterated PCA Visualization

Connect Math to Graphics 2-d Toy Example Feature Space Object Space Data Points (Curves) are columns of data matrix, X

Connect Math to Graphics (Cont.) 2-d Toy Example Feature Space Object Space Sample Mean, X

Connect Math to Graphics (Cont.) 2-d Toy Example Feature Space Object Space Residuals from Mean = Data - Mean

Connect Math to Graphics (Cont.) 2-d Toy Example Feature Space Object Space Recentered Data = Mean Residuals, shifted to 0 = (rescaling of) X

Connect Math to Graphics (Cont.) 2-d Toy Example Feature Space Object Space PC1 Direction = η = Eigenvector (w/ biggest λ)

Connect Math to Graphics (Cont.) 2-d Toy Example Feature Space Object Space Centered Data PC1 Projection Residual

Connect Math to Graphics (Cont.) 2-d Toy Example Feature Space Object Space PC2 Direction = η = Eigenvector (w/ 2 nd biggest λ)

Connect Math to Graphics (Cont.) 2-d Toy Example Feature Space Object Space Centered Data PC2 Projection Residual

Connect Math to Graphics (Cont.) Note for this 2-d Example: PC1 Residuals = PC2 Projections PC2 Residuals = PC1 Projections (i.e. colors common across these pics)