Estimating Intrinsic Dimension Justin Eberhardt UMD, Mathematics and Statistics Advisor: Dr. Kang James

Outline Introduction Nearest Neighborhood Estimators  Regression Estimator  Maximum Likelihood Estimator  Revised Maximum Likelihood Estimator Comparison Summary 2

Intrinsic Dimension Definition The least number of parameters required to generate a dataset Minimum number of dimensions that describes a dataset without significant loss of feature 3

Ex 1: Intrinsic Dimension Flatten (Unroll) x z y x y Int Dim = 2 4

Ex 2: Intrinsic Dimension 28 X 28 One Image: 784 Dimensional

Ex 2: Intrinsic Dimension Int Dim = 2 [Isomap Project, J. Tenenbaum & J. Langford, Stanford] 6 Top & Bottom Loop No Loop

Applications Biometrics  Facial Recognition, Fingerprints, Iris Genetics 7

Why do we need to reduce dimensionality? Low dimensional datasets are more efficient Not even supercomputers can handle very high-dimensional matrices Data in 1,2 and 3 dimensions can be visualized 8

Ex: Facial Recognition in MN 5 Million People 2 Images per Person (Front and Profile) 1028 X 1028 Pixels per Image (1 Megapixel) Total Memory Required:  n = 5,000,000  p = (2)(1028)(1028)= 2.11 Million Dimensions  Matrix Size: (5 x 10 6 )(2.11 x 10 6 ) = 10 billion cells  Memory: 2(10 x ) = 20 x = 20 Terabytes

Intrinsic Dimension Estimators Objective: To find a simple formula that uses nearest neighbor (NN) information to quickly estimate intrinsic dimension 10

Intrinsic Dimension Estimators Project Description: Through simulation, we will compare the effectiveness of three proposed NN intrinsic dimension estimators. 11

Intrinsic Dimension Estimators Note: Traditional methods for estimating Intrinsic Dimension, such as PCA, fail on non-linear manifolds. 12

Intrinsic Dimension Estimators Nearest-Neighbor Methods Regression Estimator K. Pettis, T. Bailey, A. Jain & R. Dubes, 1979 Maximum Likelihood Estimator E. Levina, & P. Bickel, 2005 D. MacKay and Z. Ghahramani,

The distance from x 2 to x 3 Distance Matrix N 1 0d 1,2 d 1,3 d 1,n 2 d 2,1 0d 2,3 d 2,n 3 d 3,1 d 3,2 0d 3,n N d n,1 d n,2 d n, D i,j : Euclidean distance from x i to x j 14

The distance between x 2 and the k th NN to x 2 Nearest Neighbor Matrix N 1 0t 1,2 t 1,3 t 1,n 2 0t 2,2 t 2,3 t 2,n 3 0t 3,2 t 3,3 t 3,n N 0t n,2 t n,3... t n,n T i,k : Euclidean distance between x i and the k th NN to x i 15

Notation m: Intrinsic Dimension p: Dimension of the Raw Dataset n: Number of Observations f(x): density pdf for observation x T x,k or T k : distance from observation x to k th NN N(t,x): # obs within dist t of observation x 16

t N(t,x) = 3 Notation t1t1 t3t3 t2t2 x p = 2 m = 1 N = 12 17

NN Regression Estimator Density of Distance to k th NN (Single Observation, appx as Poisson) 1 Expected Distance to k th NN (Single Observation) 2a Sample-Averaged Distance to k th NN 2b Expected Distance to Sample-Averaged k th NN 3

Regression Estimator 19 Trinomial Distribution Binomial Distribution Distance to K th NN pdf Assumptions f(x) is constant n is large f(x)V t is small

Regression Estimator Approximate as Poisson Expected distance to K th NN

CnCn G k,m Estimate m using simple linear regression 21

Ex: Swiss Roll Dataset 22 m=0.49

Datasets Faces: Raw Dimension = 4096, Int Dim ~ 3 to 5 Gaussian Sphere Raw Dim = 3 Int Dim = 3 Swiss Roll Raw Dim = 3 Int Dim = 2 Dbl Swiss Roll Raw Dim = 3 Int Dim = 2 23

Results Regression Estimator FACES ~ 3.0 ~ 2.0 ~ K = N / 100

NN Maximum Likelihood Estimator Counting Process Binomial (appx as Poisson) 1 Joint Counting Probability Joint Occurrence Density 2 Log-likelihood Function 3 4

Maximum Likelihood Estimator 26 N(t,x) = # Counts within Distance t of x # Counts btw Distance r and s is BIN

Maximum Likelihood Estimator

28 Joint pdf of Distances to K NN

29 Log-Likelihood Function

Averaging over N observations Averaging inverses over N observations (Using MLE) E. Levina & P. Bickel D. MacKay & Z. Ghahramani 30

Results MLE Estimator (Revised MacKay & Ghahramani) FACES ~ 3.0 ~ 2.0 ~ 2.1 ~ K = N / 100

Comparison 32

Comparison 33

Comparison 34

Comparison 35

Comparison 36

Comparison 37

Isomap 38

Summary The regression and revised MLE estimators share similar characteristics when intrinsic dimension is small As intrinsic dimension increases, the estimators become more dependent on K Distribution type does not appear to be highly influential when the intrinsic dimension is small 39

Thank You! Dr. Kang James & Dr. Barry James Dr. Steve Trogdon

Example Swiss Roll Data Int Dim = 2