Mathematical Analysis of MaxEnt for Mixed Pixel Decomposition Lidan Miao AICIP Group Meeting Feb. 23, 2006

Motivation Why do we choose maximum entropy? What role does the maximum entropy play in the algorithm? In what sense the algorithm converges to the optimal solution? (Is there anything else beyond the maximum entropy?)

Decomposition Problem Decompose a mixture into constituent materials (assumed to be known in this presentation) and their proportions. Mixing model Given A, x, find s Linear regression problem Physical constraints Non-negativity Sum of s’ components is 1

QP & FCLS[1] Quadratic programming (QP) Nonlinear optimization Computationally expensive Fully constrained least squares (FCLS) Integrates SCLS and NCLS NCLS is based on standard algorithm NNLS Is the least square estimation the best in all cases?

Geometric Illustration Mixing model & convex combination Unconstrained least squares Solve linear combination problem Constrained least squares (QP and FCLS) Sum-to-one: on the line connecting a1 and a2 Nonnegative: in the cone C determined by a1 and a2 Feasible set of convex combination: line segment a1a2

MaxEnt[2] Objective function Optimization method Limitation Maximize entropy Optimization method Penalty function method Limitation Low convergence rate Theoretically, Kk needs to go to infinity For each Kk, s has no closed form solution, numerical method is needed (gradient descent) Low performance when SNR is high It can never fits the measurement model as Kk can not be infinity Negative relative entropy

Gradient Descent MaxEnt Optimization formulation Minimize negative entropy Optimization method Lagrange multiplier method Gradient descent learning

Convergence Analysis of GDME Initialization: (neg-entropy) Lambda will warp the original objective function to fit the data measurement model. Searching in the feasible set True s1: 0.1619 Estimation: 0.1638 Like MaxEnt[2], the solution is obtained by warping the objective function. Unlike MaxEnt[2], the force is a vector instead of a scalar, which is not necessary to be infinity to fit the measurement model.

Convergence Analysis (cont) Take the first iteration as an example The multiplier is the scaled error vector S depends on the inner product of A and lambda The denominator of s is only for normalization Exponential function is used to generate a nonnegative number The key is the inner product

Convergence Analysis (cont) 2D case is simple for visualization Where to move? Objective? Proof

Convergence Rate

Stopping Conditions FCLS: S components are all non-negative Generates least square solutions, minimizes ||As-x|| When SNR is low, the algorithm overfits the noise MaxEnt[2]: the same with FCLS The solution is not least squares Can never fits the data perfectly GDME: S is relatively stable Is able to give least square solution The solution lies somewhere between equal abundances and the least square estimation, which determined by the stopping condition

Two groups of testing Data Experimental Results MaxEnt[2] is too slow to be applied to a big image, so the simple data in ref[2] is used Two groups of testing Data Class Mean Covariance A 73 33 29 2.5 1.6 1.3 1.6 2.5 1.6 1.3 1.6 2.1 B 89 46 74 11.3 7.3 12.2 7.3 6.0 9.4 12.2 9.4 17.3 C 109 61 100 9.6 6.1 9.0 6.1 5.2 7.7 9.0 7.7 14.9 Class Mean Covariance D 92.6 48.1 79.9 5.4 3.1 4.7 3.1 2.9 4.0 4.7 4.0 7.1 E 96.3 50.5 82.0 12.2 6.3 5.7 6.3 4.4 5.0 5.7 5.0 13.2 F 100.2 53.6 89.8 6.0 4.2 7.1 4.2 4.2 6.5 7.1 6.5 12.0 Results Mix Method Err ABC Conventional MaxEnt[2] FCLS GDME 0.0994 [2] 0.0832 [2] 0.1163 0.0789 DEF 0.3304 [2] 0.1591 [2] 0.2819 0.1363 250 mixed pixels with the abundance randomly generated The average of 50 runs Par: 500, 4

Experimental Results (cont) Apply to synthetic hyperspectral images Metrics: ARMSE, AAD, AID

Summary of GDME The same target as QP and FCLS, i.e., min ||As-x|| The maximum entropy formulation is used to incorporate the two constraints through exponential function and normalization. Does maximum entropy really play a role? By carefully selecting stopping condition, GDME on average is able to generate better performance in terms of abundance estimation. The convergence rate is faster than QP and MaxEnt[2] and similar to FCLS (based on experiments) GDME is more flexible, which presents strong robustness under low SNR cases GDME presents better performance when source vectors are close to each other

Future Work Speed up the learning algorithm Investigate optimal stop conditions (what’s the relationship between SNR and stop condition?) Study the performance w.r.t the number of constituent materials

Reference [1] D. C. Heinz and C.-I Chang. Fully constrained least squares linear spectral mixture analysis method for material quantification in hyperspectral imagery. IEEE Trans. Geosci. Remote Sensing, vol.39, no. 3, pp. 529-545, 2001. [2] S. Chettri and N. Netanyahu. Spectral unmixing of remotely sensed imagery using maximum entropy. In Proc. of SPIE, vol. 2962, pp. 55–62, 1997.