1. Mathematical Analysis of MaxEnt for Mixed Pixel Decomposition
Lidan Miao
AICIP Group Meeting
Feb. 23, 2006

2. Motivation Why do we choose maximum entropy?
What role does the maximum entropy play in the algorithm?
In what sense the algorithm converges to the optimal solution? (Is there anything else beyond the maximum entropy?)

3. Decomposition Problem
Decompose a mixture into constituent materials (assumed to be known in this presentation) and their proportions.
Mixing model
Given A, x, find s
Linear regression problem
Physical constraints
Non-negativity
Sum of s’ components is 1

4. QP & FCLS[1] Quadratic programming (QP)
Nonlinear optimization
Computationally expensive
Fully constrained least squares (FCLS)
Integrates SCLS and NCLS
NCLS is based on standard algorithm NNLS
Is the least square estimation the best in all cases?

5. Geometric Illustration
Mixing model & convex combination
Unconstrained least squares
Solve linear combination problem
Constrained least squares (QP and FCLS)
Sum-to-one: on the line connecting a1 and a2
Nonnegative: in the cone C determined by a1 and a2
Feasible set of convex combination: line segment a1a2

6. MaxEnt[2] Objective function Optimization method Limitation
Maximize entropy
Optimization method
Penalty function method
Limitation
Low convergence rate
Theoretically, Kk needs to go to infinity
For each Kk, s has no closed form solution, numerical method is needed (gradient descent)
Low performance when SNR is high
It can never fits the measurement model as Kk can not be infinity
Negative relative entropy

7. Gradient Descent MaxEnt
Optimization formulation
Minimize negative entropy
Optimization method
Lagrange multiplier method
Gradient descent learning

8. Convergence Analysis of GDME
Initialization: (neg-entropy)
Lambda will warp the original objective function to fit the data measurement model.
Searching in the feasible set
True s1:
Estimation:
Like MaxEnt[2], the solution is obtained by warping the objective function. Unlike MaxEnt[2], the force is a vector instead of a scalar, which is not necessary to be infinity to fit the measurement model.

9. Convergence Analysis (cont)
Take the first iteration as an example
The multiplier is the scaled error vector
S depends on the inner product of A and lambda
The denominator of s is only for normalization
Exponential function is used to generate a nonnegative number
The key is the inner product

10. Convergence Analysis (cont)
2D case is simple for visualization
Where to move?
Objective?
Proof

11. Convergence Rate

12. Stopping Conditions FCLS: S components are all non-negative
Generates least square solutions, minimizes ||As-x||
When SNR is low, the algorithm overfits the noise
MaxEnt[2]: the same with FCLS
The solution is not least squares
Can never fits the data perfectly
GDME: S is relatively stable
Is able to give least square solution
The solution lies somewhere between equal abundances and the least square estimation, which determined by the stopping condition

13. Two groups of testing Data
Experimental Results
MaxEnt[2] is too slow to be applied to a big image, so the simple data in ref[2] is used
Two groups of testing Data
Class
Mean
Covariance A 73 33 29 B 89 46 74 C
109 61
100
Class
Mean
Covariance D
92.6
48.1
79.9 E
96.3
50.5
82.0 F
100.2
53.6
89.8
Results
Mix
Method
Err
ABC
Conventional
MaxEnt[2]
FCLS
GDME
[2]
[2]
0.1163
0.0789
DEF
[2]
[2]
0.2819
0.1363
250 mixed pixels with the abundance randomly generated
The average of 50 runs
Par: 500, 4

14. Experimental Results (cont)
Apply to synthetic hyperspectral images
Metrics: ARMSE, AAD, AID

15. Summary of GDME The same target as QP and FCLS, i.e., min ||As-x||
The maximum entropy formulation is used to incorporate the two constraints through exponential function and normalization.
Does maximum entropy really play a role?
By carefully selecting stopping condition, GDME on average is able to generate better performance in terms of abundance estimation.
The convergence rate is faster than QP and MaxEnt[2] and similar to FCLS (based on experiments)
GDME is more flexible, which presents strong robustness under low SNR cases
GDME presents better performance when source vectors are close to each other

16. Future Work Speed up the learning algorithm
Investigate optimal stop conditions (what’s the relationship between SNR and stop condition?)
Study the performance w.r.t the number of constituent materials

17. Reference
[1] D. C. Heinz and C.-I Chang. Fully constrained least squares linear spectral mixture analysis method for material quantification in hyperspectral imagery. IEEE Trans. Geosci. Remote Sensing, vol.39, no. 3, pp , 2001.
[2] S. Chettri and N. Netanyahu. Spectral unmixing of remotely sensed imagery using maximum entropy. In Proc. of SPIE, vol. 2962, pp. 55–62, 1997.