1. NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1

2. Kernel Estimators: Parzen Windows Data True PDF Estimated PDF Shwartz, Schechner & Zibulevsky, NlogN entropy optimization 49

3. Previous Work Parametric PDF: Hyvärinen 98, Bell; Sejnowski 95, Pham; Garrat 97. Cumulants: Cardoso ; Souloumiac 93. Not accurate

4. Order statistics: Vasicek 76, Learned-Miller; Fisher 03. KD trees: Gray; Moore 03. Previous Work Not differentiable

5. Entropy Estimation Kernel Estimators: reduced complexity Pham, 03,. Erdogmus; Principe; Hild, 03, Morejon; Principe 04, Schraudolph 04, (Stochastic gradient).

6. Source Range: Continuous

7. Parzen Windows Estimator Shwartz, Schechner & Zibulevsky, NlogN entropy optimization 50

8. Minimization of Mutual Information Differentiable Computationally efficient - Currently O ( K N ) Independent Component Analysis Shwartz, Schechner & Zibulevsky, NlogN entropy optimization online code (see website) 51 2 2

9. ConvolutionSampling Parzen Windows as a Convolution Shwartz, Schechner & Zibulevsky, NlogN entropy optimization Wish it was … Discrete convolution 52

10. Efficient Kernel Estimator A.Samples of estimated sources A PDF estimation Fan; Marron 94, Silverman 82. Shwartz, Schechner & Zibulevsky, NlogN entropy optimization 53

11. A.Samples of estimated sources B.Interpolation to uniform grid (histogram) A B Efficient Kernel Estimator PDF estimation Fan; Marron 94, Silverman 82. Shwartz, Schechner & Zibulevsky, NlogN entropy optimization 53

12. C Samples of estimated sources Interpolation to uniform grid (histogram) Discrete convolution with Parzen window A B PDF estimation Fan; Marron 94, Silverman 82. Efficient Kernel Estimator Shwartz, Schechner & Zibulevsky, NlogN entropy optimization 53

13. D Efficient Entropy Estimator C Interpolation to original values A Shwartz, Schechner & Zibulevsky, NlogN entropy optimization 54

14. Can it be Used for Optimization? W separate Iterations exploiting derivatives of. Shwartz, Schechner & Zibulevsky, NlogN entropy optimization 55

15. Can it be Used for Optimization? W separate Binning fluctuations of. Fluctuations amplified by differentiation. Fluctuations slow convergence, false minima. Shwartz, Schechner & Zibulevsky, NlogN entropy optimization 56

16. Function Quantized function Quantization and Optimization Shwartz, Schechner & Zibulevsky, NlogN entropy optimization 57

17. Function Quantized function Function with a quantized derivative Quantization and Optimization Shwartz, Schechner & Zibulevsky, NlogN entropy optimization 57

18. Analytic Entropy Gradient Accurate derivative Efficient calculation Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

19. Complexity Analytic Entropy Gradient K - number of sources, N -data length. Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

20. Entropy Gradient by Convolutions Convolution Shwartz, Schechner & Zibulevsky, NlogN entropy optimization 58

21. Calculation of using convolutions. Approximation of convolutions with complexity. Distinct quantization of the derivative. Not differentiation of a quantized function. Entropy Gradient by Convolutions Shwartz, Schechner & Zibulevsky, NlogN entropy optimization 59

22. K =6 random sources, N = 3000 samples. Algorithm Signal to Interference ratio [dB] Time Basic Non-param ICA 18 4760 min. Our algorithm22 31.2 min. Jade7 40.2 sec. Infomax8 41.6 sec. Fast ICA5 31.9 sec. Super ICA performance Parametric algorithms. Non-parametric algorithms. Shwartz, Schechner & Zibulevsky, NlogN entropy optimization 60