1. Calculating the entropy on-the-fly Daniel Lewandowski Faculty of Information Technology and Systems, TU Delft

2. Introducing a function h. h - is a measure of the uncertainty about the outcome of an experiment modelled using probability distributions

3. We assume that h: depends only on the probability of the outcome of an experiment or event takes values in non-negative real numbers is a continuous and decreasing function h(p 1 p 2 )=h(p 1 )+h(p 2 ) Assumptions The assumptions forces h to be of the form: h(p)= - C log(p)

4. Definition of entropy The entropy H is the expectation of the function h. Example: x 1, x 2,…,x n are realizations of a rand. variab. X with probabilities p 1, p 2,…,p n respectively. Then the entropy of X is:

5. Units in which the entropy is measured log 2 (x) – bits log 3 (x) – trits ln(x) – nats log 10 (x) – Hartleys

6. Entropy of some continuous distributions The standard normal (Gaussian) distribution: H = 1,4189 The Weibull distribution (  =1,127;  =2,5): H = 0,5496 The Weibull distribution (  =1,107;  =1,5): H = 0,8892 The gamma distribution ( =  =5): H = 0,5441

7. Approximation of the density Y 1,Y 2,…,Y n – samples D 0,D 1,…,D n – midpoints D 0 =Y 1 – (Y 2 – Y 1 )/2, D i =Y i+1 – (Y i+1 – Y i )/2,for i=2,…,n-1 D n =Y n + (Y n – Y n-1 )/2,

8. Computations The density above the Y i is estimated as: The entropy is then computed as:

9. Grouping samples Remark: The result of calculating the entropy without grouping samples is biased – the bias is asymptotically equal to  - 1 + ln2, (  - Euler constant)

10. Numerical test – 5000 samples Entropy 1,1591,3721,3991,4581,4381,4181,324 Exact : 1,418 The red line marks the exact density function of a standard normal vrb.

11. Results – 20 iterations (1000 samples) standard normal Weibull  =2,5 Weibull  =1,5 gamma  = =5 by 1’s mean1,1490,2850,6130,274 deviation0,0250,01810,0180,0266 by 25’s mean1,4240,5460,8810,546 deviation0,0140,02220,02090,0243 by 50’s mean1,4590,5680,9040,575 deviation0,0330,02660,02670,029 by 100’s mean1,5060,6160,9430,628 deviation0,0390,0280,03320,034 Compare results to exact solutions from slide 6

12. Updating the distribution DkDk D k+2 D k+1 YkYk Y k+1 Y (N+1) D (N+1) D (N+2) before updating after updating

13. Updating the entropy H N – the entropy calculated based on N samples. where :

14. Uses the approach from the previous slide Starts updating the entropy from N=4 Written in VBA, uses spreadsheet only to store the samples Results exactly the same as computed in Matlab (for the same samples) It is not grouping samples The program - properties

15. Results Comparison of results obtained using formula and program (5000 samples – without grouping and adding the bias). formulaprogram standard normal 1,143918 Weibull  =2,5 0,290707 Weibull  =1,5 0,634027 gamma  = =5 0,280102 The program updates the entropy H N starting from N = 4.

16. Results, cont. exact solutionprogram (1000 samples) program (5000 samples) standard normal 1,41891,42721,4209 Weibull  =2,5 0,54960,54820,5709 Weibull  =1,5 0,88920,85430,8849 gamma  = =5 0,54410,52060,5544

17. Relative information – theoretical value = 2,0345