1. 23.02.03 1 Successive Bayesian Estimation Alexey Pomerantsev Semenov Institute of Chemical Physics Russian Chemometrics Society

2. 23.02.03 2 Agenda 1.Introduction. Bayes Theorem 2.Successive Bayesian Estimation 3.Fitter Add-In 4.Spectral Kinetics Example 5.New Idea (Method ?) 6.More Applications of SBE 7.Conclusions

3. 23.02.03 3 1. Introduction

4. 23.02.03 4 The Bayes Theorem, 1763 Thomas Bayes (1702-1761) Posterior ProbabilityPrior Probabilities L(a,  2 )=h(a,  2 )L 0 (a,  2 ) Likelihood Function Where to take the prior probabilities?

5. 23.02.03 5 Jam Sampling & Blending Theory 0.200.300.50 0.200.05 Now we know the origin of a worm in the jam!

6. 23.02.03 6 2.Successive Bayesian Estimation (SBE)

7. 23.02.03 7 SBE Concept SBE principles 1)Split up whole data set 2)Process each subset alone 3)Make posterior information 4)Build prior information 5)Use it for the next subset How to eat away an elephant? Slice by slice!

8. 23.02.03 8 OLS & SBE Methods for Two Subsets OLS SBE Quadratic approximation near the minimum!

9. 23.02.03 9 Posterior & Prior Information Subset 1. Posterior Information Rebuilding (common & partial parameters) Subset 2. Prior Information Make Posterior, rebuild it and apply as Prior!

10. 23.02.03 10 Prior Information of Type I Posterior InformationPrior Information Parameter estimates Prior parameter values b Matrix A Recalculated matrix H Variance estimate s 2 Prior variance value s 0 2 NDF N f Prior NDF N 0 Objective Function The same error variance in the each subset of data!

11. 23.02.03 11 Prior Information of Type II Posterior InformationPrior Information Parameter estimates Prior parameter values b Matrix A Recalculated matrix H Objective Function Different error variances in the each subset of data!

12. 23.02.03 12 SBE Main Theorem Different order of subsets processing Theorem (Pomerantsev & Maksimova, 1995) SBE agree with OLS!

13. 23.02.03 13 3. Fitter Add-In

14. 23.02.03 14 Fitter Workspace Fitter is a tool for SBE!

15. 23.02.03 15 Data & Model Prepared for Fitter Response Weight Fitting Predictor Parameters Equation Comment Values Apply Fitter!

16. 23.02.03 16 Model f(x,a) Different shapes of the same model Explicit model y = a + (b – a) * exp(–c * x) Implicit model 0 = a + (b – a) * exp(–c * x) – y Diff. equation d[y]/d[x] = – c * (y –a); y(0) = b Presentation at worksheet Rather complex model!

17. 23.02.03 17 4. Spectral Kinetics Modeling

18. 23.02.03 18 Spectral Kinetic Data Y(t,x,k)=C(t,k)P(x)+E Y is the ( N  L ) known data matrix C is the ( N  M ) known matrix depending on unknown parameters k P is the ( M  L ) unknown matrix of pure component spectra E is the ( N  L ) unknown error matrix K constants L wavelengths M species N time points This is large non-linear regression problem!

19. 23.02.03 19 How to Find Parameters k? MethodIdeaDimensionProblem Full OLS (hard) K+M  L >> 1 Large dimension Short OLS (hard) K+M  S  10 Small precision WCR (hard&soft) K  10 Matrix degradation GRAM (soft) K+M  A  100 Just one model This is a challenge!

20. 23.02.03 20 Simulated Example Goals Compare SBE estimates with ‘true’ values Compare SBE estimates for different order Compare SBE estimates with OLS estimates

21. 23.02.03 21 Model. Two Step Kinetics ‘True’ parameter values k 1 =1 k 2 =0.5 Standard ‘training’ model

22. 23.02.03 22 Data Simulation C 1 (t) = [A](t) C 2 (t) = [B](t) C 3 (t) = [C](t) P 1 (x) = p A (x) P 2 (x) = p B (x) P 3 (x) = p C (x) Simulated concentration profilesSimulated pure component spectra Y(t,x)=C(t)P(x)(I+E) STDEV(E)=0.03 Usual way of data simulation

23. 23.02.03 23 Simulated Data. Spectral View Spectral view of data

24. 23.02.03 24 Simulated Data. Kinetic View Kinetic view of data

25. 23.02.03 25 One Wavelength Estimates Conventional wavelength 3 Estimates Conventional wavelength 14 Conventional wavelength 51 Bad accuracy!

26. 23.02.03 26 Direct order Estimates Four Wavelengths Estimates Inverse order Random order Bad accuracy, again!

27. 23.02.03 27 SBE Estimates at the Different Order Direct 1, 2, 3, …. Random 16, 5, 29, …. Inverse 53, 52, 51, …. 0.95 Confidence Ellipses SBE (practically) doesn’t depend on the subsets order!

28. 23.02.03 28 SBE Estimates and OLS Estimates SBE estimates are close to OLS estimates!

29. 23.02.03 29 Pure Spectra Estimating SBE gives good spectra estimates!

30. 23.02.03 30 Real World Example Goals Apply SBE for real world data Compare SBE with other known methods

31. 23.02.03 31 Data Bijlsma S, Smilde AK. J.Chemometrics 2000; 14: 541-560 Epoxidation of 2,5-di-tert-butyl-1,4-benzoquinone SW-NIR spectra 240 spectra 1200 time points 21 wavelengths Preprocessing: Savitzky-Golay filter Preprocessed Data

32. 23.02.03 32 Progress in SBE Estimates SBE works with the real world data!

33. 23.02.03 33 SBE and the Other Methods SBE gives the lowest deviations and correlation!

34. 23.02.03 34 5. New Idea

35. 23.02.03 35 y=a 1 x 1 +a 2 x 2 +a 3 x 3 Bayesian Step Wise Regression Ordinarily Step Wise RegressionBayesian Step Wise Regression Objective function BSWR accounts correlations of variables in step wise estimation

36. 23.02.03 36 BSW Regression & Ridge Regression BSWR is RR with a moving center and non-Euclidean metric

37. 23.02.03 37 Example. RMSEC & RMSEP BSWR gives typical U-shape of the RMSEP curve

38. 23.02.03 38 Linear Model. RMSEC & RMSEP y=a 1 x 1 +a 2 x 2 +a 3 x 3 +a 4 x 4 +a 5 x 5 BSWR is not worse then PLS or PCR and better then SWR

39. 23.02.03 39 Non-Linear Model. RMSEC & RMSEP For non-linear model BSWR is better then PLS or PCR

40. 23.02.03 40 Variable selection BSWR is just an idea, not the method so any criticism is welcomed now!

41. 23.02.03 41 6. More Practical Applications of SBE

42. 23.02.03 42 Antioxidants Activity by DSC DSC DataOxidation Initial Temperature (OIT) To test antioxidants!

43. 23.02.03 43 Network Density of Shrinkable PE by TMA TMA DataNetwork density To solve technological problem!

44. 23.02.03 44 PVC Isolation Service Life by TGA TGA DataService life prediction To predict durability!

45. 23.02.03 45 Tire Rubber Storage Elongation at breakTensile strength To predict reliability!

46. 23.02.03 46 7. Conclusions 1SBE is of general nature and it can be used for any model 2SBE agrees with OLS 3 SBE gives small deviations and correlations 4SBE uses no subjective a priori information 5SBE may be useful for non-linear modeling (BWSR?) Thanks!