1. Addressing THE Problem of NIR

2. SYMPTOMS
The apparent need for, and use of more variables in calibrations, than can reasonably be justified
Algebra dictates that no more equations should be needed than variables
Difficulty in reproducing calibrations for the same constituents in the same type of samples.
Inability to reproduce wavelength selections (MLR models)
Difficulty and/or inability in relating wavelengths chosen (for MLR) or prominent wavelength bands (for PCR/PLS) to spectral features
Unexpected “Outliers”
SEC and SEP should drop precipitously when the number of wavelengths/factors equals the number of variations in the samples
Spectroscopic measurements should be accurate over the entire range of concentrations, not only dilute solutions or a small range of values
Calibrations should be extrapolatable, with a calculated reduced accuracy at the extremes
Calibration transfer should be as easily and readily performed as comparing two mid-IR spectra

3. Explanations Given OPTICAL SCATTER Lab error Instrument variations
“Wrong” transforms of spectral data
“Wrong” calibration algorithm

4. Published in Appl. Spect., 64(9), p.995-1006 (2010)
Pittcon 2011 Presentation
Published in Appl. Spect., 64(9), p (2010)

5. Simplified Experimental Design: Three Miscible Liquids, 15 Samples

6. Simplified Experimental Design:
Liquids = no scatter, Beer’s Law holds
Symmetric design = Balanced
Ranges = 0 – 100% for all components
Data Analysis
No data transform applied
CLS algorithm
Simplest algorithm
“Absolute” method
Origin: comes directly from Beer’s Law
No conventional calibration needed
No computation of factors: no overfitting
Requirements:
Spectra of pure materials known
No interactions between constituents
No Partial Molal Volume changes

7. Spectral Results vs “Concentrations”
Concentrations as Weight Fractions
Concentrations as Volume Fractions

8. Underlying Cause: Weight Fraction vs Volume Fraction
Toluene
Dichloromethane
N-heptane

9. Underlying Cause: Weight Fraction vs Volume Fraction
Concentrations expressed in different units are NOT linearly related.
Spectroscopy is sensitive to VOLUME fraction
The nature of the curvature depends on the constituent. On the average, toluene shows little or no non-linearity; the other components have opposite senses. The amount of curvature depends on the exact mixture.
Toluene

10. Underlying Cause: Weight Fraction vs Volume Fraction
Toluene
Dichloromethane
When samples are selected at random, you don’t see the underlying curvature, The error values seem randomly scattered around the average curve, just as though the error was random instead of systematic, as a result of the random sample selection. A calibration, however, will show curvature in the residuals.
N-heptane

11. Expanding Beyond the Small Dataset
The foregoing makes a compelling case that for spectroscopic analysis, EM radiation “sees” the absorbance according to the volume fractions of their components (given some assumptions to ensure ideality)
The CLS algorithm responds according to the theory
This does not necessarily ensure that conventional calibration algorithms will respond the same way.
The fifteen samples we used is not sufficient number to reliably test conventional calibrations.

12. Expanding Beyond the Small Dataset
Seeking another, larger, suitable sample set was successful in finding a pair of reports, by Willem Windig, et al, based on data suitable for our purposes:
Chemom. & Intell. Lab. Syst.,36, p.3-16 (1997)
Anal. Chem., 54, p (1992)
Data supplied by Tony Davies and Tormod Naes

13. Expanding Beyond the Small Dataset
The set contained mixtures of 5 ingredients:
2-butanol
Methylene chloride
Methanol
1,2-Dichloropropane
Acetone
Orthogonal mixture design provided 70 different mixtures, each measured twice.
Each ingredient had one of the following percentage concentrations:
10, 22.5, 35, 47.5, 60 weight percent.
The properties of the mixture design are described on the following slide:

14. Orthogonal Mixture Design Used Each of the ten plots of ingredients taken pairwise look like this

15. Pure Material Spectra

16. Pure Material Spectra Butanol Methanol Dichloromethane Dichloropropane
Acetone

17. Plot of Data Spectra (1st & 2nd readings)

18. Plot of Mixture Spectra (Maximum Acetone) (1st & 2nd readings)

19. Plot of Mixture Spectra (Maximum Butanol) (1st & 2nd readings)

20. Plot of Mixture Spectra (Maximum Dichlorprop.) (1st & 2nd readings)

21. Plot of Mixture Spectra (Maximum Methanol) (1st & 2nd readings)

22. Plot of Mixture Spectra (Maximum Methyl Chl) (1st & 2nd readings)

23. Densities Component Density 2-Butanol 0.808 Methylene Chloride 1.336
Methanol
0.7928
1,2-Dichloropropane
1.1593
Acetone
0.792

24. CLS Results (Reconstruction of max Acetone mixture)

25. CLS Results (Reconstruction of max Butanol mixture)

26. CLS Results (Reconstruction of max Dichlorpropane mixture)

27. CLS Results (Reconstruction of max Methanol mixture)

28. CLS Results (Reconstruction of max Methyl Chl mixture)

29. CLS Results (Spectral vs Reference)
Methanol Spectral Results vs Weight Fraction
Acetone Spectral Results vs Weight Fraction
Methanol Spectral Results vs Volume Fraction
Acetone Spectral Results vs Volume Fraction

30. CLS Results (Spectral vs Reference)
Acetone Spectral Results vs Weight Fraction (Butanol and Methylene Chloride held constant)
Methanol Spectral Results vs Weight Fraction (Butanol and Methylene Chloride held constant)
Methanol Spectral Results vs Volume Fraction
Acetone Spectral Results vs Volume Fraction

31. CLS Results (Spectral vs Reference)
Butanol Spectral Results vs Weight Fraction
Methylene Chloride Spectral Results vs Weight Fraction
Butanol Spectral Results vs Volume Fraction
Methylene Chloride Spectral Results vs Volume Fraction

32. CLS Results (Spectral vs Reference)
Dichloropropane Spectral Results vs Weight Fraction
Dichloropropane Spectral Results vs Volume Fraction

33. Expanding Beyond CLS
Apply PCA, PCR, PLS

34. Explained PCA Variance (SVD): Each Half
PCA Variance Fraction
PC # Variance
30%
29%
3 22% %
5 0%
6 0%
Legend:
Blue = Calibration
Red = Validation

35. PCA Residual Variance of Calibration Spectra

36. PCA Residual Variance of Components (Wt %)

37. PCA Residual Variance of Components (Vol %)

38. PLS Wt % (Q) Residual Variance of Spectra

39. PLS Vol % (Q) Residual Variance of Spectra

40. PLS Residual Variance of Components (Wt %)

41. PLS Residual Variance of Components (Vol %)