Preprocessing With focus on NIR Åsmund Rinnan , Frans van den Berg, Rasmus Bro, Søren Balling Engelsen, Lars Nørgaard , Jonas Thygesen

Introduction How often is it used? When is it used? Techniques Summary How often is it used? When is it used? What methods are used? Why is it used? NIR is a measurement technique “screaming” for preprocessing

Introduction NIR Fructose Glucose 50 75 100 50 25 Introduction Techniques Summary Fructose Glucose 50 75 100 50 25

Introduction Techniques Summary Correction of light scatter MSC/ ISC PMSC EMSC/ EISC SNV Derivation Norris&Williams Savitsky&Golay Baseline-correction Normalization Detrend Use of reference values O-PLS OSC OS SIS

Introduction Why Baseline with a slope/ curve Nonlinearity Techniques Summary Baseline with a slope/ curve Nonlinearity A “fat” baseline

ISC MSC Offset Normalization Raw spectrum Reference Techniques Introduction Techniques Summary Offset Normalization Raw spectrum Reference P Geladi, D MacDougal, H Martens (1985): Linearization and scatter correction for near-infrared reflectance spectra of meat, Applied Spectroscopy, 39, 491-500

MSC/ ISC Introduction Techniques Summary Offset Normalization

EISC Offset Baseline Normalization Techniques Introduction Summary H. Martens and E. Stark (1991): Extended multiplicative signal correction and spectral interference subtraction: new preprocessing methods for near infrared spectroscopy, Journal of Pharmaceutical and Biomedicinal Analysis, 9, 625-635 H Martens, JP Nielsen, SB Engelsen (2003): Light scattering and light absorbance separated by extended multiplicative signal correction. The application to near-infrared transmission analysis of powder mixtures, Analytical chemistry, 75, 394-404 DK Pedersen, H Martens, JP Nielsen, SB Engelsen (2002): Near-Infrared Absorption and Scattering Separated by Extended Inverse Signal Correction (EISC): Analysis of Near-Infrared Transmittance Spectra of Single Wheat, Applied Spectroscopy, 56, 1206-1214 Thennadil, SN, Martin, EB (2005): Empirical preprocessing methods and their impact on NIR calibrations - a simulation study, Journal of Chemometrics, 19, 77-89

EMSC (w/ 2. degree & wavelength) Introduction Techniques Summary Offset Baseline Normalization

PxSC Offset Baseline Normalization X = EM EI M I Techniques Introduction Techniques Summary Offset Baseline Normalization X = EM EI M I Isaksson, T.; Kowalski, B. R. (1993): Piece-wise multiplicative scatter correction applied to Near-Infrared diffuse transmittance data from meat products, Applied Spectroscopy, 47, 702-09

PMSC (window size = 35) Offset Normalization Techniques Introduction Summary Offset Normalization

SNV Offset Normalization Techniques Introduction Summary R.J. Barnes, M.S. Dhanoa, S.J. Lister (1989): Standard Normal Variate Transformation and De-trending of Near-Infrared Diffuse Reflectance Spectra, Applied Spectroscopy, 43, 772-777

SNV vs MSC Introduction Techniques Summary MSC SNV

Detrend Offset Baseline Techniques Introduction Summary R.J. Barnes, M.S. Dhanoa, S.J. Lister (1989): Standard Normal Variate Transformation and De-trending of Near-Infrared Diffuse Reflectance Spectra, Applied Spectroscopy, 43, 772-777

Derivation Savitsky-Golay Introduction Techniques Summary Offset Baseline A Savitsky, MJE Golay (1964): Smoothing and differentiation of data by simplified least squares procedures, Analytical Chemistry, 36 (8), 1627-1639 P.A. Gorry (1990): General Least-Squares smoothing and differentiation by the convolution (Savitsky-Golay) method, Analytical Chemistry, 62, 570-573

Derivation Savitsky-Golay Introduction Techniques Summary Offset Baseline

Derivation Savitsky-Golay Introduction Techniques Summary Offset Baseline

Derivation Norris-Williams Introduction Techniques Summary Offset Baseline Normalization Norris, KH (1983): in Food Research and Data Analysis (Eds: Martens H and Russwurm H Jr), Applied Science, London, 95-114 Norris KH, Williams PC (1984): Optimization of mathematical treatments of raw near-infrared signal in the measurement of protein in hard red spring wheat: I. Influence of particle size, Cereal Chemistry, 61, 158-165

Derivation Norris-Williams Introduction Techniques Summary Offset Baseline Normalization

Derivation Norris-Williams Introduction Techniques Summary Offset Baseline Normalization

Preprocessing PMSC: Sensitivite to window size Introduction Techniques Summary PMSC: Sensitivite to window size EMSC: Small effect on baseline  Detrend is better MSC and SNV are practical the same Savitsky-Golay is more robust than Norris-Williams

Preprocessing Offset Baselinje Normalization (P)MSC/ISC/SNV x Introduction Techniques Summary Offset Baselinje Normalization (P)MSC/ISC/SNV x (P)EMSC/EISC Detrend Deriveration

Thanks for the attention www.models.life.ku.dk