| Author | Message | ||
     DJDahm |
My definitions: Calibration is the processing that one does in order to find the relationships in a data set between spectral characteristics and the concentrations in the samples. Preprocessing is the act of manipulating the data prior to feeding it into the calibration process. In my view, the problem in terminology springs from the fact that people empirically try many preprocessing schemes to see which one gives the best calibration results. Then the preprocessing has essentially become part of the calibration process. I don't hang around people doing calibrations all that much anymore, but I think the preprocessing step used in the pure form (that is, done once and then forgotten about) most often is multiplicative signal correction (MSC). [Of course the exception to that is the log(1/T) or log(1/R), which is preprocessing in a very pure form.] | ||
| Tony Davies (Td) |
As Howard says, we could keep this one running for some time! I think that when we talk about "Calibration" we should be talking about the whole exercise of designing the experiment, obtaining a dataset ... to validating that the selected calibration works on a different instrument with fresh samples. After we have got the spectral data, probably used PCA to detect any really serious outliers/bad spectra, and before we use our favorite regression technique, we often manipulate the spectra with a range of mathematical techniques; this is preprocessing. It is a separate step in the calibration exercise and deserves a name. Just a footnote to all correspondents: If you ask a new question please start a new thread NOT tack it on to an ongoing discussion. Best wishes, Tony | ||
| Bruce H. Campbell (Campclan) |
To get to the basics of language, if the act of calibration using PLS, etc is processing, then preprocessing is the step prior (pre) to it. However, one could argue that all steps are processing and use of the word preprocessing is redundant. I know this doesn't answer the original question, but I just had to post this. | ||
| W. Fred McClure (Mcclure) |
Preprocessing? This is an interesting discussion. Tony's definition saftisfies me. However, no one has admitted that very fact that the matter of preprocessing is purely an art - not a science. To prove this (and I only need one case to prove it), give me your best shot at a calibration and I will give you one equally good, but by a different pretreatment path. Here is a good example. For a given data set, I suggested to Howard Mark that he and I have a shootout - he come up with his best calibration and I come up with mine. In the end, Dr. Mark would not participate bacause he wanted to know what pretreatment I was going to use. Let's face it, Gentlemen. There are still cases where stepwise multiple linear regression (SMLR) performs better as a calibration that PLS! And, NO ONE can give a reason why. Pretreatment is still a matter of personal choice. Whatever "floats your boat," use it. It is as good as any. Fred | ||
| hlmark |
That's OK, Tony. I tried to define what they are and you tried to define why people do them. I actually have a more-or-less serious problem with defining something as nebulous as "preprocessing". The problem I see with any definition, is how do you draw the line between those mathematical manipulations that we consider "preprocessing" and those we consider "calibrating"? After all, every calibration exercise is an effort to determine the analyte, combined with an "attempt to make corrections for non chemical influences on the spectrum." Note that the stuff in quotes was a cut-and-paste from your message. So how can you make the distinction (this should be good for a flame war that can last a month or more, I think!!)? I think that for any criterion you try to use, you can find an exception. F'rinstance, you might way that multi-wavelength methods, or certainly full-spectral methods are not "preprocessing"; that seems like an obvious fool-proof criterion. But then what about MSC? And you could come up with other criteria, and probably they could all be shot down that way. Maybe the only way is to define it like they used to define "life" when I took biology: there was no single criterion, because for any single criterion you could find an exception. So they said that in order to be considered "life" an object had to do the functions of ingestion, respiration, growth, reproduction, assimilation, excretion and I think a couple of others that I forget now. But maybe we'd have to do something like that for "preprocessing". It's not very satisfying, but it might be the only thing tha will work. \o/ /_\ | ||
| td |
Hello Christina, I would define preprocessing (in relation to NIR spectroscopy)as: Mathematical treatments of spectral data which attempt to make corrections for non chemical influences on the spectrum. However, some of the more advanced methods do more so then I would have to say: Mathematical treatments of spectral data which attempt to make corrections for influences not related to the analyte of interest. I expect you will get further suggestions (Over to you Howard). Best wishes and Good Night, Tony | ||
| hlmark |
Christina - "preprocessing" is simply a generic term referring to any mathematical manipulations of spectra (or in general, any data, but after all, this is a discussion group about SPECTROSCOPY!) performed before applying a calibration algorithm. Other terms used are "data transforms", "data transformations" and other similar generic descriptors. Strictly speaking, even the conversion of the measured data to log(1/R) inside an instrument is a form of preprocessing. But since log(1/R) is sort of considered the "basic" spectroscopic measurement in NIR, that calculation is usually exempt from being considered a "preprocessing" step. Howard \o/ /_\ | ||
| hlmark |
Christina - "preprocessing" is simply a generic term referring to any mathematical manipulations of spectra (or in general, any data, but after all, this is a discussion group about SPECTROSCOPY!) performed before applying a calibration algorithm. Other terms used are "data transforms", "data transformations" and other similar generic descriptors. Howard \o/ /_\ | ||
| Christina Timofeyenko (Ctimofeyenko) |
What is a good general definition for preprocessing...one for non-chemometricians? | ||
| hlmark |
Christina - Don is correct for clear transmission measurements. I would even extend his recommendation to reflectance-type measurements, i.e., transflectance, for example, as long as the sample itself is non-scattering, and you maintain the measurement geometry constant. If the sample is scattering, even a little bit, then the other major reason for preprocessing comes into play: to remove the extraneous physical variations from the data. This holds true for either type of measurement. In this case, derivatives of various orders are commonly used, as are normalization and MSC calculations. Howard \o/ /_\ | ||
| DJDahm |
I assume that you are making the measurements in transmission. The purpose of pre-processing is to make the absorption metric more linear with the concentration of the species of interest. The pre-processing that is done is the log(1/T). If I understand your problem correctly, I don't think you need to consider an additional correction. For a non-scattering sample measured in transmission, the major source of non-linearity ater taking log(1/T) is "stray light". | ||
| Christina Timofeyenko (Ctimofeyenko) |
I am building a calibration in the NIR for a mixture of DI water, HF and NH4F and was wondering what preprocessing techniques would be advised for this system. I realize that preprocessing data is system dependent so I want to see if any one had experience with this type of system. Thanks. |