| Author | Message | ||
     Sheelagh Halsey (Sheelagh) |
I have some spectra of aqueous solutions, some referenced with air and some with water. Calibrations work better with the air reference. I think it is to be expected since large signals are being substracted, plus there is the possibility of the water peak shifting with analyte concentration. Has anyone seen any references on this issue to justify my anecdotal evidence? | ||
| DJDahm |
I have no references to offer, but, not surprisingly, I have an opinion. First of all, when we use an absorber as a reference, the problem is not exactly the same as the statistical one of "small differences in large signals". With absorbers, we are counting smaller numbers of photons, not larger. If we convert to absorbance, and take log (ref/sig), we have two small signals in the case of large absorbers. For example, with an absorber we might have log (100/81) with an absorber, instead of someting like log(10,000/81) for air. Using the sqrt for counting statistics, we have 100+-10 versus 10000+-100 for the mean and uncertainties in the reference. Of course, we would rather have a reference with 1% variation instead of 10%. Second, I don't understand why anyone would ever want to use matrix referencing. If there is something to be gained by subtracting the water signal, wouldn't it be statistically better to take one water spectrum against the air reference, and subtract that spectrum from all the other specta, thereby eliminating the effect of the variation in "recounting" the water signal. My questions for the rest of you are: Is there a statistical advantage in regression analysis to using spectra from which the matix has been subtracted? Do we use matrix referencing because we enjoy seeing apparently bigger signals from the analyte pop up on the screen as we collect data? Is matrix referencing just a holdover from the days when we didn't have computers to do the spectra subtraction easily? If there is a real advantage to spectral subtraction before we do our regressions, shouldn't the instrument manufaturer/software folks make that as easy as matrix referencing? Don Dahm | ||
| Richard Kramer (Kramer) |
One good reason to use matrix referencing is in the case where there are significant non-linearities due to the matrix. In such cases it can be very helpful to use matrix referencing to remove the non-linearities rather than model them. | ||
| Richard Kramer (Kramer) |
One situation where matrix referencing is useful is when there are signigicant non-linearities due to the matrix. It can be advantageous to use matrix referencing to eliminate the non-linearities rather than try to model them. | ||
| hlmark |
Don - in NIR the limiting noise source is generally NOT counting statistics, which would give a Poisson distribution to the noise and allow the sqrt (signal) condition. The limiting noise source is the (constant) detector noise, which is due to several effects: Johnson (thermal) noise, recombination noise, and a couple of other more minor phenomena. Over the past few months (since about last October) I've been publishing a fairly thorough analysis of the effect of constant detector noise in the columns in Spectroscopy - that's the US Spectroscopy, for our international brethren. The analysis considers both the large-signal and small-signal cases and the properties are quite different from those of counting statistics, so Sheelagh might want to see which applies. In another column or two we'll be getting to a similar analysis of Poisson noise, but in any case that is not generally operative in NIR. Sheelagh - However, it is really rather rare for results to be limited by detector noise. More commonly, the limitation is what can be loosely called "sample noise" of various sorts. For clear solutions, which is what the question seems to be about, there are not too many, but temperature changes (especially in water), interactions between solvent and solute or between different solutes, can all cause problems. Mechanical instability of the instrument and/or sample cell are also possibilities. Also bubbles? Could a reaction be occuring while you're trying to measure? When using the water reference are you sure that the water has completely warmed up to ambient temperature? That's all I can think of right now, but my inclination is not to look at the limiting noise - when problem occur you're usually 'way above that. Also, to add to Richards comments about matrix referencing: quite contrary to Don's statement, use of matrix measurements became common and important only when the mathematics needed to distinguish the sample "signal" from the matirx "background" could be easily and inexpensively implemented in computers. This is what led to the (now well-known) ability of NIR to measure natural products, in which the sample and matrix cannot be separated. I suppose the reasons it is still universal is that the tools are available to do that easily, and people resist changing to a new and unknown paradigm. But changing it also means that a lot of problems that the math automatically corrects for (or "sweeps under the rug" in another view) have to be dealt with explicitely. Howard | ||
| DJDahm |
I'm out of my area here, but I still didn't see an answer to my question as to "why is it not preferable to subtract the water spectum rater than to use water as a reference?" Don | ||
| hlmark |
Don - well, there are probably cases where it wouldn't make a difference. However, the reasons that come to mind are the ones that make us use chemometric algorithms in the first place: 1) The algorithms give the best fit to the data and variability thereof without regard to the causes of the change. This is not good for understanding the underlying fundamentals, but gives the best analytical results in practice. 2) The spectra are often not known as well as would be needed. Even with water, the spectrum of water as it exists in the samples is likely not the same as the spectrum of pure water, due to the changes attendent upon the interactions of the water molecules with the solutes. Hence subtracting the spectrum of pure water is not the proper correction. 3) If the spectrum changes with the nature and amounts of dissolved materials, then even if you know how it changes with the solutes, you need to konw the composition of the sample in order to calculate the spectrum of the water as it exists in the mixture - which is what you're trying to find out in the first place - you run into the trap of circular logic - or "catch 22" as it's also called. 4) Ditto temperature effects. Howard | ||
| Shihying Chang |
By doing the matrix subtraction we assume that the linear addition (and multiplication) rule exist between constituents and their matrix (solvent). ShihYing Chang | ||
| hlmark |
Well, assumptions like that are ALMOST true, and that's why NIR works. But it works better when you subtract the actual (but unknown) spectrum empirically, rather than an approximation to the actual spectrum based on something you measure under different conditions (such as the spectrum of pure water, and then subtract that from the spectrum of a solution). Tomas Hirshfeld measured the salinity of seawater by the shifts induced on the water bands by the dissolved salts, even though they have no spectral absorbance bands themselves; this proves that the shifts are real. Howard | ||
| DJDahm |
If we start by assuming that we are working with absorbance as we tend to in NIR, then it seems to me that: 1) If there are no problems to ecounter, there is essentially no difference between using the water (or other matrix) for a reference and subtracting a water spectrum for the data EXCEPT FOR STATISTICS. I assume that the statistics are better for the subtraction, but as I say, this is not my area. 2)) There are times when problems are encountered. There could be times, like the one Richard cited, when subtraction or referencing will help the regression process. There may be times (such as the peak shifting example) when spectral subtraction (or matrix referencing)might acutaully hurt the linearity of the analyte spectra. 3) In the face of uncertainty, it is unwise to collect data with a reference other than air, (unless you know that's the best way for you particular analysis). If you collect with a water refereence then you need to recollect if it doesn't work out. With an air reference and a water sample spectra, you can choose to include the subtraction or not, as you go along with the method development. With the separate spectra you can scale the spectra before subtraction, if you desire. I'm not arguing for subtraction in general. I tell my consulting customers to avoid matrix referencing, and to always try a calibration without matrix subtraction as a benchmark, if they are drawn toward matrix subtraction for a particular analysis. I have been given no reasons by anything that has been said to change my mind. Am I missing someting? Don | ||
| Karl Norris |
A major requirement of a reference is stability. The reference should be the same every time or you will add systematic noise to your data. Water is not a stable sample, so I would not choose it as a reference. If you think water is stable, prepare two water samples, and use one as a reference and the other as a sample. The resulting sample spectrum will be an example of the noise that would be added to your data. | ||
| Michel Coene (Michel) |
If reference and sample are (re)collected at the same time, does this not remove some instrument/sample instabilities? Imagine you measure through fiber optics and a transmission cell. If you place a second cell filled with ulrapure water next to the sample cell, they will both go up and down in temperature at the same time. Idem for temperature shifts in the absorbance by the fiber, declining intensity of the lamp with age and some other low-frequency noise. If you measure both simultanuously (true dual beam), even higher frequency noise could get cancelled. Or am I just dreaming here? |