| Author | Message | ||
     Forrest Stout (forrest) Advanced Member Username: forrest Post Number: 21 Registered: 7-2006 |
Regarding my post below, I forgot that I already started an EMSC thread, a while ago. Sorry about that, in advance. | ||
| Forrest Stout (forrest) Intermediate Member Username: forrest Post Number: 20 Registered: 7-2006 |
Is the step in this discussion EMSC and its variations? This is a technique that I want to like but in my personal experience, it has been more of a cosmetic fix with varied to marginal prediction effects. This is especially true with complex samples I have examined for which pure components are not readily available. | ||
| David W. Hopkins (dhopkins) Senior Member Username: dhopkins Post Number: 104 Registered: 10-2002 |
Howard, I found that I agreed with your argument regarding standard deviations. I figured that we would expect that MSC and SNV models would show the correlation for 1 factor models, but after that, maybe not. So I did some calculations with the IDRC-2002 shoot-out data with MSC and SNV. You were right, up to 1 factor. Both PLS and PCR (using no mean centering) showed perfect correlation of 1-factor models for MSC and SNV. However, with 2- or 3-factor models, up to the 15 factors I calculated, the perfect correlation was gone. But I did observe close correlations between the models. In fact, I observed that the correlations increased, up to r = 0.995 for 15 factors for PCR, and up to r = 0.994 for 12 factors for PLS. I was surprised that the correlations persisted up to so many factors, when the regression vectors were rather noisy, and I would not even use calibrations with that many factors. This suggests that the factors are highly correlated in the MSC and SNV models. I expect that the observation that the 1-factor models are perfectly correlated will apply to any data set. I also feel that the correlations will increase with increasing the number of factors employed in the models. I expect that the number of factors supported, and where the maximum correlation occurs, will depend upon the particular data set. Further, it makes sense that the maximum correlation occurred at fewer factors with PLS as compared with PCR. That is consistent with PLS concentrating the information correlated with the analyte in the earlier factors. Therefore, it appears that the observation stands, that both SNV and MSC should be evaluated to see which method might give better performance. But the exercise has given me better respect for the SNV method. I still think that the MSC method has better theoretical underpinning than SNV. Thanks for the observation regarding the relationship of the methods, Klaas. I have not had the opportunity to review the Dhanoa paper. I cannot locate my early volumes, but I am interested to see what their experience was with the extent of the correlation of the MSC and SNV regression vectors. Best regards, Dave | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 76 Registered: 9-2001 |
Hmmm ... I reread your message after posting the previous one. One of the points that stuck with me after I first started reading up on diffuse reflection theory was Kubelka's derivation that demonstrated that even to use his theory to extract the S and K values, you needed at least two measurements. Those could be the reflection at different sample thicknesses, or simultaneous measurements of both the reflectance (R) and transmittance (T). As a practical matter, nobody is going to take the time to measure a sample at different, carefully-controlled thicknesses (and degree of sample packing); neither is any instrument company going to produce an instrument to simultaneously measure R and T, it just couldn't be competitive. From the point of view of theoretical explanation, it might be interesting to see someone not concerned with commercial viability, say, someone in academia, to make that kind of concerted effort. But it hasn't happened in the 30 years I've been involved with this stuff, and I don't see it happenning now. \o/ /_\ | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 75 Registered: 9-2001 |
Don - I think that's about the best summary of the relationship between the effect of the particle size of the samples and their diffuse reflection properties, that I've seen since I first got involved in NIR. Unfortunately, the other side of that particular coin is that, except for your own work on representative layer theory, it also sums up in a nutshell the amount of progress that has been made on theories to explain diffuse reflection. Obviously, an exact theory describing the relationship between reflection properties of powders and particle size (and shape) distribution is dependent on having an exact theoretical solution; if someone came up with that then the effect of particle size, as well as the dependence on sample composition, would fall right out of the equations. The last person to seriously attack the problem, as far as I know, was Harry Hecht. He once gave a lecture about it at a Technicon Symposium back sometime in the early '80s, and I recall him stating that the only analytic solution was one that involved an infinite number of integrals. In fact, I think it was a triply infinite number of integrals. Sounds to me like it's just an intractable problem. You know better than me, how many people have worked on it since Schuster; nobobody's been able to crack this particular nut. And I'm generally not a negative person. \o/ /_\ | ||
| Donald J Dahm (djdahm) New member Username: djdahm Post Number: 3 Registered: 2-2007 |
Dave: Well, I certainly "feel" better about the Multiplicative Signal Correction than I do about Standard Normal Variant, probably for the same reasons you do. When I talked to Harald several years ago at PittCon, we discussed (he brought it up) MSC as a "particle size correction". He said he had never been satisfied with MSC, and believed that it needed work. (I'm not sure, but I think used the term "Multiplicative Scatter Correction" when he expressed his dissatisfaction.) I really haven't studied his recent papers on MSC, but my impression is that they are based more on Chemometrics than Scattering Theory. As Karl Norris pointed out a long time ago, the biggest single effect on a NIR spectrum is particle size. It bothers me that we haven't made more progress on getting a theoretically correct mathematical form to remove it. I have always assumed (without proof) that the reason that MSC works as well as it does is because it is an effective particle size correction at the lower absorbance regions of the spectrum. There are at least three things I can think of that could be considered. 1) When an individual sample is composed of a mixture of particle sizes, the spectrum is not the same as a spectrum of the mean particle size. It is more like the spectrum of the little ones. This effect is observed regardless of the absorption level. 2) At higher absorption levels, the effect of particle size on the spectrum is less pronounced that at lower levels. 3) At higher absorption levels, the suppression of the spectrum due to front surface reflection reduces the effect of particle size on the observed spectrum. I can suggest mathematical forms that may have the right shape to correct for these various effects, but I don't have a practical way to extract the exact shape information from a single spectrum. [email protected] | ||
| David W. Hopkins (dhopkins) Senior Member Username: dhopkins Post Number: 103 Registered: 10-2002 |
Don, The method would give perfect results if all samples were simply multiples of the same parent spectrum, because the plot of each sample vs the average scan gives a slope and interecept term that gives the multiplier and any offset that corrects the scan to the average intensity. The method works well when any chemical variations between samples account for a small percentage of the total absorbance, as in a series of wheat flour samples, or pharmaceutical mixtures of one or more actives in a background of excipients. I consider that good theoretical background, compared to the assertion that in a series of samples, the standard deviation of the absorbances should be constant. There is no natural law there. It just works out that way, assuming the same conditions outlined above for the MSC. Wouldn't you agree that that is a theoretical background? It's not on the level of Beer-Lambert, but it is grounded in spectroscopy, which I believe SNV is not. Best regards, Dave | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 74 Registered: 9-2001 |
Don - I don't know of ANY theoretical (or should I say "any THEORETICAL") justification for MSC. If there is, it's statistical theory, not physical theory - but I don't have to tell you that. It was simply presented many a moon ago (probably about 150 moons) by Harald Martens, as an empirical attempt to remove the effect of scatter from the data. Since nobody know how to deal with it physically, he just picked a function that appeared to help by making the spectra sort of coincide. \o/ /_\ | ||
| Donald J Dahm (djdahm) New member Username: djdahm Post Number: 2 Registered: 2-2007 |
Dave Hopkins (or anyone else): What would you say is the theoretical justification for MSC? Don { [email protected] } | ||
| venkatarman (venkynir) Senior Member Username: venkynir Post Number: 37 Registered: 3-2004 |
Hi David, Basically SNV and MSC differ in measurments .My work experience NIR Instrument and modeling ,SNV better in on-line measurments .The varaibles are normalized .In MSC the correction factor dependent one. Venkat | ||
| Klaas Faber (faber) New member Username: faber Post Number: 4 Registered: 9-2003 |
Howard, Methods like PCR and PLS are not scale invariant like OLS. With OLS you get the same results whatever value for the constant variable. With PLS, for example, you can add a constant column to X but it will only be equivalent to mean centring when it approaches infinity. That could explain the differences in results. Klaas | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 73 Registered: 9-2001 |
Klaas - you know, that's something I never understood. Theoretically, I think you SHOULD get the same results. In the case of SNV, the mean of the data set and each spectrum is zero and the SD of each spectrum is one. In the case of MSC the mean of the dataset and each spectrum is [whatever the original mean of the data set was] and the SD of each spectrum is also equal to some constant. So there should be a one-to-one linear relationship between the data sets after the two transforms, and the results should be identical, except for the magnitudes of the coefficients. But they're not. What am I missing? \o/ /_\ | ||
| David W. Hopkins (dhopkins) Senior Member Username: dhopkins Post Number: 102 Registered: 10-2002 |
Hi Klaas, That is interesting. What it means is, if you perform a MSC and SNV on the same data set and over the same wavelength region, the wavelength by wavelength plot for each sample of the results for the SNV transform vs the MSC transform show a straight line. The slopes and intercepts vary from sample to sample, and will depend on the absorbance range of the original data. Unfortunately, this relationship does not mean that regressions using MSC or SNV should give the same results. It is the consistent observation of many researchers that sometimes MSC gives better results than SNV, and sometimes SNV gives better results, and it may be good to check both methods. I have to say, I prefer the theoretical justification of MSC, and would like to choose it all the time. The only justification for SNV is, it works. Both methods work best with careful optimization of the wavelength interval employed. Best regards, Dave | ||
| Klaas Faber (faber) New member Username: faber Post Number: 3 Registered: 9-2003 |
Hello, Perhaps it is interesting to note here that SNV and MSC are linearly related, as proved in Dhanoa et al. J. Near INfrared Spectroscopy, 2 (1994) 43-47. Klaas | ||
| David W. Hopkins (dhopkins) Senior Member Username: dhopkins Post Number: 100 Registered: 10-2002 |
Hi Benoit, Thanks for sharing your results with us. That is a textbook example of the value of pretreatments! How close to the time of measuring the samples were the reference values measured? If you use a good oven method, I would expect you should achieve a SEP value of 0.1% or less! Moisture is the easiest and the hardest parameter to measure by NIR, because of the precautions you need to take. Nice work. Best wishes, Dave | ||
| Benoit Igne (benoit) New member Username: benoit Post Number: 5 Registered: 11-2006 |
Dave, here are the SEP of the different methods I used. I know it is a little bit late but this semester, professors like to give homework and the snow makes everything slower!!! PLS - moisture - SEP (validation 2006, calibration 2003 - 2005). Autoscaling : 0.18% SNV without detending method: 0.37% MSC without detrending method: 0.42% 2nd deriv (15) + SNV: 0.19% 2nd deriv (15) + MSC: 0.21% The size of the calibration set is pretty reduced so SEP will be improved with next years samples. And as you mentionned, "2nd deriv + MSC" allows a reduction of the number of PCs (2 fewer than other preprocessing techniques). Thank you for your help, Benoit | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 65 Registered: 9-2001 |
Benoit - and also because the older instruments seem to be lasting practically forever! I have one that's about 30 years old and is still chugging along great. In fact, in some ways it seems better than new: it meets its noise spec with the lamp control set to Low, and the noise spec was intended for High lamp energy! \o/ /_\ | ||
| David W. Hopkins (dhopkins) Senior Member Username: dhopkins Post Number: 99 Registered: 10-2002 |
Hi Benoit, How about MSC? And how much different are the SEP values compared to no pretreatments? Thanks for letting us know the results you found. Usually I find that 2Der + MSC allows the use of fewer factors to obtain the same accuracy of calibrations. Therefore, the argument is that the calibrations would be more robust. P.S. Glad to hear that you are sending us colder weather, it is unseasonably warm in Battle Creek, above freezing is not welcome to the farmers or skiers. Best regards, Dave | ||
| Benoit Igne (benoit) New member Username: benoit Post Number: 4 Registered: 11-2006 |
Dave, I used 2nd derivative followed by SNV and it improved greatly the results. Even if I did not reach the performances I got without using any pretreatment, I learned a lot about the use of SNV/MSC. To comment Kenneth's post, I would say that I agree with him. However, if these obsolete technologies are still used, it is because they meet the needs of most of their users. Have a good but unfortunately cold weekend! Benoit | ||
| David W. Hopkins (dhopkins) Senior Member Username: dhopkins Post Number: 98 Registered: 10-2002 |
Ken, Cheap shot. In my opinion, you should have said nothing. Of course, I think you are entitled to your opinion. | ||
| Kenneth Gallaher (ken_g) Junior Member Username: ken_g Post Number: 6 Registered: 7-2006 |
"What I can say is that the instrument is worlwidely used for agricultural products and I do not know any lab dealing with ag products that do not own one! (hint: it is from a European company)" And what I can say is that because of history a lot of the NIR community is stuck in the past and is using hardware that is technologically obsolete. | ||
| Benoit Igne (benoit) New member Username: benoit Post Number: 3 Registered: 11-2006 |
I do not want to give the name and brand of the instrument. What I can say is that the instrument is worlwidely used for agricultural products and I do not know any lab dealing with ag products that do not own one! (hint: it is from a European company) I will read the paper Dave mentionned and try using derivative + SNV. I will let you know the results. Thank you Benoit | ||
| Bob Rosenthal (rosenthal) |
Venkynir, If you contact me on my private e-mail ([email protected], I have some information you might be interested in. Bob Rosenthal | ||
| David W. Hopkins (dhopkins) Senior Member Username: dhopkins Post Number: 97 Registered: 10-2002 |
Hi Benoit, The average scan looks nice, there should be good calibration success. I think the problem is that there is a profound slope upward toward the visible between variables 1-50, and apparently flat baseline from 50 to 100. There is a lot of color variation here that is not related to moisture or protein content. I suspect that it is the variation in slopes between variables 1-50 that is giving difficulties to both the SNV and MSC methods. Usually with SNV you should use a detrend treatment too. It would be worthwhile for you to review the original paper, R.J.Barnes etal, Standard Normal Variate Transformation and De-trending of Near-Infrared Diffuse Reflectance Spectra, Appl. Spec 43: 772-777, 1989. I suspect that you could get good results by using a 2nd derivative pretreatment before performing either SNV or MSC. It appears to me that a 15 to 21-point Savitzky-Golay 2-Der would be effective. The 2-Der would serve as a great detrend procedure, and sharpen the bands too. The 2-Der removes the offset and curvature variations caused by the variation in light scattering, but it cannot remove the multiplicative effects, so the use of 2-Der followed by either MSC or SNV can be highly effective. I think this would work in your case. The derivatives and MSC and SNV have been discussed in other threads. You might like to search on these topics on the discussions. I hope this helps you. Best regards, Dave | ||
| David von Boisman (david_von_boisman) New member Username: david_von_boisman Post Number: 3 Registered: 2-2006 |
Hi, I don�t know what instrument you have and how much smoothing etc is done before the scan is presented. But if you have a good transmission instrument then you should really not need any pretreatment for wheat in transmittange, particulary not for moisture which has a very strong signal in the spectra. SNV is very useful for reflectance measurements but I�ve newer seen anybody use it in transmittance. MSC, on the other hand, may help in reflectance for stuff like powders or meat. What instrument are you using? | ||
| venkatarman (venkynir) Senior Member Username: venkynir Post Number: 33 Registered: 3-2004 |
Hi Benoit, What David said is correct.Usually SNV gives better results and repeation factor will be good. I have doubt about your wave length region .Can we have look of the spectral data with %moisture variation . | ||
| Benoit Igne (benoit) New member Username: benoit Post Number: 2 Registered: 11-2006 |
Dear Dave, the instrument is a transmitance unit with a range between 850 and 1048 nm at 2 nm interval. I used the entire range (100 data points). The calibration process is pretty usual (MATLAB + PLS toolbox). The calibration set contains 400 samples, outlier detection by Hotelling T2 / Q residual. I joined a plot of a raw spectra. I have been observing the same phenomenon when developing protein calibration with the same spectra: no preprocessing/derivative/OSC worked fine, SNV and MSC gave less precise results. These last results were presented at Chambersburg. Thank you for your help, Benoit  | ||
| David W. Hopkins (dhopkins) Senior Member Username: dhopkins Post Number: 96 Registered: 10-2002 |
Hi Benoit, I usually have great success with moisture calibrations in wheat, and MSC and SNV usually work fine, so it is hard to understand your bad results, as you say. Can you tell us more about what you did? What wavelength region are you using? I would not recommend using the entire wavelength region that is available in some instruments, because too much U-shaped curvature is hard for either SNV or MSC to handle. Can you upload a plot of the raw data for us? Use the instructions in the panel on the left at the website. Best regards, Dave | ||
| Benoit Igne (benoit) New member Username: benoit Post Number: 1 Registered: 11-2006 |
Hello, I am developing some wheat moisture calibrations and I am facing a problem with some preprocessing techniques. While no preprocessing/derivative/OSC work fine, when I use SNV and MSC, I get very bad results (r2 = 0.5 in calibration). When I increase the number of sample in the calibration set, SNV and MSC tend to work a little bit better but far behing other pretreatment methods. Does anyone know a reason for that? By experience, I know that SNV and MSC do not work all the time very well but I never had such low correlations! Thank you Benoit |