| Author | Message | ||
     Mulualem Tigabu (Muti) |
Dear experts, I measured NIR spectra from single seeds with NIRSystems Model 6500 spectrophotometer using a modified sample cup. The modification was made by inserting a black metallic ring with a narrow window that nearly fits the size of the seed. Unfortunatelly, the spectra appeared noisy. To smooth the spectra, I applied the moving average filtering facility available in the Unscrambler software with three and ten segments. The latter treatment resulted in a nice spectra. The smoothed spectra were further subjected to orthogonal signal correction and the resulting model was excellent. For curiousity, the raw spectra were also pretreated with OSC without smoothing. Although the model was found similar with the one above, part of the spectra was still noisy. My concerns are how many segments should be used in smoothing by the moving average method? and is it computationally correct to apply smoothing and pretreamnets like OSC? I would also appreciate your suggestion for relevant literature on smoothing by moving average method. With kind regards Mulualem | ||
| hlmark |
Mulualem - it would also help if you increase the number of scans for each spectrum above the normal default value of 32. How much to increase it will be a matter of trial and error, but you should try to take as many scans as you can spend time to collect. The default settings are based on the assumption that you're using the entire illuminated area for reading the sample. Since in your case, only a small fraction of that area is returning signal, it reduces the S/N ratio considerably. Increasing the number of scans will help restore some of that S/N at the source. However, since the S/N improvement only occurs as the square root of the number of scans, you probably can't restore the spec value entirely, by this means. But it should help, and reduce the amount of smoothing (which is only another way of averaging the data) that you need to do. Howard | ||
| Bruce H. Campbell (Campclan) |
Mulualem, A rule of thumb in smoothing is to use a segment width of one about equal to the half width at half height. As many bands in NIR have half widths of about 10 nm, a 10 nm segment usually works as well if not better than other segment widths. Bruce | ||
| Mulualem Tigabu (Muti) |
Howard and Bruce Many thanks for your valuable suggestions. God bless you all. Kind regards! Mulualem | ||
| Fernando Morgado |
Dear Mr. Mulualem : I supose you check the instrument before take the spectrum and the specifications are correct ( noise near 20 RMS). You don´t indicate wich accesory are using if transport or spining Ring. If you are using spining remember this accesory rotate the sample with excentric movement. If you are measuring a single seeds you need work without movement. If you are working with transport accesory is o.k. I make a single seeds measuring using other instrument in the reflectance mode and the result spectra are repetitives and without noise in the NIR Region. I think you probably have problems in the position of the sample ( considering your instrument work O.K.)I recomend always before make some experiment check the Instrument using the diag service option disponible in the NSAS diagnostic software. For me increase the scan in this aplications is not necesary because your instrument would take good spectra with normal number of scaning ( 16-32-16)if the sample is correctly iluminates using the cups mention for you. Fernando Morgado | ||
| Christopher D. Brown |
Just as an aside, the appearance of "noise" in the spectra shouldn't dictate the application of a smoothing filter. In many many situations employing multivariate calibration, smoothing might be aesthetically pleasing, but it is deleterious to the performance of the calibration model. I published a paper back in my grad school days which proved that smoothing only degrades the multivariate signal-to-noise ratio for quantitative models. The implication is that smoothing is only useful (from the performance standpoint) if you're _very_ limited in the size of the calibration set. And the further result is that _if_ smoothing appears to help in cross-validation/validation, it implies that your calibration model is unstable, and better performance (over and above what you see with smoothing) is to be gained by collecting a few more calibration samples. (Of course, sometimes it's out of the question to collect more calibration samples.) ~ Chris | ||
| Bruce H. Campbell (Campclan) |
I think it was Howard Mark who first said to obtain your best calibration and then add a little noise. (If he wasn't the first, he at least was the one I first heard it from.) This is meant to increase the robustness of the calibration. I do have a question for Chris, and that question is: Did smoothing make the detection limit worse or not? Bruce | ||
| Christopher D. Brown |
Bruce, if I understand the context of your question, the answer is that we didn't explicitly deal with the LOD in the paper, but given that the LOD is a simple extension of the multivariate SNR, and we found that the multivariate SNR always degrades with smoothing, I would infer that the LOD always degrades. As I said above, however, if you're really short on calibration samples, then smoothing can buy you a little performance, and so most likely in this situation it would buy you a bit on the LOD as well. You still can't get to where you would be with more calibration samples, though. ~ Chris | ||
| hlmark |
Bruce: I don't recall our discussing this point, however I may have mentioned something to this effect, since I do recommend adding noise under the appropriate conditions. However, a situation that is (or may be) ALREADY noise-limited is NOT one where I would want (or recommend) to add more noise. If I made that recommentation, it would have been (as it would now be) for a situation that has S/N to spare, but has too few samples, and the intent is to try to "stretch" the avialable samples as far as possible. This would lead (hopefully) to a more robust calibration at the expense of a (again, hopefully) slight loss of accuracy. The amount of accuracy loss would then depend on the amount of added noise. But it is definitely not something I would recommend for this situation. Howard \o/ /_\ | ||
| Bruce H. Campbell (Campclan) |
Howard, In reading your reply, I realized I should have been more explicit. Adding noise can make a calibration more robust, but the primary aim is to have an "acceptable" calibration. Thus, when one has a calibration with measures showing a better than acceptable predition, such as a very good RMSEP, then one can add noise. In this case, one would have to monitor the RMSEP, or a similar measure, to ensure the acceptability of the calibration is not lost. Bruce | ||
| hlmark |
Bruce - I think we're on the same wavelength (if you'll forgive the expression!) there. I would only add that, since noise generally doesn't help, there has to be a benefit gained that more than makes up for the harm that the added noise creates. The only one I'n come up with is that,as I said, if you're short on samples and long on S/N, then for every actual sample you've got you can create 2 or 3 more from each one by adding different noise values to the new ones. Hopefully, the added samples will increase robustness more than the standard errors. Also, if my memory serves me correctly, I think I've applied it more to cases of qualitative analysis (indentification) situations rather than quantitative analysis. Actually, I just thought of one more case where I've used it (and seen it used in the literature): in verifying theoretical equations, I've taken several spectra of the same sample, averaged them together to reduce the original noise content as far aa possible, and then added artificial noise back in. The pupose of doing that was to KNOW exactly how much noise the spectra contained, so that you could compare the results with theoretical predictinos. Howard \o/ /_\ | ||
| hlmark |
Bruce - extra note: Rocco diFiggio worte a couple of really super papers using this technique \o/ /_\ | ||
| Alejandro |
Hi my name is Alejandro I thanks everybody for the oportunity. I´m new in the NIR technology but I´ve been doing a lot of work in this little time. I have a problem measuring powder detergent because near the 1900 band I have the moisture and phospate, how can i do to avoid interference of moisture, so i can measure the phospate? | ||
| Alejandro |
Hi my name is Alejandro I thank everybody for the oportunity. I´m new in the NIR technology but I´ve been doing a lot of work in this little time. I have a problem measuring powder detergent because near the 1900 band is the moisture and phospate absortion, how can i do to avoid interference of moisture, so i can measure the phospate? |