| Author | Message | ||
     Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 37 Registered: 9-2001 |
So far, mostly I have the test cases that I published, plus some other data sets that I've used here privately to test the software but didn't publish. Near as I can tell, the concept works fine. \o/ /_\ | ||
| Bruce H. Campbell (campclan) Moderator Username: campclan Post Number: 89 Registered: 4-2001 |
Howard, I had forgotten about it, at least in my conscience mind. Maybe my unconscience was working OK. But since I haven't applied that approach, maybe we would all like to hear how successfully it has been used. So could you tell us that? And if anyone else has used that approach, please let us know your results. Bruce | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 36 Registered: 9-2001 |
Bruce - you must've been at C'burg in 2002 when I presented that sort of approach all worked out. I've written it up since then, you can find it in J. Pharm. Biomed. Anal., 33, p.7-20 (2003). I also wrote it up in the Spectroscopy column earlier this year. I also implemented it in a software package (or is that getting too close to being commercial?) \o/ /_\ | ||
| Bruce H. Campbell (campclan) Moderator Username: campclan Post Number: 88 Registered: 4-2001 |
When we do a partial least squares calibration, we generally assume that the data are linear or very close to linear in all respects. However, we also know that strictly speaking, this is not always the case. Some non-linearity can creep in. Therefore, is there a way to discover this non-linearity and thereby take it into account? For example, why not use a set of least square equations that would successively try the linear case and the non-linear fit for each principle component to see which fits better? The process could be continued for fitting the residuals. For the non-linear fitting, the data could be transformed into a linear coordinate system, perhaps a log/log one. Ater the trial calibration is completed the computer program could show the best fit, followed by the next best and so on, together with the use of linear or non-linear fitting for each principle component. The experimenter could then at least gain understanding of the original data and make a choice as to whether to accept the calibration or discover why the data has a significant non-linearity and make corrections that would remove that non-linearity if possible. Doing this might help in deciding if one is incorporating a large amount of noise in the calibration. This is because pure noise is random and therefore linear fits would be better than non-linear. For example, consider the case where one is using 100 data points for each spectrum and there are 20 samples (a bare minimum) in the calibration set. Then there are 2000 points, each with noise. This is a large enough number to ensure pure random noise contributions would be statistically based. If the number of principle components is very large in the �acceptable� calibration and the best fit for the larger ones is non-linear, it could mean there is a non-linear non-random noise. Discovering the reason for the non-linear noise and eliminating it should vastly improve the calibration. Has anyone tried such an approach? Bruce | ||
| Donald J. Dahm (dahm) New member Username: dahm Post Number: 1 Registered: 1-2006 |
I wonder if Kate is asking her question in a less subtle form than the previous responants have assumed. I'm going to give a Level 1 answer. She wonders: Can "linear" methods such as PLS or PCR be used to fit data that shows a non-linear relationship between absorbance and concentration? The answer is: "Absolutely!" Almost all NIR data obtained from scattering samples, especially that obtained in Remission, has non-linearity in the plot of Absorbance vs Concentraion. In general, the high absorption end of the plot goes sub-linear (decreases in slope compared to the straight line). Sometimes the non-linearity is masked because of the contribution of other materials, the concetration of which is also changing. The power of factor based methods is that additional factors can be added to the model to fit this non-linearity. This is like fitting a curve by more and more straight lines having different slopes. This having been said, there is a danger that the extra lines added can wind up fitting errors in the data, as well as modeling the non-linearity. For this reason, one should throw out the regions of the spectra that have very high absorbance levels. | ||
| Forrest Stout (forrest) Junior Member Username: forrest Post Number: 9 Registered: 7-2006 |
I've had very mixed results with stepwise in the past because it relies on MLR results for its wavelength selection criteria, it often "overfits" its wavelength subset and potentially ignores many good wavelengths while including many that don't truly aid in prediction. | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 34 Registered: 9-2001 |
Eric - I feel I have to make one small correction to your comments: even with perfect linearity, PLS and PCR will require more than one factor if several constituents are verying independently, basically for the same reason MLR requires multiple wavelengths under those same conditions. Howard \o/ /_\ | ||
| Eric LALOUM (elaloum) New member Username: elaloum Post Number: 1 Registered: 5-2006 |
If there was a perfect linear relation between spectra and concentration, I would expect factor based methods to perform not much better than multiple linear regression with wavelenght selection (i.e. stepwise) ; besides theoretically PCR or PLS should led to only one factor (accounting for the linear trend). The fact that many factors are added in PLS or PCR is related to deviations from pure linearity. One subtle thing to point out is that it is not because a model is linear regarding its coefficients that it will not be able to represent non linearities... | ||
| Forrest Stout (forrest) New member Username: forrest Post Number: 5 Registered: 7-2006 |
What jrcuesta said, and: If you have subtle non linearity, methods such as PCR and PLS may show improvement if you augment your spectral matix (X) with nonlinear terms. E.g. Take your X matrix with dimensions of m (samples) by w (wavelengths) and add the squared or cubed spectra at the end to yield an m by 2w matrix. This gives the linear modeling method some terms to help account for the non linearity in the data. Sure, it's kind of a messy technique versus using other non-linear modeling methods, but it's worth a shot in some situations and may give the improvement you're looking for without straying from more familiar linear modeling methods. | ||
| jrcuesta (Unregistered Guest) Unregistered guest |
PCR and PLS are linear regression methods. In some cases, some kind of non linearities can be modelled with these methods, but in others not. There are other types of regressions as LWR and Neural Networks to treat non linearities. Residual Plots are a good help to find non linearities apart from the XY Plot. Regards J. R. Cuesta | ||
| Kate Smith (Unregistered Guest) Unregistered guest |
When performing calibrations using the factor based methods (PCR, PLS etc.) must the samples follow a linear trend when comparing, for example, absorbance vs. concentration or can nonlinearities (aside from nonlinearities do to noise, the instrument etc.) also be modeled? Thanks. |