| Author | Message | ||
     Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 283 Registered: 9-2001 |
Jose (both Jose's in fact) and Jerry - if you want to understand this stuff you need to learn what goes on "under the hood". To do that, forget the chemometrics; you need a good course (at least college-level) in Statistics; the relationships between the various calibration statistics are based on statistical operations like Analysis of Variance and other surprisingly simple concepts, that we otherwise never learn about. \o/ /_\ | ||
| Jos� Antonio Cayuela S�nchez (joseacayu) New member Username: joseacayu Post Number: 3 Registered: 11-2009 |
Howard, Jerry and Jos� R., I belief H. Mark hit the truth of the matter, this effect should be due to a narrow range. I sensed some of it, but I could not saw clearly that the effect was due to this cause unequivocally. Thank you sincerely, Jos� A Cayuela | ||
| Jos� Ram�n Cuesta (jrcuesta) New member Username: jrcuesta Post Number: 2 Registered: 11-2009 |
Hi Jose, my greetings too. I saw some cases where for the RSQ for validation (1-VR) gives strange values during the cross validation calculation, even getting negative values for some of the groups. When a group is out for cross validation the samples than stay for the calibration should cover the range and variation or the samples which are out. Samples which are out must not be unique, and should be represented by some others. This problem can brings good RSQ and SECs but poor SECV and 1-VR (Val RSQ). Sometimes it is necessary to take out some ouliers (unique samples) until more of the same type appear to improve the cross validation values. Best Regards Jos� Ram�n Cuesta | ||
| Jerry Jin (jcg2000) Intermediate Member Username: jcg2000 Post Number: 20 Registered: 1-2009 |
Hi Jose, You get good R2 and/or root mean square error (RMSE) when you try to validate the PLS model by using it on the same calibration data you used to derive the PLS equation. You get poor result when you apply cross validation to the model. Actually you reach the crux of model validation. If the error (true-estimate) is obtained by testing the calibration equation directly on the calibration data, a chance is that you will get a good result, and the R2 increases with the number of factors in the model. The problem with this direct validation is that the error is actually an estimate of model error, not the prediction error. According to Tormod Nas and Tomas Isaksson's text theory (Multivariate calibration, NIR publications), prediction error includes model error and estimation error. The latter is not accounted for in the direct validation, so the R2/RMSE based on direct validation is over-optimistic. That is why many people argue that a proper model validation should use a second validation data set or testing set. That's said, the direct validation is still useful if you care only about the modeling itself, instead of using it for prediction. Cheers! Jerry Jin | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 282 Registered: 9-2001 |
If the SEP and RMSEP are about the same for the validation data as for the calibration data, then a small R^2 indicates that the RANGE or the distribution of the validation samples is too small. The number of samples has relatively little to do with it except indirectly, insofar as as larger validation set is more likely to have a larger range, also. \o/ /_\ | ||
| Jos� Antonio Cayuela S�nchez (joseacayu) New member Username: joseacayu Post Number: 2 Registered: 11-2009 |
Hi, Jos� Ram�n, The instruments used were a Labspec (ASD, Inc.) and a Luminar 5030 (Brimrose Corp.). As you know, I have also used an InfraXact (Foss) in several works. The oilseed is argan seed (Argania spinosa), and the parameters measured are humidity, oil content, and fatty acids (oleic, linoleic, palmitic, etc.). I think the problem is not overfitting or underfitting, as I tested with different numbers of PCs with very similar results. I know RSQ is not the best statistic; SEC coefficients were also good. But the truth is I do not understand the mechanism that makes possible to obtain an outstanding result in the calibration R2 and SEC, being bad the same statistic (R2) for the cross-validation. I also know the best way to assay the goodness of a calibration is the external validation. In this case, the calibration set was small too much to do it, although with Luminar is possible try it (55 samples). Moreover, despite the interest of parameters such as RMSEP or SEP/sd, in my opinion it would be much more clear to express the results as 'mean accuracy', that is, the mean and standard deviation of the differences in each sample of the external validation set between the predicted value and the reference, expressed in percentages Thank you very much and greetings, Jos� A. Cayuela | ||
| Jos� Ram�n Cuesta (jrcuesta) New member Username: jrcuesta Post Number: 1 Registered: 11-2009 |
Hi Jos� Antonio, I think that more details are needed in order to give you some reasons for that. Which instrument are you using?.Population statistics. Parameters measured. Types of oilseeds. ....etc. It is important the number of terms used in the calibration, because there is always a risk to have underfitting or overfitting. RSQ is not the best statistic to trust, its value can be close to 1 but the SECV or SEP can give poor results. Best regards J.R.Cuesta | ||
| Jos� Antonio Cayuela S�nchez (joseacayu) New member Username: joseacayu Post Number: 1 Registered: 11-2009 |
Hello, and thank you very much I have a little previous experience on development of NIR calibrations for intact fruit quality prediction. At present I'm working in the prediction of quality parameters of some oilseed, and I'm not sure understand the results. On one hand, for some parameters the calibration coefficients indicate clearly a good predictive potential. On the other hand, for other parameters clearly the predictions are unsuitable, both calibration and FCV plot lines being horizontal. If we can have another hand, on this we have for other parameters very good calibrations, with R2 values 0.95 to 0.98 but R2 for the full cross validation from 0.35 to 0.45. In my opinion the cause of this result is the small size of the calibration set, at present 44 or 55 samples (depending on the spectrometer, since two differents instruments were used). Also, probably due in part to insufficient homogeneity in the distance from the spectrometer to the sample in the acquisition of spectrum in different samples I would like know your opinion, that would be very appreciated Thanks again, Jos� A. Cayuela |