| Author | Message | ||
     Gavriel Levin (levin) Senior Member Username: levin Post Number: 26 Registered: 1-2006 |
Hi guys, The usual procedure in the Unscrambler is to run the regression with cross validation activated. Then you can choose how many samples you want to leave each time an iteration is run - 1, or 2, or 3 or as many as you wish. Naturally, the number of samples you leave out in each iteration of the cross validation depends on the size of the sample set. In small sets, below 60 I would probably recommend 1, in largers sets I usually do 2 and then with higher sets more samples. Because this is done randomly, the final SEP (which is the most important parameters that comes out of it, could slightly vary if you run the same set with 1 or 2 or 3 left out with each iteration. Once you have a reliable basic model, you wish to truly test the SEP that can be expected in the real continuing implementation on line - by actually predicting samples that were not part of the original set used to calibrate. This gives a better feel for the error to be expected in predictions. My experience with many applications, both in the pharma, chemical, polymer, food applications etc. that the SEP so determined is always larger than that indictaed from the initial cross validation obtained for the initial calibratrion set. I hope this contributes some. Gabi Levin Brimrose | ||
| Michel Coene (michel) Senior Member Username: michel Post Number: 41 Registered: 2-2002 |
I meant balancing good predicting capability and the correct estimate of your predicting capabilities. "Leave one out" is an example of such a strategy. Say you have a batch process, you might want to leave complete batches out, to avoid the correlations within a single batch. Models with seasonal influences (butter) might require yet another strategy, based on what data you have available. | ||
| Antoine Cournoyer (antoine_cournoyer) New member Username: antoine_cournoyer Post Number: 2 Registered: 9-2006 |
Thanks to all, you confirmed what I though, there is difference between each software used. But still in general, it is correct to say that RMSECV=RMSEV=RMSEP and that it is different from RMSEC. In a visual way, RMSEC represents the difference between each points and the curve used to fit the data (when building calibration model) and the RMSECV is the value calculated when a leave-one-out is done. Also, Michel, you've talked about routines used to calculates the best "splitting" possible for calibration set and validation set, do you have any reference on that ? Where could I find those ? Thanks again, Antoine | ||
| Tony Davies (td) Moderator Username: td Post Number: 133 Registered: 1-2001 |
Lez Dix, Yes it would normally but NOT in this case! Tony | ||
| Lez Dix (lez_dix) New member Username: lez_dix Post Number: 2 Registered: 10-2006 |
as an add on. Usually when a C is added to V or P ie Cv it normally refers to a relative error as opposed to an absolute error. eg an SEV of 0.01 on a result of 1 would be a Cv of 1% ie 1% of the absolute result. I think! | ||
| Tony Davies (td) Moderator Username: td Post Number: 132 Registered: 1-2001 |
Antonie, Just to clarify, David lost a "C" in the RMSECV abbreviation, which stands for root mean square error of cross-validation (also known as "leave-one-out"). In this terminology we sometimes use "V" for validation or "P" which might mean prediction or performance. They are all the same thing (or they should be!). If we just say RMSEC then the C stands for calibration. I am sorry that it is so confusing but we have to live with it. Best wishes Tony | ||
| David Russell (russell) Junior Member Username: russell Post Number: 29 Registered: 2-2001 |
To answer the original question, RMSEC is almost always the error calculated using the calibration set objects only. The use of RMSEV or RMSEP implies that your are either generating the statistics on new objects or some that were left out of the calbration either as a separate test set or through the use of a cross-validation scheme. | ||
| Michel Coene (michel) Senior Member Username: michel Post Number: 40 Registered: 2-2002 |
The difference is how you calculate the error between predicted and reference. If you use the same samples which are in your calibration to validate your model then you "cheat" and will get a optimistic estimate of the models performance. Splitting up your samples in 50% calibration and 50% validation will improve your accuracy estimate... but decrease your model! So many routines exist which try to find a balance. | ||
| Antoine Cournoyer (antoine_cournoyer) New member Username: antoine_cournoyer Post Number: 1 Registered: 9-2006 |
Dear all, This is really enlightening for me. I'm studiyng chemometrics for a couple of month and sometimes things are confussing. I was wondering what is the difference between RMSEP and RMSEC ? Also, I often see RMSECV and RMSEV... I'm not quit sure about the difference. I've seen it could be of different signification from the software used, each have a different "wording", am I right ? Could someone help me out with this ? Thank you, | ||
| Forrest Stout (forrest) Intermediate Member Username: forrest Post Number: 16 Registered: 7-2006 |
Pedro, here's a few recommended references dealing with variance discussions: Faber K, Kowalski BR. Propagation of measurement errors for the validation of predictions obtained by principal components regression and partial least squares. J. Chemometrics 1997; 11: 181. Forrester JB, Kalivas JH. Ridge regression optimization using a harmonious approach. J. Chemometrics 2004; 18: 372. (Good analysis of variance equations and how to incorporate variance indicators in model selection.) Additional valuable references are cited in these papers. I hope this helps address some of your concerns. | ||
| Pedro Castro Nunes Fiolhais (pedro_fiolhais) New member Username: pedro_fiolhais Post Number: 3 Registered: 9-2006 |
Hi Forrest, This is exactling what I was looking for "(how close is the prediction to the "true"/reference value?). Kowalski and Faber have published work on variance calculations for multivariate modeling variance." can you give me the references? | ||
| Forrest Stout (forrest) Member Username: forrest Post Number: 14 Registered: 7-2006 |
Pedro, I concur with your observation that often times RMSEP is treated as the single value for measuring model performance. You may be looking for a variance or precision measurement (~reproducibility of prediction) to accompany RMSEP, which is primarily a bias or accuracy measurement (how close is the prediction to the "true"/reference value?). Kowalski and Faber have published work on variance calculations for multivariate modeling variance. (I can get you the references if you're interested.) These equations lead to looking at measurements such as the regression vector 2-norm (Euclidean norm), leverage, etc. for indications of prediction variance/precision. Also of interest is the effective rank or degrees of freedom of the model. When evaluating the performance of a model, it is often essential to look at such model performance measurements, in addition to the RMSEP to get a more complete picture of model performance. | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 49 Registered: 9-2001 |
Pedro - I think it's a fundamental difference between the way engineers and scientists (especially any scientist using Statistics) think about error. As an engineer, you probably consider that there can be a maximum amount of error allowed for a measurement, corresponding to a tolerance you place on the specification for a part. To a scientist, though, there is no absolute "maximum" amount of error, there is only the decreasing likelihood of getting any given error, as the magnitude of the error gets bigger and bigger. This goes back to Gauss, who was among the first people to realize that this can happen, and for whom the "Gaussian" distribution was named. The widespread applicability of the Gaussian distribution leads us to think about error that way. Very large errors are always theoretically possible, but the chance of them happenning are so infinitesmally small that we discount them. But since we can't say that they'll never happen, we can't put an upper limit on it. So to be realistic, we place some (rather arbitrary, to be sure) specification on what we consider the probability for a "likely" range: 95%, 99%, 99.9% etc.; these correspond to 2.5, 3, 4 standard deviations of the Gaussian distribution. So if you want to specify the range that corresponds to "always" meaning less than 1% chance of being outside the range, then you say 3 standard deviations. If you want to specify the range that corresponds to "always" meaning less than 1 part in 1000, then you say 4 standard deviations. And so forth. But it's always a probabilistic statement. We do it because near as we can tell, that's the way the world works. A lot of this stuff was developed by a guy named Shewart, especially with application to quality control; he was an engineer working for the telephone company, in the 1930's or so. \o/ /_\ | ||
| Pedro Castro Nunes Fiolhais (pedro_fiolhais) New member Username: pedro_fiolhais Post Number: 2 Registered: 9-2006 |
Howard and Kenneth, thanks for your post. maybe I was a bit confusing in my post, and just to make me clear, I want to say that I normally see RMSEP in the articles. What I never see is the RMSEP associated to the error of the calibration method �� 2 x RMSEP�. Normally people use RMSEP only as a value to chose the best calibration (I strongly agree with this use). What I never see in NIR/PLS articles is the error of the method. I am an engineer, and for me one thing is clear, every method has an error. So when you present a value, the value only makes sense if you present also the error of the value. Ex: 2�0.1 So my question is how do I calculate that error (Ex:�0.1)? | ||
| Kenneth Gallaher (ken_g) New member Username: ken_g Post Number: 2 Registered: 7-2006 |
I would second Howard's remarks on "black box" analytical instruments. Instruments on one level have gotten so easy to use that "anybody" can use them. The result is that some believe that what ever comes out of them is TRUTH. All analytical instruments and methods have errors. A very good place to start is - if there is one - the ASTM method for whatever you are running - although not always there is usually a good estimate of errors there. For a NIR/PLS method that I know works - IE NIR is appropriate and the NIR hardware is solid - NIR is in fact a good test of the reference method and of the laboratory running it. I have seen great labs and ones that generated random noise. Likewise NIR can easily distinguish between an intrinsically poor lab method like the old olefins in gasoline method +/- 3% error vs olefins via NMR which is more like 1%. | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 48 Registered: 9-2001 |
Pedro - there are lots of possible answers to your last question; the one that would apply would depend, among other factors, on what you're reading. If you're reading articles heavily oriented toward chemists, with no reference to the underlying statistical or chemometric background, it may simply be that the authors don't appreciate the importance of specifying the accuracy of the analytical method. Unfortunately, to at least some (and maybe many) chemists, the whole instrument and calibration process has become simply a "black box" where samples go in and numbers come out, with no appreciation for what goes on inside. Another possible explanation is that the terminology for specifying what you are labelling RMSEP has become very confused over the years, with some people using RMSEP to mean something different than what you are using it for, and other terms (and acronyms) for what you are calling RMSEP. So in this case, it may be that the RMSEP is being reported, but called something else. You might look to see if the papers you read present one or more of the following values: SEP, PRESS, MSEP, SECV, RMSSR. If they do, then see if you can find out how the corresponding value is calculated, it may be what you are calling RMSEP. That's not guaranteed, however, since, as mentioned above, each of these terms is used in other ways, too. A consistent, and recommended, set of calculations is presented as a concensus standard of the ASTM: Standard Practice E1655, but here again, most chemists are probably not aware of this standardized way of reporting the results of mutivariate methods of analysis. As for the other question: "IS THIS REALLY LIKE THIS?" when referring to: Ypredicted � error = Ypredicted � 2 x Standard Deviation = Ypredicted � 2 x RMSEP the answer is "yes, but". All the various ways of calculating the "alphabet soup" of terms I listed above (and some other, that I did not list) are attempts to figure out, from the calibration data and some data not in the calibration, how much deviation you can expect from your predicted results. These are all expressed in terms of standard deviations, but represent different ways of making the estimate. The multiplication by two is intended to give what are called the "95% limits", which means just that: it does not mean that you will never see a value beyond those limits, just that values beyond the 95% limits should not occur more than approximately 5% of the time. So if you measure 100 samples, you can expect that roughly 5 (maybe 4, maybe 7) of them will be in error by more than 2X the RMSEP. It's actually very straightforward statistics; it's unfortunate that few chemists study that. \o/ /_\ | ||
| Pedro Castro Nunes Fiolhais (pedro_fiolhais) New member Username: pedro_fiolhais Post Number: 1 Registered: 9-2006 |
Dear All I have studied NIR and Chemometrics for the last 6 months, and now I have to build a multivariate calibration (PLS) based in the NIR spectra, for moisture determination in lyophilasated compost (Pharmaceutical Industry). My doubt is in the error of the method that I have to present to the authorities, in order to validate the method. In the books I have read (for instance �A user-friendly guide to Multivariate Calibration and Classification�, NIR Publications), I saw that for a calibration equation with small BIAS, the error of a prediction should be � 2 x RMSEP (Root Mean Square Error of the Predictions), and this error have an approximate 95% confidence interval for y. The explanation for this errors in the books, is that the �� 2 x RMSEP� is a very good approximation of the empirical �rule of experience� � 2 x Standard Deviation (in a normal distribution, 95% of the scores are within 2 standard deviations of the mean). This makes no sense to me. This is like saying: Ypredicted � error = Ypredicted � 2 x Standard Deviation = Ypredicted � 2 x RMSEP IS THIS REALLY LIKE THIS? The other question I have is: If the error of a NIR calibration is �� 2 x RMSEP�, why in every article I have read this value is never presented? I am sure that the error is important information. |