Ist derivative vs 2nd derivative prep... Log Out | Topics | Search
Moderators | Register | Edit Profile

NIR Discussion Forum » Bruce Campbell's List » I need help » Ist derivative vs 2nd derivative preprocessing « Previous Next »

Author Message
Top of pagePrevious messageNext messageBottom of page Link to this message

Bilal Ahmad Malik (elp09bm)
Junior Member
Username: elp09bm

Post Number: 7
Registered: 7-2011
Posted on Monday, August 08, 2011 - 8:16 am:   

Hi All,

Could anyone please advice.

Thanks,
Bilal
Top of pagePrevious messageNext messageBottom of page Link to this message

Bilal Ahmad Malik (elp09bm)
Junior Member
Username: elp09bm

Post Number: 6
Registered: 7-2011
Posted on Thursday, August 04, 2011 - 9:53 am:   

Hi Barry,
Thanks for your response.
As you pointed out there looks to be something wrong.The above result were using support vector regression.
But now I thought I will use PLs and I tried first
using all the 90 samples for cross validation and got following results.

RMSEC RMSECV Preprocessing LVS
19.5 27.19 Istderivative 7
27.4 67.32 2ndderivative 6

Then I did usual method of splitting the data into training and test.
I used 60 for training and 30 for test.
the results were as follows

RMSEC RMSECV RMSEP Preprocessing LVS
29.13 36.4 37.3 Istderivative 6
20.9 70.8 82.39 2ndderivative 6

As you could see here in this model as well that I am getting better results with first derivative.
if you need more information like the graphs I am getting I could send.
Top of pagePrevious messageNext messageBottom of page Link to this message

Barry M. Wise (bmw)
Junior Member
Username: bmw

Post Number: 7
Registered: 2-2011
Posted on Wednesday, August 03, 2011 - 2:29 pm:   

Hi Bilal:

You wouldn't generally expect a model with second derivative to be consistently better (or worse) than one based on first derivative data. It depends on the data. If it you have only a constant baseline problem, then first derivative would probably be better, whereas if you have a problem with a sloping baseline, you'd expect second derivative to be better. In general, derivatives "blow up the noise," so you want to use the minimum derivative that works.

Thanks for including the table. If your model based on first derivative has an RMSEC of 13.5 but a RMSECV of 34.5, then it is pretty over-fit. How many PLS or PCR factors did you use? You might actually get a better RMSEP on the test set if you backed off by one or two. You'd likely also wind up with a model that was a little more robust--less sensitive to instrument drifts and minor contaminants in future samples.

Looks like either you dropped a digit in your table for the second derivative, or if not, your model is WAY over-fit with a RMSEC of 1.45 and RMSECV of 97.32. What else is different about this model besides the derivative? I'm guessing something.....

BMW
Top of pagePrevious messageNext messageBottom of page Link to this message

Bilal Ahmad Malik (elp09bm)
New member
Username: elp09bm

Post Number: 5
Registered: 7-2011
Posted on Wednesday, August 03, 2011 - 11:15 am:   

I have NIR data which for which I am trying to develop a calibration model using PCR,PLS and SVR(support vector regression).
My result show better SEP value when using Ist derivative followed by autoscale as compared to 2nd derivative though RMSEC is better when using 2nd derivative.
I was expecting better SEP value as well when using second derivative as well. This happens in case of PLS,PCR,SVR.
Could any one take the pains of explaining if it is acceptable behavior or is something wrong.
I am using PLS_ToolBOx_621 for analysis.
SVM
RMSEC RMSECV RMSEP R^2Cal R^2Pred Preprocessing
13.5 34.5 20.39 0.99 0.95 Istderivative
1.45 97.32 82.39 0.99 0.78 2ndderivative
Kindly help.

Add Your Message Here
Posting is currently disabled in this topic. Contact your discussion moderator for more information.