Author |
Message |
Pierre Dardenne (dardenne)
Senior Member Username: dardenne
Post Number: 50 Registered: 3-2002
| Posted on Tuesday, October 05, 2010 - 7:29 am: | |
Hi Jody, I am quite surprised with your results. May I give some comments; - The gap between SEC and RMSECV is too wide indicating a too small data set. - 12 factors for a problem which seems to be a binary mixture (water-fat as protein is quite constant) seems a high number. (Our meat data set (reflection 650 spl; 0.3-35% SECV 0.5 with 4 factors). With your quite narrow range one would expect +/- 0.2% as RMSECV with an error of the reference method around 0.1% - The RMSECV is going down until factor 19 showing perhaps an inappropriate setup of the CV (duplicates?). A CV error is increasing generally earlier. Best regards, Pierre |
Jody Hoefs (hoefs)
New member Username: hoefs
Post Number: 2 Registered: 9-2010
| Posted on Tuesday, October 05, 2010 - 6:47 am: | |
This is the report for the high range, we are using Sensologic software in combination of the polytec diode analyser both are German companies. Perhaps something you can use: Polytec PSS-M Polychromator details: - Transmission design for maximum sensitivity and extremely low scattered light (contrast ratio > 1:50,000) - Lower dark current, also in NIR range, due to new Hamamatsu detector technology - Models covering 5 standard spectral ranges between 550 nm and 2,500 nm - Pre-aligned and supplied with wavelength calibration data (�� < 0.5 nm) - Electronics can be directly attached - With integrated shutter for dark signal measurement Report High FFA range: Name : 08july2010 Comment : Filename : C:\MROPLO~1\EOCOPY.CPF CWS version : 2.10 ------------------------------------------------------------------------------- SERIES: Name : FFA for probe (1) extended Comment : Created : 05/06/2009 Modified : 09/02/2010 No of spectra : 1200 No of datapoints : 501 Wavelength range : 1100..2100 nm, 501, 2 nm steps No of properties : 1 Properties : ffa ------------------------------------------------------------------------------- CALIBRATION: Name : FFA (high) PLSR P1 extended Comment : Created : 05/06/2009 Calibration method : PLSR Calibration Set : 80 Spectra 889-944 961 964 969 976 987 990 993-994 1024 1027 1032 1039 1050 1053 1056-1057 1087 1090 1095 1102 1113 1116 1119-1120 Selected property : ffa Property range : 1.775 to 6.395 Selected wavelength range : 1114..1600 nm, 244, 2 nm steps Transformation : SNV(abs) - Absorbance - SNV Int. Calibration number : 001B001F ------------------------------------------------------------------------------- REGRESSION RESULTS PLSR CROSS VALIDATION Selected CVS : 5 Number of factors : 12 (Selected Maximum : 20) Number of factors RMSECV ----------------- -------- 0 1.400543 1 1.236108 2 1.106175 3 0.774568 4 0.681779 5 0.667040 6 0.594106 7 0.534346 8 0.459576 9 0.361227 10 0.323735 11 0.306105 12 0.281469* 13 0.267258 14 0.252092 15 0.250368 16 0.242642 17 0.256191 18 0.252295 19 0.232173 20 0.238797 ----------------------------------------------------------------------------------- OUTLIER DETECTION: Number Predicted Actual Difference 'T' 'H' 'D' 'S' 921 3.91604 3.38100 0.53504 3.28406* 1.28078 1.09024 0.861864 944 5.76820 6.39500 -0.62680 -3.81258* 1.19167 1.34260 0.353435 969 1.99545 2.03000 -0.03455 -0.20436 0.90700 0.00278 6.642235* ----------------------------------------------------------------------------------- Multiple correlation coefficient : 0.992705 Standard error of estimate : 0.183082 Root mean square error of cross validation : 0.281469 Index of systematic variation : -14.127 Index of random variation : 25.5268 Signature : ____________________ <end> |
venkatarman (venkynir)
Senior Member Username: venkynir
Post Number: 120 Registered: 3-2004
| Posted on Monday, October 04, 2010 - 8:58 pm: | |
Dear Mark ; Thanks for your suggestion .However the manufacturer of DAS is OEM company , they could not help in this .We NIRs lovers have work for it. I have observed that Dark value unaltered . The reference changes might be due deposit of materials ? I have observed a micron grew was made to fix window materils , there the oil residual deposits . I might case the problem. I am working on it. Your valubale suggestion , I will take note of it. |
Howard Mark (hlmark)
Senior Member Username: hlmark
Post Number: 353 Registered: 9-2001
| Posted on Monday, October 04, 2010 - 12:16 pm: | |
Venky - I recommend that you contact the manufacturer of your instrument for recommendations on how to troubleshoot the varying reference signal problem \o/ /_\ |
venkatarman (venkynir)
Senior Member Username: venkynir
Post Number: 119 Registered: 3-2004
| Posted on Monday, October 04, 2010 - 10:35 am: | |
Dear Jody Hoefs Thanks for the message . My range is 1% to 4 % at one end . We are getting good result at High FFA. The main problem here is Reference changes . I am using Flow cell with saphire window. I have changed the fiber 1000 micron senstivity increased but Ref,problem persist. I have to use it for on-line measurment . If I am correct you have used MEMs based NIRS . Is it Can you send more details on detrend and SNV. I have worked on SNV.But it is not yiled expected change over . |
Jody Hoefs (hoefs)
New member Username: hoefs
Post Number: 1 Registered: 9-2010
| Posted on Monday, October 04, 2010 - 3:39 am: | |
Hi, For our customer we are using the Polychromatic Diode Array and have seen that we can reach 0.03% in SECV range up to 0.15%FFA and 0.19% range 0.15-2.831 and a high range as well. So three ranges to cover the whole area, I have used SNV detrend and spectral ranges 1114-1600+1850-2080nm. Second derivative had the problem that when you change a lamp or measuring over time that the results were not repeatable. With FTNIR we don't see this problem, only with diode array. We are using a Hellma probe pathlength 5mm as well and unfortunately I just got the message that the tip has been broken off. Probably something in the stream hit the probe tip. Hope this helps. |
Howard Mark (hlmark)
Senior Member Username: hlmark
Post Number: 352 Registered: 9-2001
| Posted on Wednesday, September 29, 2010 - 9:55 am: | |
Venky - if random variability is affecting your results, which seems likely from the fact that the dark readings are changing, then there's no data transform or algorithm that will "magically" fix it. First the problem has to be solved, THEN you can start to consider ways to optimize your calibration. \o/ /_\ |
venkatarman (venkynir)
Senior Member Username: venkynir
Post Number: 117 Registered: 3-2004
| Posted on Wednesday, September 29, 2010 - 6:49 am: | |
Dear Mark & David ; Thanks for your sugesstion. You have not answered can go with PLS with ratio method that is taking part of the spectra dividing it with processed spectral data. Here the flow is uniform , there in no chance for exteternal elements to play . I have read papers , all them used MLR with selective wavelength. It is good for experimental but pratical it is difficult. |
Howard Mark (hlmark)
Senior Member Username: hlmark
Post Number: 351 Registered: 9-2001
| Posted on Wednesday, September 29, 2010 - 4:53 am: | |
Venky - I agree with Dave about the spectroscopy, but I would go even one step further back. The fact that the dark reading changes when you're measuring dynamically means that something about the measurement conditions is affecting the instrument. Presumably the static measurements were made in a laboratory, where environmental conditions (temperature, humidity, electrical noise, vibration, etc.) were well-cotrolled, while the dynamic conditions are out in a plant where the environment is not well-controlled. Since a "dark" reading by definition does not involve any light, your problem seems to be even more basic than the spectroscopy, and I don't foresee you being able to get any stable results until that is solved. After the dark readings are stable, then you want to consider factors that would affect the light readings, which is where the spectroscopy comes in. Another condition you would want to look at, once you're assured of a stable instrument, is whether the sample is homogeneous. The points David made are good ones, but I would also recommend that you verify whether the composition of the sample material you're measuring is constant and uniform. \o/ /_\ |
David W. Hopkins (dhopkins)
Senior Member Username: dhopkins
Post Number: 164 Registered: 10-2002
| Posted on Wednesday, September 29, 2010 - 3:23 am: | |
Hi Venky, I think it is possible you need to look at what is happening in the cell in the dynamic mode, when the sample is moving through the cell. It sounds like you may have bubbles or schlieren lines due to the uneven flow and mixing in the cell, which would mess up your measurements. You need to solve this problem before proceeding, because I think you should be able to use the static models on the dynamic situation. I would certainly recommend that you stick with the PLS (or PCR) models, so you can use the full spectral models. Then you have the possibility to use a method to qualify that the spectra are good before giving a result. From what you say, you would be smart to test that the spectra/baseline are high quality, to avoid giving erroneous result. There is no magic treatment that can substitute for good spectroscopy. Best regards, Dave |
venkatarman (venkynir)
Senior Member Username: venkynir
Post Number: 116 Registered: 3-2004
| Posted on Wednesday, September 29, 2010 - 1:01 am: | |
Dear All I am working on on-line measurment of FFA value (low value 2% to 10 %) for sunflower oil. Iam using DAS of range 1000 nm to 2200 nm with 512 elements . I have done pre-porcessing like (1) Baseline, (2) Smoothing (3 ) Dervative for the captured spactral data . I have used flow cell with path length 5mm in transmission mode. I got good linearity and calibaration in static condition. But in the dynamic , the Dark and reference are subject to change and mislead the calibration . (a) I need help whether PLS to be applied or MLR? (b) In the case of MLR which oil band I have to use( oliec group samples ) (c) If I use PLS , can I use one varaible as ref and ratio metric approach is good in case of on-line measurment? (d) Any simple approach to measure low FFA in on-line . |
Malcolm Ray Brown (bro609)
Junior Member Username: bro609
Post Number: 7 Registered: 7-2007
| Posted on Thursday, June 17, 2010 - 6:07 pm: | |
Thanks Tony. I'll try taking out the water bands as well. It's good to hear some positive feedback on the T stabilisation approach. I met with Christel Solberg at the NIR2009 conference in Nov last year, and we had a good discussion about our respective work on NIRS and scanning salmon. We didn't specifically discuss T effects ( I wasn't aware of this problem then) - but I'm sure I could follow up with her if necessary. Malcolm |
Tony Davies (td)
Moderator Username: td
Post Number: 233 Registered: 1-2001
| Posted on Thursday, June 17, 2010 - 5:54 am: | |
Hello Malcolm, The lady that Bruce was thinking of is Prof. Christel Solberg of Bod� University Norway. I have tried to e-mail her but no reply; she may be on holiday preparing to enjoy the midnight sun! Anway you seem to have found some good advise, I have used the temperature stabilisation technique successfully. The other thing you can try, assuming you are using PLS, is to take out the water bands. The calibration might not be so good but it should remove the temperature problem. Best wishes. Tony |
Malcolm Ray Brown (bro609)
Junior Member Username: bro609
Post Number: 6 Registered: 7-2007
| Posted on Thursday, June 17, 2010 - 1:58 am: | |
Thanks Dave, Karl, Bruce and Venkatarman for you collective responses. Bruce � I recently visited the NIRS group at NOFIMA, Norway led by Jen Petter Wold. Their work on salmon suggested that fat calibration can produce systematically different results as a function of T (eg. may be linked to shifting of water peak with lower T) � though they did not specifically correct or account for this in their modelling (maybe they had good T control). Dave � I have not specifically plotted moisture against fat (have not measured moisture), but there is a general well-recognised inverse relationship between fat and moisture in fish (protein is relatively constant). So, as you suggest, this may be the main factor contributing to the T sensitivity of my model. I have gone through the exercise of repeat-measure predictions of fat on my fillet over a 2h period as the T warms from 0 to 9C � and yes, I see a straight (sloping) line, eg. with predicted fat changing from 11% wet weight to 10%. I can�t be 100% sure that this is not due to metabolic breakdown, but at that T and timescale I suspect not. The other thing that has been suggested to me by a colleague, is to add into the calibration a �temperature stabilisation set� eg. scan up to a dozen individual samples at several temperatures (upper and lower extremes of scan temperatures and midpoint) � and put this into the calibration. My understanding is that, within the revised calibration the changes in spectral response from T are then interpreted as noise. This approach is also documented in the paper by Segtnan, V.H. et al. (2005). Low-cost approaches to robust temperature compensation in near-infrared calibration and predictions situation. Appplied Spectroscopy, Vol 59 (6), pp. 816-825. So, I�ll also give this a try to see it improved the robustness of the model. I further take your point Dave of the many other factors that may be associated to model robustness, such as sample presentation and analytical error. Thanks again. Malcolm |
venkatarman (venkynir)
Senior Member Username: venkynir
Post Number: 104 Registered: 3-2004
| Posted on Monday, June 14, 2010 - 11:29 pm: | |
Dave ; You are partilly corrcect .Temp would not effect FAT only certain extend. But when it cross the limit it goes wrong . I do agree moisture plays vital role in calibration of other produts . We can provided compenstion for it that was suggested by indirectly . |
David W. Hopkins (dhopkins)
Senior Member Username: dhopkins
Post Number: 153 Registered: 10-2002
| Posted on Monday, June 14, 2010 - 10:48 pm: | |
Hi Malcolm, My experience with meat measurements showed that fat measurements were not sensitive to temperature, while moisture measurements can be very sensitive. I think that your fat calibration may be sensitive to temperature if there is a correlation between moisture content and fat content in your training set. Have you plotted fat, protein and moisture against each other pairwise, to check for correlations? Or possibly, the scattering in the fish fillets is changing with temperature. If you plot the predicted fat contents for individual samples at various temperatures, do you see a straight line? I expect a straight line for moisture calibrations, from my experience. If the plot is more random, there may be light scatter problems. In either case, I suspect you need to increase the number of samples in your training set. What is the level of fat in fish fillets? Does it degrade metabolically, so that it may be critical to make your lab reference measurements on the same day as you make the NIR measurements? Does your lab obtain good reproducible results if you submit blind duplicates? There are many reasons that a fat calibration might not be robust, and you need to examine all possibilities. Best wishes, Dave |
venkatarman (venkynir)
Senior Member Username: venkynir
Post Number: 103 Registered: 3-2004
| Posted on Monday, June 14, 2010 - 8:52 pm: | |
To my knowledge I have not seen up to 0 to 45deg.c the FFA not affecting the spectral pattern . |
Bruce H. Campbell (campclan)
Moderator Username: campclan
Post Number: 122 Registered: 4-2001
| Posted on Monday, June 14, 2010 - 8:42 pm: | |
Malcolm, The first thing to do is contact the Norway department that uses NIR to study fish. They may have encountered your problem. I don't remember the name of the department but they are located in northern Norway. Bruce |
Karl Norris (knnirs)
Senior Member Username: knnirs
Post Number: 34 Registered: 8-2009
| Posted on Monday, June 14, 2010 - 7:57 pm: | |
Malcolm, I am not aware of fat spectra being sensitive to temperature, but the water band changes dramatically for small temperature changes around 0 degrees. See S.R. Delwiche, R.E. Pitt, and K.H. Norris. Cereal Chem. 69(1):107-109. 1992. Karl |
Malcolm Ray Brown (bro609)
New member Username: bro609
Post Number: 5 Registered: 7-2007
| Posted on Monday, June 14, 2010 - 6:49 pm: | |
Hi, I�ve developing models to predict fat in fish fillets, but recently noticed significant day-to-day variation in prediction values. Prior to testing, the fish had been chilled in saltwater/ice overnight. After removing each fish and quickly excising the fillet and scanning, the T of the fillet may be in the range -1 to 4 C. Though this is a relatively narrow range, a separate lab experiment has shown that predicted fat changes significantly over this range (eg. from 14% at -1C, to 11% at 4C in the same sample). I have done a bit of reading on "T compensation" options within calibration models - still a little confused, but wondering if anybody has advice: eg. 1) should I be simply trying to build a "global calibration" model, incorporating many samples spread across the range of temperature of scanned 2) measure T of fillets prior to scanning (eg using an infrared thermometer) and make a correction based on the lab experiments I have done on T vs predicted FAT. This assumes all fillets have the same T/predicted FAT relationship. 3) accept that there will be some noise in the data related to T - but minimise as much as possible T sample fluctuations (there are some logistical constraints here associated with sample volume/ throughput). I appreciate it may be difficult to give a definitive answer, especially without a better knowledge of the process, but any thoughts are appreciated. Thanks, Malcolm |