| Author | Message | |||
     Karl Norris (knnirs) Junior Member Username: knnirs Post Number: 7 Registered: 8-2009 |
Ralf, I forgot to mention that the noise in the 2000 to 2200 nm region is about eight times higher than in the 1500 to 1650 nm region for the same three flour spectra you referred to. Karl | |||
| Karl Norris (knnirs) Junior Member Username: knnirs Post Number: 6 Registered: 8-2009 |
Ralf, I have been absent from this problem for the past week, and I need to inform you that the instrument used for the flour spectra was an older instrument having a wavelength spacing of 2 nm. The newer instruments have a wavelength spacing of 0.5nm and an even lower random noise level. I am confused by your statement about the signal being spread out over 500 nm and then dividing 500 by 7 to get 71 resolved points. Karl | |||
| Ralf Marbach (ralf) Member Username: ralf Post Number: 12 Registered: 9-2007 |
Hi Andrew, Karl, Don, All There is a formula for the best possible limit of detection: LOD = 3 / sqrt(g� *invS *g) [ppm] The factor, 3x, is just a matter of definition: most people define LOD as 3x the RMS error. The meanings of signal g and noise S are described in my Aug 12 post below. If g is the true response spectrum of the analyte in question, melamine, then the LOD is from measurement, i.e. specific. (Otherwise, if g is something else that just happens to correlate, then the �LOD� is from statistical analysis.) Karl has shown the melamine spectrum. It has some nice spectral features. The log(1/R) amplitudes are approx. 0.4 AU around 1500 nm and approx. 0.7 AU at >2000 nm. This being 100% melamine, I divide by 1,000,000 ppm and estimate the amplitude of the melamine response spectrum in milk powder at roughly 0.4e-6 and 0.7e-6 AU/ppm, respectively. (I can see Don shudder and reach for the keyboard, and rightly so, because I am wildly assuming here that the scatter properties of the melamine sample are close to those of milk powder � I am aware of my sins, but here I only need a rough approximation, factor 2 or so is good enough.) Next, observe that Karl�s hardware noise floor is superb. The Foss 6500 instrument has a huge light throughput. One of the slides in his presentation (couple up from the melamine spectra, entitled �instrument noise with 3 flour samples�) shows a noise level on the order of 3 microAU peak-to-peak, which is less than 1 microAU RMS. (BTW: I am confused about this plot, Karl: it looks like the point spacing plotted is roughly 5 nm, but I thought the 6500 gives a point every 0.5 nm?) Now, assume your task were to detect whether a trace amount of (very finely ground) melamine has been sprinkled onto your Teflon reflection standard or not. The noise matrix S in the LOD equation will then only contain the hardware noise floor, on its diagonal (plus some baseline offset noise but let�s not split hair here). Looking at Karl�s spectrum we see that melamine has a response of about 0.7e-6 AU/ppm over roughly 500 nm in the combination region. On the 6500, this corresponds to 500nm / 7nm=71 resolved points. (Please correct me if needed, Karl.) Assuming that on each of these 71 points the hardware noise is 1.0 microAU RMS (Karl?), we insert the approximations for the combination region into the formula above and get: LOD = 3 / ( sqrt(71) x (0.7microAU/ppm) x (1/1.0microAU RMS) ) = 0.5 ppm (Adding the first overtone region into this calculation would improve the result only marginally, because the response amplitudes add in squares, so I only include the combination region into this rough estimate here.) �0.5 ppm� sounds wonderful � in praxis we usually don�t get close for two reasons. First, many of us work with instruments, and integration times, where the hardware noise floor is 100x worse than above. Second and more important, in real situations the matrix S contains not only the hardware noise floor but also the variances from ALL things other than melamine varying in the spectra: spectral differences in milk powder brand to brand, sampling variations, different moistures, temperature effects, etc. These spectral �noises� are much larger in amplitude than the hardware noise floor and, when added to the matrix S, will �eat� the response signal up and leave only very little signal to stand above the noise floor, if any. So everything is very application specific. Luck is needed so that some of the response peaks of the trace analyte �survive� the larger noises, which tend to quickly fill up the 71 available dimensions with variances larger than the hardware noise floor. The LOD formula above is from Norbert Wiener. Here is the impressive part: He proved that this is the best possible result GLOBALLY. In other words, whatever fancy math or procedure is employed, we all fall under the same limit for �limit of detection�. If the application is instationary, i.e., g = function(time), then the formula applies to that moment in time. If the problem is stationary, g = constant(time), then the same LOD limit applies always. Having a stationary measurement system is preferred in praxis, of course. It takes an instrument with very good noise floor and good long-term stability, plus skilful sample presentation, and then some luck in how the spectral peaks of the interferents overlap the analyte to achieve an LOD in the ppm range in the NIR. The fact that he used only narrow spectral ranges makes Karl�s achievement all the more remarkable. Ralf VTT Optical Instrument Center MTT Multantiv | |||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 262 Registered: 9-2001 |
Andrew - I just read something interesting, especially in the context of this discussion: in the latest issue of American Laboratory, the article by David Coleman and Lynn Vanatta on "Statistics in Analytical Chemistry" quoted William Horwitz (of the FDA and AOAC) in a 1988 symposium as saying: "In almost all cases when dealing with a limit of detection or limit of determination, the primary purpose of determining that limit is to stay away from it." \o/ /_\ | |||
| Andrew McGlone (mcglone) Member Username: mcglone Post Number: 15 Registered: 2-2001 |
Howard - quite right but I'm abit more interested than that since things can change as opportunities arise (e.g., I borrow or buy a new instrument or learn some tricks...). So this w/e I read over Karl Norris's presentation and the NIRnews paper he referred me to. Marvellous stuff, such care with getting the experimental detail controlled and understood in terms of effect; certainly avoids all that confusion or 'lack of trust' you get with PLS-type analysis. Anyway I have no doubt Karl Norris can drive down to 2 ppm for talc in avicel as the only limit appears to be detector noise, which looks to be less than +/- 1microA units (a graph in his presentation shows instrument noise). And so when Karl purposely steps back from that level and says Melamine adulteration detection is unlikely to happen below 100ppm levels, in any practical sense, I'm pretty ready to believe him. In fact, as I read his presentation on detecting flour adulteration I got the feeling that actually 1000 ppm might be the more prudent bet. I might just have gone full circle here... | |||
| Johan Trygg (trygg) New member Username: trygg Post Number: 1 Registered: 1-2006 |
Hi all, Thanks for an interesting discussion. Tried to post earlier, but was unable to get it passed. Regarding estimation of pure profiles, the PLS model can actually estimate the pure component response profiles by rotation of the regression coefficients of the PLS model. The pure component response profile estimates are found via, K=B*inv(B'*B), where B is the regression coefficient matrix of an inverse regression method, e.g. PLS. Please see the paper from 2004, Trygg J, Prediction and spectral profile estimation in multivariate calibration, JOURNAL OF CHEMOMETRICS 18 (3-4): 166-172 MAR-APR 2004 When unknown interferents are present in the sample spectra it was demonstrated that the OPLS method was much more successful, compared to PLS, when interferents were present in the sample spectra. Best Regards, Johan Trygg, Ume� University, Sweden | |||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 261 Registered: 9-2001 |
Andrew - I'd say that what you need to think about most, is to do some experiments to tell you what your LOD is using YOUR instrument and under the conditions YOU can provide. \o/ /_\ | |||
| Karl Norris (knnirs) New member Username: knnirs Post Number: 5 Registered: 8-2009 |
I wish also to refer you to a follow-up paper in NIRnews Vol.17 #7 (2006). You are right that LOD for NIR involves a lot of specifics. The current NIR instruments have reduced the random noise by a factor of eight, so I believe I could now demonstrate a measurement at the 2 ppm level. | |||
| Andrew McGlone (mcglone) Member Username: mcglone Post Number: 14 Registered: 2-2001 |
someone told me last night that I was really splitting hairs to worry about the difference between 1000 and 100 ppm, and that I really don't do enough 'real' NIR to know. I agreed. Still I'm obviously not the only person in this business running around with the 0.1% LOD figure in their head. But 10 ppm now? I better read your 2001 paper Karl and reset my expectations alright! And I should probably find out exactly why LOD is not particularly useful for NIR. I didn't know that although I could perhaps guess that any LOD figure with NIR is so loaded in specifics to particular sample/matrix effects or similar that any generalisation is void. Hmmm, it's a pity for me as having even a rough sort of LOD in my head makes life a little easier when people come calling and ask if NIR might do this or that. The alternative is that I might just have to think... | |||
| Karl Norris (knnirs) New member Username: knnirs Post Number: 4 Registered: 8-2009 |
Karl Norris also has a publication showing measurement of talc in avicel at less then 10 ppm. "K. H. Norris Amer. Pharm. Rev. 4(6)2001". This was using an instrument marketed in 2000. Does this mean LOD of 0.001%? As others have stated LOD is not a very useful parameter for NIR. | |||
| Andrew McGlone (mcglone) Member Username: mcglone Post Number: 13 Registered: 2-2001 |
I'm awfully confused now. Karl Norris suggests an LOD of 100 ppm (0.01%). Lu et al. suggest a LOD of 1 ppm (0.001%). Bengt Norlund (in starting this thread) goes with the common mantra that NIR typically will have an LOD of 1000 ppm (0.1%). I'm certainly no expert on NIR but I'm done enough here and there to have happily swallowed the same mantra that Bengt Norlund has. So I was very sceptical of that initial melamine result (1 ppm). But if someone like Karl Norris is getting down to 100 ppm then I obviously need to reset my expectations. Or is the game just so different when doing discriminant type analysis, like Lu et al and Norris appear to be doing, rather than quantitative analysis where you tie your data to a numerical scale rather than to a simple in/out type metric? | |||
| Karl Norris (knnirs) New member Username: knnirs Post Number: 2 Registered: 8-2009 |
Chuck, It seems I was not clear when I discussed the temperature effect on melamine detection. The temperature sensitivity is caused by the OH bands of water in the sample, not the melamine. I doubt that the spectrum of melamine changes with temperature in the 15 to 30 degree range. Sorry, Karl | |||
| Charles E. Miller (millerce) Member Username: millerce Post Number: 14 Registered: 10-2006 |
Greetings Karl! Thanks a lot for sharing the link to the Workshop talks- some fascinating reading! I must admit, looking at the high-res spectrum of melamine is somewhat reminiscent of the work that Tony and I did some time ago trying to understand the source of the omnipresent sharp 1440 nm band of crystalline sucrose. We postulated that this sharp band is an overtone of the stretching mode of a specific OH group that is �frozen� into the crystal structure in such a geometry that it is not able to H-bond with neighboring molecules. I don�t recall us getting too much grief from the spectroscopy experts back then�. Perhaps a similar case exists for NH in melamine? I would assume that X-ray crystallography-based crystal structures of melamine have been published in the literature, but didn't see it in my brief review of the Proceedings. If so, then this might explain this band�s non-linear and non-stationary behavior due to temperature and moisture sensitivity. Regards, Chuck | |||
| Karl Norris (knnirs) New member Username: knnirs Post Number: 1 Registered: 8-2009 |
Sorry to report that I just stumbled on this forum. I will not discuss the merits and pitfalls of PLS, as most of you know I don't use PLS. I wish to address the real question of how NIR can best help food processors avoid melamine adulteration in their products. Yes,we can develop calibrations for measuring the melamine content of food products, but these should be resticted to detecting levels of 100 ppm or higher using present NIR technology. Thanks to Ron Rubinovitz I have a high-resolution spectrum of melamine, and this spectrum shows many sharp absorption bands. Two bands, at 6812 and 5116 cm-1 stand out because they are strong and very narrow. I estimate the bandwidth to be less than 12 cm-1 (<3nm) for the band at 6812 cm-1. It appears the two publications by Mauer et al and Lu et al focused on the band at 5116 cm-1, but I like the band at 6812 cm-1 because it is more isolated. Both of these absorption bands are compromised by the broad absorption bands of water when we look at food products. This means that the user must be very alert to moisture changes in the sample as well as wavelength shifts from sample temperature. I have done a bit of investigation of the detection of melamine in food products as a result of an invitation to make a presentation at a USP workshop in June. All of the .PPT presentations are available at:http://www.usp.org/meetings/workshops/foodProteinWorkshop2009.html My presentation is the second paper in:Breakout Session A. In my paper I introduce a very simple method for detecting adulteration for incoming ingredients in a food processing plant. This is my procedure: 1. Have available 6 or more different calibrations for protein on the product to be tested. 2. Perform an NIR scan of the sample to be tested. 3. Predict the protein content of test sample with each of the different calibrations. 4. Observe the range of the predicted proteins. 5. If the predicted range exceeds a value (~5% of the average), the sample is declared as adulterated. I did not have samples or spectra to test this procedure with true spectra so I did a computer simulation to evaluate the idea. My computer simulations indicated this procedure should detect adulteration of wheat flour with melamine at the 0.25% level. The procedure is not specific to the adulterant, but should detect any abnormal spectra including abnormal moisture levels. I have done added work on the detection of melamine at low levels, and I predict that a temperature change of 1 degree on the sample will produce a change of more than 10 ppm in the predicted melamine content. A small change in moisture content will cause a similar shift in predicted melamine content. Yes, these efects can be minimized by including sample temperature and a range of moisture content in the calibration samples, but the papers by Dr. Mauer et all and Dr. Lu et al did not address these questions. It should be noted that a calibration on samples where the changes were made by adding a contamination fits that kind of sample, but is not likely to fit samples in a food processing plant. My Thoughts, Karl Norris | |||
| Donald J Dahm (djdahm) Senior Member Username: djdahm Post Number: 26 Registered: 2-2007 |
First I will explain my position on Beer's law in the context of the previous discussion in this thread. Ralf said: "In the melamine example, measurement requires that the melamine response spectrum, in units of, e.g., [Absorbance/ppm], is determined (Step 1). If the absorbance features of the response spectrum turn out to be so small that it�s lost in the hardware noise..." Notice that he said "first step", and he called it a "melamine response spectrum", not a melamine spectrum . I was cautioning that if you extract the melamine response spectrum from data in a sample set of melamine in milk, you better be prepared to see large deviations from a spectrum of pure melamine, and expect it to be sample set dependent. If Beer's law really held, we would be in much better shape in that regard, but I'm sure there would be plenty of other things to worry about. If anyone wants to argue about Beer's law any further, I will post a response to Howard and Gabi in a new thread soon. | |||
| Gabi Levin (gabiruth) Member Username: gabiruth Post Number: 12 Registered: 5-2009 |
Hi jerry, I was unable to load a 99 kb file I prepared. Please send me an e-mail and I will send you some stuff. Gabi Levin [email protected] | |||
| Charles E. Miller (millerce) Member Username: millerce Post Number: 11 Registered: 10-2006 |
Jerry: If the loadings are generated from PCA, then they are constrained to be orthogonal. For PLS loadings (and weights), the orthogonality status depends on the specific algorithm used- but they are nonetheless constrained in some manner by the algorithm. These constraints effectively prevent loadings and weights generated by these methods from expressing the response profiles of pure components, for virtually all practical applications. If, in addition, the data are mean-centered prior to modeling, then loadings/weights can only describe variability about the mean- i.e., �difference effects�. However, please note that this does not preclude effective qualitative/interpretative analyses of loadings, as it has been shown that loadings can, under some circumstances (or with some rotation), express the difference between the spectra of 2 pure components, or the net change in a material�s spectrum as a result of some �treatment�. Hope this is useful. Regards, Chuck | |||
| Jerry Jin (jcg2000) Intermediate Member Username: jcg2000 Post Number: 16 Registered: 1-2009 |
Hello,guys A quick question before closing the "melamin" thread. What are you looking for in a loading vector? Are you trying to correlate a loading vector (say, the loadings of the 1st PLS factor) to the spectrum of the analyte of interest? The first several loading vectors, in their own definitions, certainly have appealing shapes. But they are not equivalent to any spectra. Cheers! Jerry Jin | |||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 254 Registered: 9-2001 |
I see the discussion group software edited out some of my text. When I was talking about the angle, I said it was angle theta, and also when I referred to the cosine, that was also to cosine (theta), where theta is the angle between a particular ray and the beam axis. \o/ /_\ | |||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 253 Registered: 9-2001 |
Gabi (made sure I got it right, this time!) - A small correction to what I think is a misunderstanding on your part: when I talked about the different light paths, I wasn't talking about powdered solids, I was talking about clear liquids. There's a more detailed write-up in Spectroscopy 13(11), p.18-21 (1998) (and also as chapter 29 in Chemometrics in Spectroscopy, Elsevier (2007)). Basically, the second law of Thermodynamics tells us that it's impossible to have a perfectly collimated beam that contains finite energy. Therefore, every light beam, even from the best laser, spreads out a bit, and therefore some rays are at an angle to the beam axis. Those rays at an angle to the beam pass through the sample through a pathlength that is increased by a factor = 1/cos(<theta>), where <theta> is the angle a given ray makes with the beam axis, and in an absorbing medium would suffer slightly more reduction in intensity because of that. A completely thorough description, then, would have to integrate the exponential falloff of intensity over the pathlengths for all the angles of the rays in the beam, taking into account the distribution of intensity at every angle. \o/ /_\ | |||
| Gabi Levin (gabiruth) Member Username: gabiruth Post Number: 11 Registered: 5-2009 |
Hi guys, Howard, thanks for correcting, I appreciate it. I would like to give a simple example why (truly, in clear liquids where the situation is a little better than in solids) the laws of Beer Lambert still work - and that is to demonstrate the exponential nature of the absorption which stems directly from physical laws that govern the absorption of electromagnetic radiation Take the case of launching light in the range from 1100 to 2100 nm into water in a cuvette. You will see that the "surviving" signal is weak at ~1450 and 1930. If you change the path length by reducing it you will see that the increase in signal (on a relative scale to the previous signal) is mcuh higher at 1930 than in 1450 and of course both increases are higher than at ~1300 - this is because the value of the absorption coefficents there are much larger than the coefficient at 1300. The value of the negative in the exponent is larger, therefore the change in signal is larger. Now, the key factor here is simply the difference in absorption coefficients in the different wavelengths as the concentration of pure water can be taken as 1, the exponent has only the path length and the coefficient. This simple relation between coefficient of absorption,concnetration and absorption holds independent from our experimental and mathematical human limitations. The fact that in solids there is no clear definition of path length as Howard put it - does not change the applicability of these laws. Apparently the issue of path length in diffuse reflectance or forward diffuse transmission (seeds, tablets) raises many questions. Since I am a practitioner all I can say is that we have very good success in measuring various parameters in various seeds - (oleic, linoleic, protein, starch etc.) regardless of their thickness. I am not sure I have a good explanation for that, but it works. And low and behold, when you analyze loading weights for each constituent, they tell you a very clear story. | |||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 252 Registered: 9-2001 |
Whoops! I think I have to apologize to Gabi, for calling him by the wrong name. Sorry about that, Gabi. \o/ /_\ | |||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 251 Registered: 9-2001 |
Don - I have to agree with Igor that, while you are correct in principle, Beer's Law is in fact a Law. Given your own definition, Don, "Given the teaching we all remember, that a "law" is a summary of observation for which there is no known exception ...", Beer's Law is a Law because there ARE no exceptions to the observations that Beer's Law describes. The problem seems to be one of communication. Igor has to understand that Don is a physicist, and didn't study the same things we chemists studied about analytical spectroscopy. Don has to understand that he doesn't know all that we are taught about the conditions (clear non-scattering media, plane parallel boundaries to the sample, transmission through the sample to be no less than roughly 10%, etc.) under which Beer's Law can be expected to hold. Under the necessary, specified, conditions, there are no exceptions to the observations, and therefore it is legitimate to call it Beer's Law. Furthermore, Don is missing a key point, too. In order for Don's definition that "the intensity of a beam of light passing through a uniform matrix is reduced as an exponential function of distance" to hold, the light wave-fronts have to be perfectly plain and parallel. However, no beam of light carrying a finite amount of energy can be perfectly collimated. Therefore, even to Don, if he doesn't accept Beer's Law as we chemists understand it, then he cannot accept his own Bouguer-Lambert law, for the same reason (because rays at an angle to the beam will traverse a different pathlength through the sample than rays parallel to the beam axis). Therefore everybody, physicists and chemists both, accept the fact that the physical equations describe an ideal world, and that the real world, and our mathematical descriptions of it, are approximations to each other, approximations that work better the more closely the specified conditions for those approximations are adhered to. I don't know exactly when chemical analytical spectroscopy was first developed, but it certainly came into its own during World War II. So we see that in the real world, at least, Beer's Law has been around for a long time and been used very successfully during that period. I think, though, that there's a piece missing in the specification of it. First of all, Don has to recognize that chemists prefer to exponentiate the expression of the exponential reduction of intensity, in order to define a quantity proportional to the concentration of the absorbing species (which we call Absorbance). I think there should be no problem about this, since it is a straightforward mathematical operation on the data. Chemists then extend the use of Beer's Law to saying that the absorbance of a sample at a given wavelength is the sum of the absorbances of the individual absorbance in the sample at that wavelength. This is equivalent, in Don's exponential equation, of saying that the epsilon term, representing the absorptivity of the material, is equal to the sum of all the epsilons of the different components of the material, at that wavelength. It is this summation of the physical origins of the absorbances that is the basis for all the mathematical constructs and manipulations that we perform on the data arising from it. \o/ /_\ | |||
| Gabi Levin (gabiruth) Junior Member Username: gabiruth Post Number: 10 Registered: 5-2009 |
Hi Donald, Well, it appears to still draw attention - if the Beer lambert law was totally false, then we would not be able to extract anything. The Beer Lambert law does rely on understandable physical phenomenonon - the vibrations follow clear physical laws, that even if we can not express them precisely in our limited math capability, their root nature exists. The laws of absorbing electromagnetic energy at a given wavelength still hold, even if we can not express all the possible inetractions in a manageable mathematical formula. One aspect of these laws is that the intensity of light that passes through a substance is also exponential to the negtaive of the absorption coefficients of the substances present at the given wavelength, and the concentration of these substances along the path of light. Therefore to say that the Beer Lambert law is false is to say that these physical laws do not apply to the intercation of the electromagnetic radiation in the NIR region with the substances. Since I trust that you do not mean that these laws do not apply, therefore, we can accept that the Beer law applies, but the exact mathematical expression is unknown to us due to the complexity of these interactions. It is not that the law, is false, it is that our math is not developed sufficiently. However, Chemometrics is an honest attempt to put our limited math to work for us in a favorable way. Therefore, to claim that loading weights are false and are dangerous to use is like telling us that - hey guys, your torch light is too weak to show you a perfect way, thus you'd better stay in the dark and feel your way with your hands. Because that is exactly what is prescribed by saying that chemometrics shall not be used unless you can set up a world wide, universal experiment that will give you the assurance that your measurement is correct. I prefer to use the torch light even if only to show me where not to go sometimes (when a regression seems fine but can not be substantiated by the physics that governs the absorption process) or where to increase my effort to establish a robust enough substantiation by proper statistical supporting evidence. Thanks, Gabi Levin | |||
| Donald J Dahm (djdahm) Advanced Member Username: djdahm Post Number: 25 Registered: 2-2007 |
"or else the Beer-Lambert law is false as well." Of course, Beer's law is "false". Even if you force the mathematics to extract the spectum of the component of interest (a step that I think is a very wise one), the extracted spectum will depend on the concentration range of the analyte in the sample set (and the concentrations of the other absorbers as well). Given the teaching we all remember, that a "law" is a summary of observation for which there is no known exception, we should call it "Beer's approximation". For homogeneous samples, it holds only to the extent that the "matrix" (or environment of the absorber) does not change with the concentration of analyte. For scattering samples, it does not hold at all. It is NOT just the Chemometrics that is suspect. It is our treatment of the data as well. [The law that is really the "LAW" is the Bouguer-Lambert law, which says that the intensity of a beam of light passing through a uniform matrix is reduced as an exponential function of distance.] | |||
| Ralf Marbach (ralf) Junior Member Username: ralf Post Number: 10 Registered: 9-2007 |
All, This discussion about melamine, which seems to have calmed down now and hopefully has put even the most superficial reader into a contemplative mood, has �nicely� shown the confusion and fragility affecting much of chemometrics today (dancing around the golden PLS calf � looking at the calf critically from time to time, but always dancing around it). I thank all participants for advancing my course. The melamine example is another good opportunity for me to spread my message: change is needed. What we had here is an application example, difficult in practice but simple in principle, where a number of senior chemometricians can not agree on a concept as fundamentally important as selectivity and how to test for it. The goal, �good results�, is somehow clear to everyone and shared, but how to get there and what tools to use to prove arrival is contented. How can this situation be fixed? � All it takes is to acknowledge, and live by, the following simple fact: �ALL calibration results, from measurement applications OR from statistical analysis applications, univariate or multivariate, inverse model or classical model, etc. etc., can be mathematically written in this form: b = invS*g / (g� * invS * g) where b is the regression vector (�b-vector�) in units of, e.g., [ppm/Absorbance]; g is the spectral signal vector used in the calibration [Absorbance/ppm]; and invS is the inverse of the covariance matrix of the spectral noise used in the calibration [Absorbance^(-2)].� Starting from this basic fact, discussions can be precise. If the analyst controls the shape of g and, based on spectroscopic expertise, can demonstrate that this is the correct spectrum of the analyte of interest, then it�s a measurement. If not, it�s statistics. The estimate of the spectral noise matrix used in the calibration has nothing to do with this difference. It may have an effect later, in Step 2 of the proof of selectivity, but is has no effect in Step 1, which is the issue here because it defines the borderline between the two kettles of fish. In PLS, the signal vector implicitly used in the calibration is, g_PLS = X�yr / (yr�*yr) where X is the matrix of calibration spectra [Absorbance] and yr is the vector of reference values [ppm] (both mean-centered if the calibration is centered). If, out of habit, you have been crazy enough to apply PLS or another statistical calibration method to a measurement application (instead of directly measuring and be done), and now wonder whether the result can somehow be pulled back to the side of measurement or whether selectivity has been lost in the indiscriminate search for correlation and the result is now �only� on the side of statistical analysis, look at THAT spectrum, g_PLS, to see whether you �measure� something interpretable or whether you merely use �something� for correlation. Re. factor interpretation � Interpreting the spectral shapes of loading vectors, or b-vectors for that matter, is risky and unnecessary, actually, in BOTH measurements and in statistics. These shapes are affected, and in the NIR often dominated, by the �invS� used in the calibration and thus tell little about the �g�. Numerical ill-conditionedness and algorithmic constraints come on top, making spectral interpretation of loading vectors miserably non-quantitative in practice even in moderately challenging cases. We all know it, it�s flimsy stuff. Fortunately, there is no need for it. � One can directly look at the �g� spectrum used in the calibration, interpret its spectral shape, and determine whether the first step of the proof of selectivity (measuring the right thing) is passed or not. This is so much nicer than trying to peek at g through long narrow �tubes� in multidimensional space (factor projections). Nobody out there can seriously argue that looking at loadings is nicer than looking at the spectrum-used-as-the-signal directly. Ill-conditioned systems are best judged by looking at the inputs, g and S, not by looking at any of the outputs. Ralf VTT Optical Instrument Center MTT Multantiv | |||
| Bengt Nordlund (bengtn) Member Username: bengtn Post Number: 15 Registered: 11-2003 |
Hi There have obviously been some discussion already before the article was published, and this discussion here shows that there is a need to have a discussion about scientific results that are published. Just because something have been accepted to be published does that not automatically mean that the results are accepted by the scientific community. Graemes argument to publish the article are understandable and I also hope that several other will try to repeat the invetigation and publish their results whatever the results may be. This is of course important as it is not only a scientific discussion it has also to do with peoples health. The technical aspects that have been discussed here is of course also a valuable help for those that make similar investigation on how to validate their findings in a proper way. Best regards Bengt | |||
| Richard Kramer (kramer) Intermediate Member Username: kramer Post Number: 17 Registered: 1-2001 |
[I agree that statistical validation is a critical process for such methods- however, even this process has some level of subjectivity: how do we determine (and express) the limited condition/validation space that is used to interrogate the model?] Chuck, you are quite correct to point out, in this way, that assembling an adequate validation data set is not only the most critical step, but can also be the most difficult step. Doing it right is a tall order. It requires assembling a validation data set which is statistically significantly representative of all future unknown samples on which the calibration is intended to be used. To get it right every time would require a superhuman degree of prescience. However, I disagree that it need be subjective in that, should the validation set be inadequate, this will quickly become apparent when unacceptable numbers of candidate unknown samples fail to qualify for analysis by the calibration. In such cases, this is an unmistakable indication that the validation set is inadequate and needs to be supplemented. I do agree that appropriately balancing, a priori, the requirements for an adequately comprehensive validation set against the realities of the time and costs involved can involve a certain amount of what might appear to be "subjectivity," but I'd prefer to say, instead, that it requires a certain amount of insight and experience. That's where a consultant experienced in the field can provide valuable assistance. ;) | |||
| Richard Kramer (kramer) Intermediate Member Username: kramer Post Number: 16 Registered: 1-2001 |
[Still Richard has to prove that there are many critical cases where loading vectors did not prdeict good relevance but good empirical work finally substantiated a good, reliable, high quality correlation that is robust to withstand the possible small variations in spectra due to minute changes in the chemical over all make up of the matrix.] I do not accept the validity of the premise that this must be proved. Also, "many" is not exactly a scientifically precise word. I have seen the occasional application where the appearance of the loading vectors was encouraging but the method failed to validate to the required level. In some cases, the project was simply killed before the reasons for the failure were understood. In some cases, it was determined that the calibration was misdirected by an irrelevant correlation (in some of those cases the method was adjusted successfully, in others of those cases it was not possible to eliminate the inconvenient correlation.) In some cases, it was determined that the instrument in question was functioning as an expensive pseudo random number generator and the initially promising results were neither stable nor exploitable. In some cases, there simply was insufficient SNR to permit validation to the level required for the purpose in question. In all of these cases, the determining indication came from the validation work. In none of these cases did the determining indication come from the loading vectors. I've also seen cases where the loading vectors defied interpretation yet the method validated solidly and was deployed successfully. As long as each candidate unknown is screened to determine if it was adequately represented in the validation data set, it is completely scientifically justified to rely on the results of that calibration. (Note that some people erroneously qualify candidate unknown samples against the training data set rather than the validation data set, and that other people don't qualify candidate unknown samples at all, but that is a topic for another day.) | |||
| Gabi Levin (gabiruth) Junior Member Username: gabiruth Post Number: 9 Registered: 5-2009 |
Well, while I can definitely produce a good number of examples where the evidence provided by the loading vectors was substantiated by good statistical and empicrical work - and no surprise at that, let us examine if we have one example where loading weights predicted that statistical work will substantiate it, but that fianlly such work failed to do so. Because if Richard is right about the claim that loading vectors are meaningless, or even worse "dangerous" then we should expect to see many cases where potentially good correlations (based on loading vectors), that should have been fully substantiated by good empirical statistical work were fianlly found to fail as a result of elaborate and extensive staistical work. I have not seen one such a case, in hunders of cases where I was involved in analyzing regressions using loading weights.Richard can of course always claim that if one did more statistical empirical work the failure may still come. Possibly, but if e.g., after thousdands of distillation batches, regressions that were determined to be valid early in the game based on loading vectors are still valid, do we need more empirical work until such failure occurs? Indeed, there is such case, where loading weights would predict that staistical work would substantiate it - but it does not and this is melt flow index in polypropylene. However, here we are looking at a property that only partially depends on chemical structure, and at the same time depend on other parameters that are not chemical. In such case, it is to be expected that loading weights will not tell the whole picture. As a matter of fact, 10 years ago, I was involved in such case and after analyzing the problem of melt flow I told the people involved, that there will never be a UNIVESRAL calibration for all the different grades of PP - and that to derive any useful regression between NIR and melt flow index would require dividing the different "GRADES" of PP to sub-groups where the melt flow index is a function of the chemical structure only within that group. Still Richard has to prove that there are many critical cases where loading vectors did not prdeict good relevance but good empirical work finally substantiated a good, reliable, high quality correlation that is robust to withstand the possible small variations in spectra due to minute changes in the chemical over all make up of the matrix. In fact, I have seen such cases where initailly the staistics shows good potential, but scientific evidence did not show good relevance - and eventually when sufficient empirical work was done, the regression fell to pieces. Therefore, the duty of proving that loading vectors are dangerous and meaningless rests with those who claim it is. I have the evidence that it is very useful, can help people focus their work in good direction, discard options that may end up with useless data that leads to nowhere and even worse, lead people to rely and expect from NIR more than it can possibly give. If anything is dangerous, is concluding from a set of samples that a measurement is possible, without verifying that it has scientific merit, and then investing huge effort in that direction just to find out that when they have sufficient data, it all falls apart. I have also witnessed people trying almost everything to salvage huge amount of work just because it is difficult to admit that all the effort was in vain. If this is not dangerous, what is? And finally as I said already, please do not get confused, even the best case where loading weights show strong relevance, still needs a very good, reliable comprehensive enough empirical staistical work to do the final validation. But at the same time, do you want to spend lots of energy where it is quite clear that the probability of coming up with a robust method based on empiric foundations only is low. Gabi | |||
| Charles E. Miller (millerce) Junior Member Username: millerce Post Number: 10 Registered: 10-2006 |
This is an interesting and relevant discussion. There are several issues at play. I have a few comments to add: For detection problems, one often uses the Receiver/Operator Characteristic (ROC) curve for QA and optimization. This curve shows the detection performance at varying thresholds in the 2D space defined by the true positive rate (= sensitivity) and the false positive rate (= 1-specificity). The extremes of the ROC curve are defined by two �trivial� cases: one where the detection threshold is set so low that everything is called �positive�, and the other where it is set so high that positives are never obtained. The optimal threshold is somewhere in between, and is partly defined by the acceptable levels of false positives and false negatives (i.e., risk assessment). Although this approach to detector optimization can be very effective, there are some challenges involved: - Statistical optimization requires an estimate of the �acceptable� false negative rate for the problem; politically, this can be difficult, if not impossible, to obtain - Detectors based on multivariate modeling typically involve more than one �threshold� to tune (Q/residual, T2/leverage, others�), thus leading to a more complex multivariate optimization problem, and - As Richard notes, one must interrogate the detector with an �appropriate� set of validation conditions in order to generate these curves. I agree that statistical validation is a critical process for such methods- however, even this process has some level of subjectivity: how do we determine (and express) the limited condition/validation space that is used to interrogate the model? Regarding the interpretation of inverse (PLS, MLR, SVM..) models, this suffers from both the propensity for user error (over-zealousness) and lack of the ability to express results in a precise manner that is required for regulatory/legal purposes. As a result, it�s very difficult to allow its use as a primary quality assessment tool for such models. However, I wouldn�t go so far as to call it ��meaningless, academic, window dressing�. Precise mathematical expressions relating model parameters (loadings, scores, weights..) to regression vectors for PCR and various PLS algorithms have been published in many chemometrics texts. Also, I have tried to show over the years that if one is �armed� with understanding of the mathematical properties of these model vectors, one can indeed obtain tangible confidence in a method�s efficacy. Indeed, I�ve found this knowledge to be neither meaningless nor inconsequential to managers and decision-makers, especially those who have chemistry/physics/spectroscopy experience! Finally, as Ralf and others have noted in the past, direct �Beer�s Law� type models have the appealing property of essentially �forcing� one of the �loadings� to BE the spectrum of the target analyte: thus reducing the need for such interpretation. All for now- Cheers, Chuck | |||
| Richard Kramer (kramer) Member Username: kramer Post Number: 15 Registered: 1-2001 |
Gabi and I seem to be talking past each other. My point is that there is NO justification to rely on the "evidence of loading vectors" as indicative of anything. Doing so is dangerously erroneous. Just because the loading vectors appear to be reassuring in shape or form does NOT prove that the calibration is exploiting a stable, reliable correlation. There is no justification for looking to the loading vectors for "scientific evidence" that a calibration is proper since there is no way to explain, based on first principles why a given set of loading vectors results in a given set of calibration coefficients. Concepts such as net analyte signal can be useful since if the NAS goes to zero, that would indicate that the calibration isn't possible, but direct inspection of the loading vectors does not yield any "scientific" conclusions about the correctness or incorrectness of the calibration. The only available "scientific evidence" that an empirical calibration is suitable for its intended purpose is the scientific evidence which comes from appropriate, comprehensive validation. The essential element isn't inspection of the loading vectors, the essential element is to spend sufficient time and resources to adequately demonstrate that a given measurement of an unknown sample may be validly used to estimate the properties of that sample. Everything else is unscientific window dressing. | |||
| Gabi Levin (gabiruth) Junior Member Username: gabiruth Post Number: 8 Registered: 5-2009 |
It appears that Richard has completely forgoten what I wrote in my initial response - what I wrote was that without this evidence of loading vectors I don't care to continue and rely on statistical empricial success in establishing a correlation - but I also said that with that condition satisfied I will accpet the latter part, which includes the sufficient emprirical substantiation required to satisfy the statistical issue. I never intended and never said that the loading vectors are sufficient by themselves - I said that when they are lacking I will not waste my time going further - in situations when life threatening exist - as I said, if one wishes to rely on empirical method to determine to a degree that satisifies him how much Vitamin C is present in orange juice - he is welcome, but not when it is babies food. I hope that Richard will finally accept that without the "scientific" evidence the statistics don't necessarily prove anything, and will be useless in many cases. Besides - the argument that loading vectors don't prove anything is fundamentaly wrong - or else the Baer Lambert law is false as well. I agree that for safety and health reasons we shall not relay on them solely, but that dos not mean thay are deception. Gabi Levin | |||
| Richard Kramer (kramer) Member Username: kramer Post Number: 14 Registered: 1-2001 |
[It is an educated statistics - in the sense that it can teach us what is being used for the statistics analysis and we can determine if what is being used for the statistics is relevant or not. If it is relevant, then it will hold water under variations that are expected in naturally occurring products as well as chemically synthesized products. ] I disagree strongly with this viewpoint expressed by Gabi. It is not only in error, it is DANGEROUSLY in error. It suggests that by some sort of "proper editorial review" we can certify an empirically derived calibration will "hold water." The reason this is so dangerous is, perhaps a bit subtle. One aspect of the problem is the phenomenon of looking until an acceptable result is found and then stopping. Examining loading vectors and such in order to identify "reassuring" spectrally sensible features which "explain" why the calibration works is a prime example of a case where the risk of this dangerous phenomenon is high. Just because one might find such features in an empirically derived calibration does not mean that those features, though the may be reassuring to the human eye-brain pattern recognition engine, have anything to do with a reliable, exploitable correlation. To suggest that when we find such reassuring indications we can relax and release those calibrations for use in critical health and safety regulations is dangerously superstitious. Just because we might notice some features in the loading vectors which make us believe that the correlation(s) embedded in the calibration are based on reliable spectral features does not make it so. The calibration remains an empirical calibration whether or not we look for and find such "reassuring" features. Accordingly, we must continue to continuously (redundancy intentional) treat the calibration with every bit of suspicion which is required when exploiting an empirical calibration. That means a sufficient degree of validation with a comprehensive validation data set. That also means qualifying each and every measurement of an unknown sample to determine whether the validation data set used to qualify the calibration was sufficiently representative of the measurement made on the unknown sample, or whether the measurement of the unknown differs in a statistically significant way from the measurements comprising the validation data set. That also means ongoing validation of the calibration at appropriate intervals an to an appropriate degree. Personally, I always look at the loadings and such things, but I NEVER rely on the results of the examination to draw conclusions about the suitability of a calibration for the intended purpose. That can only be established by appropriate initial and ongoing validation. When the calibration is empirically derived this is not only the minimum which must be done for each sample, it is also the best which can be done. Examining loading vectors and other such things can be interesting and fun, but it does not allow us to conclude or say anything of any value about the reliability or appropriateness of the calibration. Reliability and appropriateness can only be demonstrated by proper validation at the time the calibration was developed, together with an appropriate degree of ongoing validation as the calibration is deployed and used. This is always true for every calibration whether or not we look for and find "reassuring" features in the loadings. When a calibration is properly validatated initially and on an ongoing basis (and proper validation includes qualifying each and every measurement of an unknown sample), the results of that validation is the only thing which matters. To express it using Howard's terminology, proper initial and ongoing validation is the ONLY scientific basis to justify the use of an empirical calibration for any purpose no matter how critical or inconsequential that purpose might be. | |||
| Ian Michael (admin) Board Administrator Username: admin Post Number: 23 Registered: 1-2006 |
I think this is an important discussion on many levels. In case those interested do not have access to the paper through a JNIRS subscription, I have made the paper freely available at: http://www.impublications.com/nir/abstract/J17_0059 I will probably revert it back to its normal state for subscribers only in a few weeks, but in the meantime it is downloadable by anyone. | |||
| Gabi Levin (gabiruth) Junior Member Username: gabiruth Post Number: 7 Registered: 5-2009 |
Richard claims that ssuing loading weights or vectors is an excercise in self deception. This is a statement I certainley disagree with. If this is true then the basic Baer Lambert law is also a deception. The loading vectors are a critical facet of the PLS or similar method. Before the PLS we used MLR which was trying to use wavelengths as specific to the measured constituent as possible. PLS is not just a statistical method. It is an educated statistics - in the sense that it can teach us what is being used for the statistics analysis and we can determine if what is being used for the statistics is relevant or not. If it is relevant, then it will hold water under variations that are expected in naturally occuring products as well as chemically synthesized products. If it is not relevant, then the statistics is bound to fail, at some time, and it is not possible to tell in advance when and why it will fail. Therefore, for determinations of e.g. ash in organic matter, that bears no life threatening results in the event of fialure it is fine - go ahead and use it. But for malamine in babies food and simlar applications - this is an absolute NO situation. One more point - NIR is being used successfully to measure density of polyethylene, gasoline, etc. Of course density does not have spectrum of its own and yet the correlation there is definitely based on chemical foundation - density is a function of the chemical structure, chain length, etc. These have spectral features. However, at the same time, melt flow index of polypropylene has eluded chemometricians for many years. Although it depends on the length of chains and chemical structure, it is also a function of other parameters in the process which do not have a reflection in the spectrum. Therefore, it is an ambigous property that to date is not "safely" measured by NIR. Gabi Levin | |||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 249 Registered: 9-2001 |
Gabi - I've attached a .PDF copy of the column to this message. Well, I was going to attach it, but the discussion board said it was too large to upload. But there's another copy of it in the .ZIP file with the data, so you can get it from there. Since the data file from the shootout was too large to include as an attachment (even ZIPped up), and I don't seem to have your e-mail address, I uploaded it to yousendit.com. The link below should allow you to retrieve it: https://www.yousendit.com/download/Y1RyZm1YTkFrUm52Wmc9PQ The samples were run on two instruments, and the original Shootout instructions included, as part of the exercise, to see how well a calibration could be transferred between instruments. In analyzing the data for the column, I found that the calibration based on the random data transferred at least as well as those based on the actual reference data. Richard: In the light of your comments about validation being all that's needed to justify confidence, I would be interested in your interpretation of these results. Also, "justify by scientific means" are not my words, they're the FDA's, and they are part of the regulatory requirements. There are, after all, only a few ways to do these things: science, magic and miracles. Since there haven't been any documented miracles since the time of Moses, that doesn't leave us too many choices. I know which one I prefer. Imperfect as it may be, it's still the best we've got. We shouldn't throw it away lightly. \o/ /_\ | |||
| Richard Kramer (kramer) Member Username: kramer Post Number: 13 Registered: 1-2001 |
Howard mentions "justified by 'scientific' means. Gabi mentions "specificity" Ralf mentions measurement being superior to statistical analysis, by which I take his meaning to be an expression of a preference for "direct measurement" (whatever that may be) over empirical calibrations such a chemometric calibrations based on spectral measurements. The point I'm trying to make is that, when the calibration is empirically derived by techniques such as PCR or PLS, it is meaningless to assert that one understands and can prove how and why the calibration works by such things as examining the loading vectors. Doing so is an exercise in self deception. Empirical calibrations can only be validated empirically. Otherwise they would qualify as calibrations based on first principles. It is essential to recognize this fundamental fact and to understand its implications so that one may approach the application and validation of the calibration appropriately. The only way to validate to the required, user specified degree, is to challenge the calibration with a sufficient number of sufficiently representative validation samples. Anything else is meaningless, academic, window dressing. | |||
| Ralf Marbach (ralf) Junior Member Username: ralf Post Number: 9 Registered: 9-2007 |
All, This discussion about melamine in milk powder is a prime example about what�s wrong with chemometrics today. There is a fundamental difference between measuring something and making a statistical analysis of something, and this difference is not widely enough acknowledged. Only measurements can be selective, statistical analyses can not � that is in fact the dividing line, by definition. Proof of selectivity a.k.a. specificity comes in two steps. First, you need to show that you measure the right spectral signal (of the analyte of interest) and, second, you need to show that you measure the right signal correctly. Statistical analyses can not be selective because, by definition, the first step is not taken. They can only correlate, not measure. Measurement is preferable over statistical analysis anytime, any way you look at it. It should be used whenever possible. In the melamine example, measurement requires that the melamine response spectrum, in units of, e.g., [Absorbance/ppm], is determined (Step 1). If the absorbance features of the response spectrum turn out to be so small that it�s lost in the hardware noise, measurement is not possible. Then, and only then, should statistical correlation be considered � chemically unspecific but potentially still useful if the amplitude of those larger absorbance features that hopefully exist and happen to correlate with the melamine concentration will continue to correlate in the future. So, if you like, let PLS or some other statistical algorithm search for correlations and spend a year or two trying to validate their longevity. BUT don�t even think of trying to prove �selectivity� in this case � it is not defined, you can only demonstrate a �high chance of continuation of correlation� or something like that. As long as the community at large and the regulators in particular don�t clearly distinguish between these two different kettles of fish, the current confusion and waste will continue. But it�s easy: Richard�s arguments make sense when the issue is statistical analysis; Gabi�s arguments make sense when the issue is measurement. Me personally, I prefer measurement a million times, so I am kind of allergic against the combination of �NIR� and �ppm.� But that does not mean that all statistical analyses are in vain. They are all black holes gulping time and money, but sometimes they can produce useful results. Ralf VTT Optical Instrument Center MTT Multantiv | |||
| Gabi Levin (gabiruth) Junior Member Username: gabiruth Post Number: 6 Registered: 5-2009 |
Hi Howard, Thanks, no I did not read, and I would be happy to try the data set. Gabi | |||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 248 Registered: 9-2001 |
Gabi - you may be interested in reading (or rereading, if you've seen it before) my Spectroscopy column: Spectroscopy, 22(6), p.20-26 (2007), where I did essentially the same exercise, using the publically available dataset from the Software Shootout at the 2002 IDRC (I just checked and it seems to be temporarily unavailable from the IDRC website, due to the site undergoing revisions to prepare it for the next IDRC), but I can send you a copy of that data set, if you'd like to try working with it. \o/ /_\ | |||
| Gabi Levin (gabiruth) New member Username: gabiruth Post Number: 5 Registered: 5-2009 |
Thanks, Howard, well said. The requirement for "scientific" means is not just an arbitrary requirement on the part of government clerks that wish to put sticks in the wheels of the chemometric wagon. The requirement is based on the fact that - and in particular in complex systes such as baby food which contain substances derived from natural products, such as milk, starch etc. All these are not pure, perfectlty synthesized chemicals. Therefore, when we need to calibrate for a minute contituent, with a given set of calibrations samples, and more so for a small one, the chemometrics with its powerful ability to "track" and "find" correlations between spectral variations and a set of reference values necessitate that we verify by scientific means, such as loading weights that what we measure is really what we think we measure. I can vouch myslef, that I had events where initially a calibration was obtained for a low level constituent, but the loading weights did not support the relevance to the spectrum of the measured constituent, and finally the calibration did not hold water when faced with additional variations occuring in the product. If we continue to treat chemometric as a mathematical wizard without requiring the substantiation by scientific evidence, we will repeat the same mistakes that led so many people to regard it with great suspicion. As I was once told by one of my early teachers of practical chemometrics - give me 10 spectra of anything and 10 arbitrary values and I will produce a very good calibration curve. Thanks again, Gabi Levin | |||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 247 Registered: 9-2001 |
Richard - I know you feel strongly about your position that the only factor of importance in validation is accuracy. But medical applications are different than industrial applications: each and every one involves the possiblity of someone dying, if a mistake is made. You can't equate that cost to the monetary costs that are normally considered in industrial applications. The regulations (in the US, at least) specifically state that analytical methodology must be justified by "scientific" means. I don't know all how that's defined by the FDA, but given what I've read about it in other legal contexts, it's a lot more than just someone's intuition as to whether they feel that a given study is "scientific". Therefore I agree with Gabi, but on different grounds (I also agree with his grounds, BTW. I've said for a long time that when I go into a drugstore to get some Lipitor, or even aspirin, I'm very grateful that the FDA is very conservative and insists on multiple verifications that what I get is what it's supposed to be). My other grounds for agreeing with Gabi is that chemometrics does not exist in its own universe, separate from the one that all the rest of us, and all the rest of science, exist in. And until we stop treating it as separate, and can show that it is really is part of the larger scientific framework, chemometrics will never be accepted by the larger scientific community except as a useful "trick" for getting results. This was demonstrated at my Pittcon talk. In what was almost the simplest possible chemometric experiment, the chemometric results were extremely well self-consistent, but disagreed markedly with the known correct answers. At that time I had not yet been able to explain the discrepancy (in fact, I still have not). A small group of us are working on that; I'm hoping to have some results by EAS. This is all to put chemometrics into the main stream of science. Otherwise, as I said, it will not be accepted - nor should it be. \o/ /_\ | |||
| Gabi Levin (gabiruth) New member Username: gabiruth Post Number: 4 Registered: 5-2009 |
Hi Richard, While I tend to agree with you that the ultimate goal is to prove the ability to predict the presence or lack of presence of melamine we can not ignore, nor should we, that when the FDA is evaluating any analytical method, not only NIR, it looks for a feature of the method called specifity - which means - prove to me that the analysis is specific to that constituent that is being measured, and that you are not by some combination of conditions prevailing in the calibration set measuring something else, and that if at some other time these conditions will not prevail the method will fail. In the case of melamine this has higher importance due to the life threatening aspect. Therefore, and since no one can examine all possible conditions of adulteration, specifity must be proved as part of the evaluation of the NIR method. The argument that it is sufficient that suspect samples will be subjected to other methods as well, does not hold, because our concern is not suspect samples, it is samples that are false negative, i.e., are not detected as suspect due to the lack of specifity. Many thanks to all Gabi Levin | |||
| Richard Kramer (kramer) Member Username: kramer Post Number: 12 Registered: 1-2001 |
[1. Any regression of melamine levels must prove beyond a shadow of a doubt that it relies on spectrum details that are unique to melamine. For that, a detailed well substantiated analysis of loading weights (from Unscrambler or equivalent chemometric package) is required. Without it I will personally not believe any regression.] Gabi, I do not believe this first point is correct. All that need be demonstrated is that the presence of melamine above a specified level will be detected with a specified degree of confidence. Inasmuch as the calibration is, by nature, empirical, it is both futile and unnecessary to speculate on the reasons that the detection is reliable so long as the reliability is properly validated (see, for example ASTM 2617). Furthermore, if the spectroscopic method is used to screen samples, and if any samples which are flagged are subjected to confirmatory testing by another method (which would generally be standard procedure), there is no need to try to demonstrate that the spectroscopic method flagged melamine and not some other harmless constituent. [2. Once such distinctive evidence is established, the next step is a thorough validation that needs to be divided in two - one by using contaminated samples to known values and proving that there is no case where detection is missed, and the other by obtaining - and hopefully it is possible - contaminated samples from real production batches and showing zero missed detection.] "In no case" is both unrealistic and impossible to demonstrate statistically. The best which can be done is to demonstrate that the method is validated to perform to a specified degree of confidence. The appropriate degree of confidence (for a given detection threshold) is up to the users and/or regulators to specify. It would generally be established by a risk-based approach. [3. Once this has been done sucessfuly it will be required to determine the minimum sample size from production batch that needs to be scanned very carefully to verify that in the event that melamine is not uniformly distributed (and it is probably not uniformly distributed due to the low percentage) within the batch volume its presence will still be detected.] This sampling issue is not unique to a spectroscopic detection method. Indeed, spectroscopic methods, which can generally measure larger aliquots than methods such as HPLC, might offer sampling advantages over such other me Also, since the purpose of adulteration appears to be deceptively increasing the results of standard protein measurements, to be successful the adulterants would have to be uniformly distributed with respect to the sample size used for those protein tests. Acordingly, an alternative screening method would need to demonstrate a specified degree of detection reliability with respect to samples of that size. | |||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 246 Registered: 9-2001 |
Gabi makes a good point. This type of situation has not been ignored. The FDA, USP and their European, Japanese and other countries' counterparts have regulations, and standards, dealing with the requirements for analytical methods. These apply to all analytical methods, not only NIR. I'd been involved several years back, in an exercise to create a NIR method that could be approved under the general guidelines and requirements. This has been published in a pair of papers: "Validation of a Near-Infrared Transmission Spectroscopic Procedure, Part A: Validation Protocols"; Mark, H., Ritchie, G.E., Roller, R.W., Ciurczak, E.W., Tso, C. and MacDonald, S.A.; J. Pharm. & Biomed. Anal. 28(2), 251-260 (2002) "Validation of a Near-Infrared Transmission Spectroscopic Procedure, Part B: Application to Alternate Content Uniformity and Release Assay Methods for Pharmaceutical Solid Dosage Forms"; Mark, H., Ritchie, G.E., Roller, R.W., Ciurczak, E.W., Tso, C. and MacDonald, S.A.; J. Pharm. & Biomed. Anal. 29(1-2), 159-171 (2002) While the papers can be read for their content, the more important part of them, in the context of this discussion, is the references to the FDA, ICH, USP and other regulatory documents. The requirements can be summed up in a key phrase: it's up to the proposer of a new method to demonstrate to the FDA that the method in question is "suitable for the intended purpose". The proposer must define what constitutes "suitablity", and then convince the FDA that both his definition and analytical method are satisfactory, and that the method in fact meets the criteria set forth for being suitable. \o/ /_\ | |||
| Gabi Levin (gabiruth) New member Username: gabiruth Post Number: 2 Registered: 5-2009 |
Hi guys, I read the string - I did not read the article. As always, (and I am still a Brimrose guy so this has not changed although I am asscoiated with another business on a small scale) I am trying to be a practitioner - detection of melaine in baby food at below 0.1% - and if I am correct well below the 0.1% is not a theoretcial issue. It is first and foremost a regulatory issue. At stake are babies lives and well being. Anything less than 99.99999% or so confidence that melamine will be detected at the required level will not satisfy the health and the consequential legal aspects of the problem. Therefore, on the road to application of NIR for that purpose we have to look at: 1. Any regression of melamine levels must prove beyond a shadow of a doubt that it relies on spectrum details that are unique to melamine. For that, a detailed well substantiated analysis of loading weights (from Unscrambler or equivalent chemometric package) is required. Without it I will personally not believe any regression. 2. Once such distinctive evidence is established, the next step is a thorough validation that needs to be divided in two - one by using contaminated samples to known values and proving that there is no case where detection is missed, and the other by obtaining - and hopefully it is possible - contaminated samples from real production batches and showing zero missed detection. 3. Once this has been done sucessfuly it will be required to determine the minimum sample size from production batch that needs to be scanned very carefully to verify that in the event that melamine is not uniformly distributed (and it is probably not uniformly distributed due to the low percentage) within the batch volume its presence will still be detected. All said, the above is not to discourage for the sake of discouraging, it is for the sake of realizing what we are up against when dealing with a life and death issues. Many thanks, Gabi Levin [email protected] | |||
| Graeme Batten (graemeb) New member Username: graemeb Post Number: 2 Registered: 5-2009 |
The attachment I posted did not seem to be easy to access so I am trying the direct approach..Graeme Before the paper by Chenghui Lu et al. was submitted to JNIRS I was aware that people in the milk industry were of the opinion that the level of detection of melamine in milk or milk powder was not good enough for routine use. I am not surprised that the paper by Chenghui Lu et al., reporting determination of melamine in milk powder published in JNIRS 17(2) 59-67 (2009), has attracted concerned comments from several NIR scientists who do not accept that NIR spectroscopy can detect components at ppm levels. Both I and the referees were of the opinion that the Lu et al. paper is a useful first step towards a reliable calibration and with this reservation I made the decision to accept it for publication. The Chenghui Lu et paper should be read in conjunction with the paper by Lisa J Mauer et al. �Melamine detection in infant formula powder using near- and mid-infrared spectroscopy� published in J Agric & Food Chem, 2009. 57(10) pp 3974-3980. These authors also report detection of melamine at the 1ppm level using NIR. In both studies the sample sets are minimal. The authors of both papers used the region of the 4900-5300 cm-1 region of the spectrum which relates to melamine - NH stretching and bending. The use of cross validation and LS-SVM analyses may not be the ultimate tests of the reliability of the melamine calibration models but if different approaches achieve similar outcomes that can be regarded as giving some confirmation. The authors of both papers included statements which effectively say more work is needed to establish reliable calibrations for routine applications. In my opinion one or two studies do not answer all the questions about a topic. Good science must stand the test of time and that involves a) Studies designed to see if the original findings are obtained if the experiment is repeated; or b) Studies designed to disprove the original findings. I will be disappointed if the discussion on the papers by Lu et al. and Mauer et al. does not encourage further studies ( and the publication of those studies whatever the conclusions may be). | |||
| Graeme Batten (graemeb) New member Username: graemeb Post Number: 1 Registered: 5-2009 |
| |||
| Richard Kramer (kramer) Member Username: kramer Post Number: 11 Registered: 1-2001 |
I haven't read the article, but detection down to 0.1% is not inconsistent with work we've done on the question. | |||
| Bengt Nordlund (bengtn) Member Username: bengtn Post Number: 14 Registered: 11-2003 |
C. Lu , et al, j Near infrared Spectroscopy, 17, 59-67 (2009) Rapid detection of Melamin in milk powder by Near infrared spectrocopy. | |||
| Jerry Jin (jcg2000) Member Username: jcg2000 Post Number: 15 Registered: 1-2009 |
Why don't you post the source of that article so people here can read it before make a judgment? Best wishes. Jerry Jin | |||
| Bengt Nordlund (bengtn) Member Username: bengtn Post Number: 13 Registered: 11-2003 |
Hi I assume that most of you have read the article about measurement of melamin in milk powderby Chenghui Lu. I am a little surpriced that no one have questioned the result from that. Looking through this discussion forum it is easy the find recommendations to never look for concentrations below 0,1 % as that is not possible to analyse with NIR. Are there anyone that have tried to copy his method with succeful result ? Anyone else than me that question if it at all possible to measure anything at such a low concetration? Best regards Bengt N |
Comment from Editor-in-Chief, JNIRS