| Author | Message | |||
     Charles E. Miller (millerce) Member Username: millerce Post Number: 13 Registered: 10-2006 |
Don: thanks a lot for rabble-rousing; and Howard- thanks much for the historical perspective. If I may contribute to the rousing of rabble�. >> I think the �next generation� is becoming fairly well-schooled in Chemometrics, but attention to theory is decreasing.<< Perhaps. I strongly agree that an understanding of the theory behind the chemistry, physics and measurement science behind the data is not only useful but necessary to get the most out of practical NIR applications, regardless of the math used. I also believe that the usage of such knowledge is equally valid in either a �front-end-loading� manner (i.e. to optimize preprocessing and model building), OR (yes, Ralf) �downstream�, to interpret various model parameters. However, I think I understand why you�re concerned that reliance on chemometrics has eroded appreciation of the underlying theories behind the data. For one, many of the popular easy-to-use model-building tools (i.e,, PLS, MLR) do not require such knowledge, and thus don�t encourage the search for it. Also, I agree with Richard that even models generated by such methods can be effectively and safely deployed with sufficient due diligence. I can also understand why Ralf�s insistence on the �single-signal-vector� paradigm raises red flags with theoreticians and theory-savvy NIR practitioners, who are keenly aware of the molecular-level and light/matter dynamics of NIR spectroscopy. I, personally, don�t share Ralf�s self-promoting grim assessment of the state of chemometrics: despite what many might think, over the past several years chemometricians have developed a diverse repertoire of preprocessing and model-building tools, each of which can allow different types of theory/prior knowledge to be brought to bear to a problem under different conditions. Yes, the developers of these tools tend to be chauvinistic about THEIR method, but the user�s key is to understand that different methods are likely to be optimal for different sets of conditions, assumptions and project objectives. Best Regards, Chuck | |||
| Richard Kramer (kramer) Intermediate Member Username: kramer Post Number: 19 Registered: 1-2001 |
>>Understanding the chemical and physical environment of the analyte in the given application and then devising a measurement system with a time-invariant response spectrum is, in general, the key. Everything else, incl. chemometrics, is straightforward in comparison. Linearity is nice-to-have but is usually not the major concern in NIR spectrometry.<< I agree completely about invariance being essential and linearity being nice to have but not essential. As the discussion of Beer-Lambert law has explained, it is almost never a question of whether or not the response of the system is linear, it is, instead, a question of how non-linear the system is. If we were dealing with linear systems, then PLS calibrations requiring only a single factor would be the norm. The fact that such calibrations are quite rare is a reflection of the innate non-linearity of the systems we have to work with. However, I would NOT say that the goal is a static time-invariant response. Such a thing is seldom achievable. What is needed is a time-invariant response ENVELOPE within which the response resides. Practical calibrations need to be able to handle samples which wander around within the response envelope. It is also important to qualify every candidate unknown sample to ensure that it lies within the space spanned by the data set which was used to validate the calibration. It is the closest we can come, in a practical sense, to confirming that the response for that sample is likely not to have drifted outside the defined response envelope. | |||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 258 Registered: 9-2001 |
Ralf - what you're saying is true enough, but its a whole lot easier to say than to do. consider your point (2): "(2) Scattering effects (�physical matrix�) � these can affect the linearity AND the stationarity and, in practice, it is the latter that hurts quantitative measurement most. Changes in the effective optical pathlength are a major pain in NIR diffuse reflection measurements. People understanding optics and the Maxwell equations can still be creative here." Don has spent the better part of the last 10-15 years trying to address just that problem, and much as he may be embarrased by my saying so, he is probably the world's expert on that subject, for at least the last 20 years, and certainly at least since Harry Hecht tackled it, about 30 years ago. Don has pushed the limit of what is known about the problem of describing diffuse reflection beyond what anyone else had previously been able to do, but nevertheless eventually had to come to the conclusion that it's intractable. If you think there's a way to make further progress, then I recommend that if you haven't already, you read Don's book "Interpreting Diffuse Reflectance and Transmittance", NIR Publications (2007) and bring your suggestions to Don's attention. Knowing Don and knowing how hard he's worked on the problem, I'm pretty sure that if you can help him make more progress, I'm sure you'll find a warm reception. \o/ /_\ | |||
| Ralf Marbach (ralf) Member Username: ralf Post Number: 11 Registered: 9-2007 |
All, The remark Don made at the end of the melamine threat, in the short words of a physicist, was very important. The detail discussion below has not paid enough attention to this fundamental aspect, so I repeat it here in the words of measurement science. I hope this will re-focus the discussion. �For measurement purposes, the response spectrum of the analyte should have TWO properties. You like it to be (1) LINEAR and you really need it to be (2) TIME-INVARIANT.� A measurement is linear if the response spectrum of the analyte (in units of, e.g., [Absorbance/ppm] in the melamine example) does not depend on the concentration of the analyte [ppm]. The amplitude of the analyte�s measured [Absorbance] then scales linearly with the [ppm] in the sample. A measurement is time-invariant (a.k.a. �stationary�) if the response spectrum of the analyte does not depend on any of the, usually many, things/effects other than the analyte concentration that vary over time. In other words, assume the analyte concentration could somehow magically be fixed at some value: Would the analyte�s measured [Absorbance] then be constant over time? � Or would it change over time with changes in the chemical environment (matrix effects), physical environment (e.g. particle size), temperature, etc. etc.? I guess we all agree that the latter is a tougher problem in practice than the former. Personally, I don�t even mind a little non-linearity of response. It is easy to straighten up in the final result. But dealing with a response spectrum [Absorbance/ppm] that varies over time, or potentially can do so, is tough. Not being able to know/control/counter all the effects that can cause instationarity of response can be a main risk. Measurement accuracy in the NIR, in my experience, is often limited by just that, lack of response stationarity. Spectral pre-processing methods are an important part of the optical measurement system because they play a major role in determining the linearity and stationarity achieved in �your� measurement system (consisting of your sample, your spectrometer, your sampling optics, and your way to pre-process the raw spectral data). But the focus of this discussion should be on achieving time invariance, not linearity. A discussion about how straight the line is when plotting, log(1/R), or whether that line has the same slope as the �original� Lambert-Beer �law� has under its ideal cuvette conditions, is irrelevant if there is no reliable response spectrum to measure in the first place. In my opinion, the situation is like this: (1) �Simple� effects like lack of monochromaticity, finite NA of illumination beam, etc. � these affect �only� the linearity and are easy to describe mathematically. In practice, no problem. (2) Scattering effects (�physical matrix�) � these can affect the linearity AND the stationarity and, in practice, it is the latter that hurts quantitative measurement most. Changes in the effective optical pathlength are a major pain in NIR diffuse reflection measurements. People understanding optics and the Maxwell equations can still be creative here. (3) Matrix effects (�chemical matrix�) � these can also affect both, linearity and stationarity, and the latter is again more challenging in practice. Stationarity of response should therefore be estimated/investigated/improved in each application case. People understanding molecules needed here. Understanding the chemical and physical environment of the analyte in the given application and then devising a measurement system with a time-invariant response spectrum is, in general, the key. Everything else, incl. chemometrics, is straightforward in comparison. Linearity is nice-to-have but is usually not the major concern in NIR spectrometry. Ralf VTT Optical Instrument Center MTT Multantiv | |||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 257 Registered: 9-2001 |
Don - when I said "... all its faults are known ...", my meaning was "known to modern science". But I agree with our assessment: people using NIR get enamored with the chemometrics (it's easy to do that!) and tend forget about all the other sciences, and scientific facts that are involved, and that a "fully rounded" NIR practitioner should know. Certainly there are people who don't know all the things that we had to learn the hard way, and this is disappointing, because they are not making best use of the available knowledge, and likely a few who don't know hardly any of it. That's all probably good for consultants like me, because when they come up against "mysteries" they'll call for help, and that's good for business, but overall, it's not good for the field. As I said, though, in the earliest days, none of us knew what the limitations of either log(1/R) or the K-M function were, or what conditions would have to be satisfied for the "real world" to agree to the theory. Historically, yes, you're correct about Karl's early detector systems. But he soon started trying to further improve his results by applyig different data transforms. Also, he wasn't unaware of the use of log(1/R) in transmission spectroscopy, or of the existence of the K-M function (as we all did, even if none of us really knew much about its underpinnings) of of the similarity of the K-M curve to the log (1/R) curve. But without the mathematical properties of the log(1/R) function that I described earlier, the comercialization of the technology would never have been so successful. People would have tried it, and then put the instrument on the shelf when it became apparent that continual recalibration was more trouble than it was worth and created more expense than it saved. The reasoning would have been something like: "if I have to run all those samples (using the Kjeldahl or other "wet lab" reference method) to keep recalibrating the instrument, I might just as well just run my samples directly and be done with it". NIR became a commercial success not because it gave better results, or because it was high-tech, or even because the NIR computer could be hooked up to the corporate computer to allow the results to go directly into the corporate computer's P/L calculations, but because the instrument could directly and demonstrably save the company money, compared to using the reference method for all their analyses. (Parenthetically, there were three main sources of these savings, which are still valid and operative today, which is why NIR is still a common and popular technique: no need to continually purchase the chemicals that the reference techniques required, no need to dispose of the used chemicals, and reduction of the manpower needed to run the analytical laboratory, with the concomitant reduction in total salaries. Other savings occur because of the speed of analysis, which obviates the need to store or quarantine material until the lab results can demonstrate that it's within spec, and other less direct, but no less important, savings.) \o/ /_\ | |||
| Donald J Dahm (djdahm) Senior Member Username: djdahm Post Number: 28 Registered: 2-2007 |
In truth, I don�t care what you call it, but I�m pleased that it gets under your skin when I say it more of an approximation than a law. And it pleases me doubly when it spurs you into writing up such valuable information for our young friends. (Almost everybody is young compared to Howard and me.) When you say: �So why is the log(1/R) transform so universally used today, when we know about all its faults?�, you suggest something that is not in my experience. That is, I do not observe that most NIR practitioners are aware of all it faults. Which is why I started this �diatribe� in the first place. I think the �next generation� is becoming fairly well-schooled in Chemometrics, but attention to theory is decreasing. A bit of an addition to the Karl Norris story: He likes to point out that the reason that the first work was done in log(1/R) was that his detector was set up in the logarithmic mode for transmission when he first did remission work; and it worked, so in the absence of any reason to change, he kept using it. As far as the Kubelka-Munk function goes, it is �theoretically correct� only when the amount of absorption by any one particle is very small; when the sample is �infinitely thick�; and when the degree of divergence of the beam is constant throughout the sample. There�s a figure in our book [D-7] that shows how the metric log(1/R) departs from the absorbing power (that's what I call the quantity that would really follow Beer�s law). The take home is that log(1/R) is far more linear with absorbing power for thin samples than thick ones. I've tried to upload a copy of a portion of the figure that shows how well log(1/R) can approximate the absorbing power in thin samples. (IMPublications holds the copyright.) The top line is the absorbing power, and the bottom line is { log(1/R) }.
| |||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 256 Registered: 9-2001 |
Don - if you're going to insist that we call Beer's Law an "approximation" instead of a "Law", then I have to insist that we do the same, for example, to Newtons laws of motion, and even of gravitation, to Coulomb's Law, Gauss' Law and all sorts of other physical laws, which are all idealizations of reality, that "work" only under certain (and usually unspecified escept to the physicists when they are learning about them) conditions. Newton's laws of motion "work" only in the absence of friction, air resistance, or other external influences. Einstein showed that they "work" only at low speeds and in the absence of gravitational fields. How low is "low" speed? How little friction is allowed before you can apply the idealization to the real world? All these secondary conditions are ignored for a whole slew of physical "laws", so I don't understand why you pick on Beer's Law to say that it shouldn't be given the status of a "law". In the previous post, I also pointed out that Beer's Law was defined only for clear solutions, so that the question of diffuse remission doesn't (or at least shouldn't) even come up, and I also said that chemists learn that in practice, the transmittance of a sample should be greater than roughly 10% (to avoid nonlinearities from measurement artifacts, wuch as stray light). However, just as in the case of Newton's Laws, in practice you can use Beer's Law as long as the actual physical conditions are "good enough" approximations to the idealized conditions, so that the errors involved in their use are "small enough" and the approximation therefore "good enough" to provide useful results. As John Coates has been saying lately: "good enough" is often good enough. So whether Beer's Law gives results that are "good enough" depends on what you're doing, as well as on the allowable error level. So why is the log(1/R) transform so universally used today, when we know about all its faults? Here's a bit of a history lesson: the use of "Beer's Law" to process NIR reflection data for use in quantitative chemometric calibration arose and became popular for an entirely different reason than any theoretical benefit (I know, because I was there and involved in the discussions at the time). Here's what happened: when Karl Norris invented modern NIR analysis as we know it today, one of the first things he tried doing was to optimize his results by transforming data different ways. He tried just about everything and anything you could think of, and some things that nobody except him thought of. Karl could never demonstrate conclusively, or even reliably, that any one transform was invariably better than any of the others. Among them, of course, were the Kubelka-Munk transform, and the log (1/R) transform. The Kubelka-Munk transform was considered "theoretically justified" (none of us at the time knew any better) and log(1/R) was borrowed from transmission spectroscopy. Furthermore, if you plot the two functions on the same graph, the curves are remarkably similar: they both evaluate to unity at R=1, and they both go asymptotically to infinity at R=0. That, to us at the time, was plenty of justification for accepting log(1/R). However, as a practical matter, the log(1/R) transform had one characteristic that gave it an enormous advantage over all the other data transforms, including the K-M transform, when the spectral data was used for quantitative ("chemometric") analysis. Due to the nature of the logarithmic function, a multiplicative change to the data was transformed into a uniform change in the value of the log(1/R) function, regardless of the value of R. One of the most common sources of the multiplicative change in NIR data at the time, and even today, is a change in the reference reading. If the reference reading should change without a corresponding change in the sample readings, then the computed reflectance and therefore the computed absorbance would change, and if the change in the reference reading is the same at all wavelengths (also a common type of change), then the change in reflectance is proportional to the reflectance, leading to a uniform change in log (1/R). It's easy to show that when the readings are used in a calibration model and all the absorbances change by the same amount, the change in the computed analyte value is equivalent to a change in the B0 (constant) term of the calibration model, without any effect on the other coefficients of the model. No other data transform has that property. Therefore, if something should happen, such as the reference reflector becoming dirtied, that would affect the reference readings, using any other data transform means you would have to entirely recalibrate your instrument, while if you use the logarithmic transform you could simply correct the constant term of the calibration model, i.e. make a "bias correction" - a much simpler and faster procedure. It was this operational simplification in the face of an obvious and common condition, plus the knowledge that it was "similar" to the "theoretically correct" (K-M) transform, that impelled all the instrument manufacturers and all the users to make the log(1/R) data transform their standard and default data transform. To say nothing of the fact that if such a small change to an instrument (i.e., a dirty reference reflector) would require recalibration of the instrument, there would be no hope of ever developing a calibration that could be transferred between instruments. Since conditions haven't changed to this day, I don't expect any other data transform to become ascendent, regardless of the theoretical benefits (or lack thereof). \o/ /_\ | |||
| Bruce H. Campbell (campclan) Moderator Username: campclan Post Number: 117 Registered: 4-2001 |
I find it interesting that in the discussions on the derivations of the Beer/Lanbert/Bouguer relationships that nobody has mentioned the need for the light source to be purely monochromatic. And if we want to extend the discussion, we could go into Ringbom plots and log absorbance usages. Bruce | |||
| Donald J Dahm (djdahm) Senior Member Username: djdahm Post Number: 27 Registered: 2-2007 |
Written primarily to those new to the field: Over in another thread (Melamin in milk powder), there was the beginning of a discussion �Beer�s law�. Most of what I say here is from my book (Interpreting Diffuse Reflectance and Transmittance), published by our discussion forum host. I am of the opinion that the phrase �Beer�s law� is used very loosely, and many of us believe that �Beer�s law� is a much better description of reality than it actually is. That would have included me twenty years ago. I hope that through a lot of effort, I now have a better understanding. In that thread, Gabi Levin responded to my (deliberately inflammatory) post by saying: One aspect of these laws is that the intensity of light that passes through a substance is also exponential to the negative of the absorption coefficients of the substances present at the given wavelength, and the concentration of these substances along the path of light. Here he mixes words from two different laws, the �Bouguer-Lambert law� and �Beer�s law�. The Bouguer-Lambert law is a law from physics that says that the fall off of intensity is given by exp(-kd), where k is the absorption coefficient, and d is the distance traveled. The Bouguer-Lambert law applies to a single sample. There is not a concentration term included in this law. Beer�s law applies to a series of samples and states that the absorption of light by a solution changes exponentially with the concentration, all else remaining the same. Instead of an �absorption coefficient�, Beer�s law uses the concept of �absorptivity�. In Beer�s law, the �absorption coefficient� for a sample winds up being a product of absorptivity and concentration. In analytical chemistry, the term most often used is molar absorptivity. Another term used is "absorption cross-section", which is the "absorptivity per molecule" (obtained by dividing the molar absorptivity by Avogadro�s number). When one combines these two laws, it is frequently referred to as the Beer-Lambert law (as Gabi did), which then says that the quantity of light absorbed by a substance dissolved in a non-absorbing solvent is directly proportional to the concentration of the substance and the path length of the light through the solution. The law is commonly written in the form A=�cd, where A is the absorbance, c is the concentration in moles per liter, d is the path length in centimeters, and � is a constant of proportionality known as the molar absorptivity. The law is accurate only for dilute solutions; deviations from the law occur in concentrated solutions because of interactions between molecules of the solute, the substance dissolved in the solvent. (Unfortunately, the term Beer�s law is often used for this combined law.) Here again, there is no mention of absorption coefficient. As far as Howard�s carefully written response as to the points that I was �missing�, I recommend it to all as technically sound. Pay special attention to the fact that we have extended Beer's Law by saying that �the absorbance of a sample at a given wavelength is the sum of the absorbances of the individual absorbance in the sample at that wavelength�. This is the foundation of most of the multivariant analyses we do when using NIR. Having said that, Howard knows full well that applying Beer�s law to highly scattering samples is a huge departure from the specified conditions of Beer�s law; much greater than that for the Bouguer-Lambert law, where to be an exact description, �the light wave-fronts have to be perfectly plain and parallel�. Just as I chose to be inflammatory, Howard chose to be pedantic. I, in fact, have sat in the classes that explained the limits of Beer�s law. Perhaps shamefully, I deliberately choose to leave out a phrase out of statement that �a law is a summary of observations for which there is no known exception�. That phrase was �under the specified conditions�. Of course, one is entitled to call it Beer�s law. But then one ought to restrict the usage to the specified conditions. I don�t know how bad an approximation has to be before one is entitled to use the word �false�, but I think we�re there when we use Beer�s law for remission data. [The Kubelka-Munk function is better, but still only an approximation.] My point is that we in the NIR community use the term �Beer�s law� in such a loosy-goosy way that we should just call it an approximation instead of a law. I�m not denying that using the approximation has been valuable. In the book, I say that we make an important assumption that �something akin to Beer�s law is in operation� in scattering samples. However, as I learn more, I�m no longer convinced that using log(1/R) instead of R makes the Chemometrics work all that much better. Anybody have any data on that? |
Absorbance vs absorbing power