| Author | Message | ||
     Donald J Dahm (djdahm) Intermediate Member Username: djdahm Post Number: 19 Registered: 2-2007 |
Andrew: You are absolutely correct. Even though the term "extinction" is used to mean "absorption loss plus scatter loss" (particularly for those cases where all the scattered radiation leaves the path of observation), the term "extinction coefficient" is frequently used by Physicists to mean the imaginary part of the refractive index. From Wikipedia: The parameter used to describe the interaction of electromagnetic radiation with matter is the complex index of refraction, �, which is a combination of a real part and an imaginary part: � = n + ik Here, n is also called the index of refraction, which sometimes leads to confusion. k is the extinction coefficient, which represents the damping of an EM wave inside the material. Both depend on the wavelength. The ASTM T1A1 committee and Harry Hecht (in Wendlandt and Hecht in the chapter that we reproduced in our book) used the term "Absorption Index" instead of "Extinction Coefficient" for this term. That seems much more satisfying to me. We then have the "refractive index" and the "absorption index" in the "complex refractive index". Incidentally, in your equation " alpha = 4*pi*k/lambda where k is extinction", I think there is an "n" missing from the numerator, but it's hard to keep it all straight, so I may be wrong. If that's not confusing enough, we frequently use different symbolism for the same thing. What you (and Wikipedia) called "k", I call "kappa", and what you call alpha, I call "k". | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 123 Registered: 9-2001 |
Well, here's what I said before (more or less): It's not very difficult, but it takes some algebra to prove that this is so. If you want to look up the details, they're in my book: "Principles and Practice of Spectroscopic Calibration". Don kinda said it in a nutshell: if there's a change in the reference reflectance, then the measured reflectance changes in proportion to the reflectance of the sample at that wavelength. When you compute the logarithm (to any base) or those reflectance values, therefore, the absorbance change is constant (i.e., the same at all the wavelenths) regardless of the value of the absorbance. When you multiply the new absorbances by the coefficients of the model, therefore, the computed constituent value changes by a constant amount, regardless of the value of the constituent. This is equivalent to having a change of the zero point of the instrument, and can be compensated for, or corrected, by changing B0 (i.e., the constant term) of the calibration model, without having to change any other coefficient in the calibration model; this is equivalent to a zero adjustment. Thus by changing only the B0 term of the model, you get the same performance, and the same computed values for the analyte in any sample, as you would have computed before the reference reflector changed. This is what I meant by saying that it doesn't affect the results: you get the same results without changing the model, except for B0. No mathematical function (including K-M) besides the logarithm, has the property that a proportional effect on the computed transmittance becomes a constant change in the value of the function of transmittance, and therefore only the logarithm allows you to make the correction that way. If you use any other functional transform on the data, a change in the reference reflector (or any change in the insstrument equivalent to that type of change) would require you to recalibrate the instrument to determine all the new calibration coefficients, not just b0. \o/ /_\ | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 122 Registered: 9-2001 |
What happened? I responded to Yjunhan's question a few hours ago, then I suddenly got Don's response to the question indicating that I could respond with more detail (as in fact I already had), as though I had never made a response. So then I clicked in the link and found that my response had not been posted!! I'll wait a while to see if anything comes up, otherwise I guess I'll just have to do it again. Strange things are happenning in byteland. \o/ /_\ | ||
| Donald J Dahm (djdahm) Intermediate Member Username: djdahm Post Number: 18 Registered: 2-2007 |
Howard can give a more detailed response, but I assume that what he was saying was that a new "error" that has the effect of reducing the incident beam by some amount will have a multiplicative effect on "R", which translates to an offset (bias) in log(1/R). | ||
| yjunhan (yjunhan) Junior Member Username: yjunhan Post Number: 7 Registered: 7-2006 |
Howard, I don't quite understand your statement that log (1/R) doesn't affect the analysis directly if an instrument's reference reflector changes, for any reason (is replaced, gets dirty, etc.) While K-M does. Could your further explain that to me? Thanks | ||
| Andrew McGlone (mcglone) Member Username: mcglone Post Number: 12 Registered: 2-2001 |
Don, you have hit the conceptual problem on the head. Extinction justs asks to be interpreted as something like the loss due to absorbance plus scatter. My confusion started because I already knew that an extinction coefficient is defined as the complex part of the refractive index. And that the absorption coefficient is directly related to it (something like: alpha = 4*pi*k/lambda where k is extinction). Now an absorption coefficient is always defined in terms of an exponential decay (cf natural log units). So when I first came across the concept of Absorbance, as a measurement in decadic or log10 units, it was mightily confusing when the associated decay constant was sometimes referred to as an extinction coefficient. It is easy to get confused and start to wonder if you have missed a key concept like extinction = absorption plus scatter. You go to the books and find extinction talked about in that way, which merely ends up confusing you more. So I'm very glad to hear that Applied Spectroscopy has moved solidly away from the term extinction. | ||
| Donald J Dahm (djdahm) Intermediate Member Username: djdahm Post Number: 17 Registered: 2-2007 |
It sometimes seems we delibertely try to make things confusing to newcomers with our various definitions and terminology. For example, in this discussion, I had trouble knowing whether "launched" light, means light incident on the sample, or light scattered by the sample. The idea of extinction arose in Physics by considering the falloff in intensity of a beam though the atmosphere. The term was used to describe the dimming of the light beyond that which would be expected by the geometry of spreading out from a point. Today we could say that extinction is the sum: absorption plus scatter. The use of the word got carried over to specgtroscopy, though the journal Applied Spectroscopy now discourages its use. Gabi: I have been wondering: where did you get your model of "reflectance"? It has certain things in common with the "Intuitive Model" that I ascribe to Karl Norris. I beleive that under some conditions, it would need an extra term or two, even if we could measure everything perfectly. | ||
| Gavriel Levin (levin) Senior Member Username: levin Post Number: 49 Registered: 1-2006 |
Hi Howrad, As a chemist for a good number of years before becoming materials engineer and later again involved in applied NIR I must say that the division is not totally between chemists and physicists - many chemists whose focus is on theoretical aspects of chemistry, and particularly theories related to spectroscopy that is the result of electronic quantum states (as different from vibrational quantum states) use the Naperian base e log, at the same time, all the chemists whose focus is on practical aspects, such as electrochemistry, pH related chemistry, and many spectroscopic applications prefer the log base 10 for convenience. The log base 10 is much more convenient because it is already engraved into our minds that the world comes in 10, 10(2), 10(3), etc, or 10(-1), 10(-2) etc. The whole decimal concept is well rooted in our fundamental way we see the world. It is almost only in the US that part of the world is still in lb, miles, inches, foot, yards etc. However, since electricity came into the physics and real world later, at that time the watts, the volts, and the amperes, are all "metric" in the way they are concieved in the US as well. Gabi Levin | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 121 Registered: 9-2001 |
This is something Don and I had discussed in the past. We finally realized that it resolves to a cultural issue: chemists use common log (log base 10) and physicists use Naperian log (log base e)! \o/ /_\ | ||
| Gavriel Levin (levin) Senior Member Username: levin Post Number: 48 Registered: 1-2006 |
Hi The only reason why the LN (or natural log) has been used more widely earlier is because as I have written above the equation for the absorption of light is related to the exponential with e as the base of the exponent I1/I0(1) = R*exp(-alpha1*d)where as before alpha1 is the absorption coefficien at wavelength 1 and d is the path length. and of course the simple thing is to do is take the LN (rather than the LOG which is traditionally reserved for logarithm in the base of 10) So, as expected the only difference will be a constant, 1 divided by the LN of 10 in the natural base of e. LN(10) =2.320258 and 1/2.32 is as you said 0.43429 Gabi Levin Brimrose | ||
| Andrew McGlone (mcglone) Member Username: mcglone Post Number: 11 Registered: 2-2001 |
I've just read down the thread. Mention of choice between napieran and decadic log systems is amusing to me as it can be quite annoying if authors don�t spell out the difference in their papers. I was confused for a long time about the difference between extinction and absorption coefficients just because some authors were not defining what they were using, presumably assuming everyone would know it. I initially thought they really were conceptually different parameters when I first started doing NIR. Hey, they certainly had different units of OD cm-1 (extinction) or just cm-1 (absorption). Anyway, different researchers seem to offer exclusively one rather than the other from their experiments for reasons I couldn't quite fathom in my early days. After struggling on through relevant chemistry and physics literature I've come to the happy conclusion that they just differ by ln(10)=0.434. That is, only the log system used is different. | ||
| Gavriel Levin (levin) Senior Member Username: levin Post Number: 47 Registered: 1-2006 |
So as not to be the devil, the origin of the log conversion comes from the fact that the fundamental absorption of a chemical A, at wavelength W1, is gien by A= R*exp(-alpha1*d)R being a constant of the set-up and other physical parameters, alpha1 being the absorpttion coefficient at wavelength W1, and d being the path length. Thus the expression for the light that came from the sample at wavelength W1 will be I1 = I0(1)*Rexp(-alpha1*d) and the ratio between the measured light to the returned light will be (need to make sure that we know I0(1) stands for I0 at wavlength W1, etc.) I1/I0(1) = R*exp(-alpha1*d) and then LOG(I1/I0(1) = 0 + (-alpha1*d) so by using -LOG we get a positive numebr. Back to the issue of diffuse reflectance, in my mind, because there is no practical way to collect all the light that went "through" the sample, and at the same time to make sure we don't collect anyhting that is not coming from the sample, the actual values of the intensity of light coming to the detector are influenced by too many things. Now, for those who use an internal diffuse reflector for measuring the intensity of launched light at each wavelength, this measurement is also subject to all the inherent problems, then the I0 is also a value which is dependent on many instrument factors, and the only "remedy" to avoid troubles is by maintainng everything in good order and reproducible condition. One remedy in the event of hardware problems as Howard mentioned is to perform :"calibration transfer" by having such fubction built for the instrument. I hope I cleaned some of my devilish nature. Gabi Levin | ||
| Donald J Dahm (djdahm) Intermediate Member Username: djdahm Post Number: 16 Registered: 2-2007 |
Thank you, Howard. While I would certainly be willing to believe that a "linear" metric would work better in PLS, I must confess I was guessing as to what the differences of using various non-linear metrics would be. I have no data, and little expertise. However, I doubt if there would be any signifcant difference between K-M and log(1/R). Regardless of the practical benefit, a truly linear metric is a bit of a "holy grail" to me, and worth finding even if the reason is only intellectual curiosity. As far as the transmission data goes, if one can exclude the radiation that has been scattered by a "non-clear" sample from reaching the detector, the log(1/T) metric will be more linear than by using hemispherical detection. That was what I meant by using a small area detector. For clear samples, the detector arrangement is not particlualry important. | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 120 Registered: 9-2001 |
Yjunhan - a couple of points: One definition of "linearity", that is being proposed for an ASTM practice, is "That property of the relationship between two sets of data, such that a straight line provides as good a fit, in the least-square sense, as any other mathematical function." This coincides with our intuitive sense of linearity; and is essentially the same one proposed in Spectroscopy; 20(1); p.56-59 (2005). In NIR, the two sets of data of most interest are the sets comprising the predicted and reference values for a set of samples. Everything else the same, a non-linear relationship between the sets of data will have larger values for SEE and SEP than a linear relationship will, although these standard errors will reflect systematic effects rather than random effects. However, Don made one misstatement: it is not always so simple to compensate for non-linearity merely by using another factor in a PLS model (see Spectroscopy; 13(6), p.19-21 (1998), also several followup columns starting in issue 10 of that year.). And even if you can, we all know that the fewer factors in a PLS model, the better off you are. While linearity is sometimes by touted as being a benefit of K-M or log 1/{R|T}, the only benefit for analysis that can be justified is, as Don said, using log (1/T) when measuring transmission through clear samples. In reflection measurements, however, there is another benefit of using log (1/R), which has nothing to do with linearity, doesn't affect the analysis directly, and is of a secondary sort. This benefit comes into play if an instrument's reference reflector changes, for any reason (is replaced, gets dirty, etc.) If you go through the mathematics of the calibration (regardless whether you're using MLR, PCR, or PLS) you find that in that case, you're justified in correcting for the change in reference reflectance by making only a bias change to the calibration model. If you use any K-M or other data transform, you would need to perform an entirely new calibration, to correct for that effect. \o/ /_\ | ||
| Donald J Dahm (djdahm) Member Username: djdahm Post Number: 15 Registered: 2-2007 |
The first thing I would hasten to say, is that when you are doing PLS, you don�t need to worry too much about what transformation you use. In fact, R will probably work about as well as the transformed data (though the magnitudes of prediction errors might be a bit different, I don�t know). The main difference is in the number of factors it takes to describe the relationship between the metric selected and concentration. One thing that I have been fighting (for what seems forever) is the illusion that K-M should in theory be linear with concentration if there is no change in scatter from sample to sample. That is simply not true. There are benefits claimed for log(1/R) in that peak shapes better maintain integrity. [Peter R. Griffiths, �Practical Consequences of math pre-treatment of NIR reflectance data: log (1/R) vs F(R)� JNIRS 3, 60-62 (1995)] If you desire a metric that has optimal linearity based on what we now know, my �practical� advice is: In transmission on scattering material, use a small area detector, and log(1/T). In remission, use thin samples in transflection, and log(1/R). The K-M data should have a larger range of linearity than log(1/R) for infinitely thick samples. If you can get hold of the back issues of NIRnews, most of this is discussed in various �Speaking Theoretically� columns. | ||
| yjunhan (yjunhan) Junior Member Username: yjunhan Post Number: 6 Registered: 7-2006 |
Thanks very much for all the instructions. I learned a lot. I'd also like to hear your insights associated with Kubelka-Munk vs. log1/R. I am dealing with agricultral ground samples, thus using diffuse reflectance mode. Somewhere I heard that K-M is most suitable for this mode, and I saw many papers was using K-M. However, many other papers are using log1/R, and also some article was claimed that log1/R offer better linearity. Then I am confused, in NIR calibration, Beer-Lambert law doesn't really hold, especially, when PLS utilized, the calibration model itself wasn't really consider A=kC at each wavelength. And even if it is true that log1/R give better linearity, I think PLS can generate another set of lantent variables or even by using a little more latent variables, there always exist a comparatable calibration model when using K-M tansformed dataset. Probably what I stated have a lot of mistakes, so pls point out my misunderstand, and explain me what is linearity in NIR calibration mean. And K-M and log1/R, which is better to use if I analyze, say, soy beans. And why? Thanks so much! | ||
| Donald J Dahm (djdahm) Member Username: djdahm Post Number: 14 Registered: 2-2007 |
Gabi: I (we) were merely trying to help yjunhan understand why he had seemingly come across a logical inconsistency. 1) His books say Absorbance is log(1/T). 2) His instrument gives the same response for the functions log(1/R) and Absorbance; Therefore { log(1/T) = log(1/R) }. If I were being my normal smart-alecky self, I would say you are not being the Devil�s advocate, you are being the Devil himself advocating ignorance. But I will be serious. I don�t know exactly where you are coming from, but if you are feeling unappreciated, I know that no one would give a hoot about my work (which I see as education) if there were not people like you pushing the technology for its practical benefits. And I�m glad that turns you on. As far as being �exciting� goes, I frequently say that my goal of trying to understand what is going on in scattering samples is an ideal career for anyone who �loves boredom and has an innate fear of success�. Now to everyone else: The issue raised by Gabi is an interesting one. Are we in such a bad state of understanding when it comes to scattering samples that we would be better off ignoring theory and rely exclusively on empirical relationships? Is that, in essence, what we are doing? Is that one reason why we have had trouble in gaining wider acceptance for the technique? | ||
| Gavriel Levin (levin) Senior Member Username: levin Post Number: 46 Registered: 1-2006 |
Well, looks like I will be the devils advocate in a way - while in clear liquids one can really set up an experiment where he removes the sample, and measures the intensity of the "impinging" or launched light at each wavelength I(sub zero, super script lambda for wavelength)and so on for all wavelengths, then place the sample, and measure the Id at all wavelengths, that is assuming nothing changes between the time you measured the launched light and the transmitted light. In reflectance you can not do it, because you only collect a fraction of the diffusely (or hemispherically reflected light) depending on your optics. So you never have a true value for the reflected light. Thus reflectance is a relative term that reflects what a specific optical design can do. However, all this has no relevance to what people are trying to achieve by these measurements - a measurement that can be shown to truly predict what's in there. For that end the only thing of importance is that once you have a certain optical set-up, you will maintain it constant, so your measurments day after day, week afetr week will have the same relation to what is truly in there. To achieve that the instrument engineers go to great efforts. Once the instrument is "stabilized" and reproducible the true values are insignificant from the practitioner's point of view. I know it sounds "boring" to the theoreticians, but being able to help manufacturers of various products at a higher efficiency, with lower % of rejected products, etc. is much more exciting to me. Gabi Levin Brimrose gabi Levin | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 119 Registered: 9-2001 |
Yjunhan - mostly I can only repeat what Don and Tony said, but I want to do it with somewhat of a change of emphasis: The mathematics of the calculations are identical, whatever the teminology, as both Don and Tony explained. The difference is in the experimental setup used to generate the data. Whether you're measuring transmission or reflection, you measure I and Io and calculate the ratio, and the terminology used for that ratio depends on the experimental setup (transmittance or reflectance, respectively). The confusion comes in when the logarithm is calculated; Tony and Don are trying to systematize that, but the confusion persists in the literature. The logarithm of the ratio is, as you noted, often (but not always) called "Absorbance" regardless of which experimental setup gave rise ot the data. To clear up one other point that may have been glossed over a bit, the decision to calculate the negative logarithm of the ratio is not an arbitrary one. When everything is ideal, then as Don shows in his book, the negative logarithm of the specified ratio is proportional to the concentration of the absorbing species, or in the case of multiple absorbing species in the sample, equal to the weighted sum of the concentrations of all the absorbing species. In practice the majority of cases are not so ideal. Nevertheless, the use of the logarithmic transform provides a relatively simple and straightforward transform that approximates the actual behavior of a wide variety of samples sufficiently well to be of practical use. \o/ /_\ | ||
| Tony Davies (td) Moderator Username: td Post Number: 160 Registered: 1-2001 |
Dear Yjunhan, Yes NIR can appear VERY confusing! Absorbance A = ln(I0/Id) Where ln is the Napierian log, I0 is the incident illumination and Id is the detected illumination. Chemists usually use the decadic absorbance A= log(I0/Id) = log 1/T where T is the measured transmittance. At the dawn of the modern period of near infrared spectroscopy Karl Norris had been making transmission measurements of samples using a logarithmic amplifier so that the output was log 1/T when he moved the detector so that it measured the reflected energy, R, the output of the system was log 1/R. He found that this was a useful measure of sample absorption but it was not �absorbance� (and the rest is history!). So, if you are measuring a non-scattering sample in transmission mode then Absorbance is the correct term but if you are measuring in reflection mode then you should use Log 1/R but if you do not tell the truth to you computer and ask for absorbance it will look exactly like the Log 1/R spectrum because the computer will have used R in place of T. The reflectance, R, is what the spectrometer actually measures and there is no reason why you should not plot your spectrum in R but most people do not find this useful. There is a lot more to this story and most of it is in the book by Don and Kevin Dahm recently published by NIR Publications. Of course while I was writing my answer Don got in and gave it! However I hope I have added a little more background so you get mine as well. Note that changing the postions of I0 and Id gets rid of the negative sign. Best wishes, Tony | ||
| Donald J Dahm (djdahm) Member Username: djdahm Post Number: 13 Registered: 2-2007 |
Our friend John Bertie (my definition guru) says: �Absorbance is a widely used term for decadic Absorbance, napierian Absorbance, and other quantities. Chemists usually mean the decadic Absorbance. Confusingly, the term Absorbance is also widely used for the negative log of the ratio of the final to the incident intensities of processes other than transmission, such as attenuated total reflection and diffuse reflection." I tend to use the confusing definition. It helps to think of the Absorbance function as being "general", and Absorbance as being only for tranmission data. From our new book, which is advertised on the NIRPublications home page: (please excuse the fact that the subscripts aren't (sub") C-1. What Is Absorbance? The Absorbance function is well known by most people who do spectroscopy. Suppose we have a light beam of intensity I0 impinging on a sample, and an intensity of Id striking the detector. We could report the fraction (or percentage) of incident intensity that strikes the detector. However, more often we use the Absorbance function. The Absorbance function is defined as the negative logarithm of the fraction of the incident intensity that strikes a detector. Generally, in the book, we use the natural log in the definition of Absorbance and give it the symbol Ae, so that { Ae = ln(I0/Id) } A modern instrument can present the results of a measurement in many forms. One that is very commonly used is the decadic (log base 10) Absorbance. We can express this as { �log(Id / I0) }. In a transmission experiment, using T to mean the fraction of light transmitted, since { T = Id / I0 }, we can write an expression for the Absorbance function as { log(1 / T) }. Similarly, for the case of measurements made in remission, where { R = Id / I0 }, we can write an expression for Absorbance as { log(1 / R) }. It is sometimes said that Absorbance is only rigorously defined for the case of transmission through a non-scattering sample as { log(1/T) }. However, the Absorbance function has certainly been useful in NIR remission spectroscopy, where it is usually given the definition {log(1/R)}. We prefer to say that the meaning of the Absorbance function is only well understood for the caseof transmission through non-scattering samples. | ||
| yjunhan (yjunhan) New member Username: yjunhan Post Number: 5 Registered: 7-2006 |
Most NIR literatures are shown in terms of log1/R; absorbance is seldom seen. My NIR spectrometer software offer the option to directly tansform signal into absorbance, reflectance or log1/R, but the spectrum in Absorbance and spectrum in log1/R actually overlap, look exactly the same. Are absorbance and log1/R basically the same thing? If so and plus A=log(1/T, it derives that transmittance and reflectance are the same thing. I know this derivation result is so wrong, but I didn't know where I got it wrong. Waiting for the correction and advice, thanks very much. |