| Author | Message | ||
     Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 438 Registered: 9-2001 |
Don - can you define what you would consider a sufficiently "well-controlled" study"? Maybe between all of us we could manage something. \o/ /_\ | ||
| Donald J Dahm (djdahm) Senior Member Username: djdahm Post Number: 68 Registered: 2-2007 |
The �Peter Griffiths� study was on carbazole in a matrix of varying absorption. The experimental work was done by Jill Olinger. It was reported in the 1992 edition of Burns and Ciurczak�s Handbook, as well as in Analytical Chemistry in 1988. The matrix was a mixture of NaCl and graphite. I think the matrix itself would be a system a bit like one you would be interested in. Unfortunately they did matrix referencing in that study and, as I recall, did not record spectra of the matrices against a normal reference. In any cases, when I asked Peter if any more data existed he said it was discarded if there had ever been any. The original interpretation of the data in that study was, in my opinion, flawed, and lead to some of the� arguing� that Howard talked about. [Among other things they used mass fractions rather than volume fractions.] My reinterpretation of the data is the 3rd edition of the Handbook in a chapter by Griffiths and Dahm. It�s also on pp 133-146 of our book. As far as Art's data goes: It turns out that the case of systematically replacing non-absorbing (PTFE?) with totally absorbing (carbon black?) particles is a favorable case for linearity. I assume that the sintering affects the scattering power of the sample in a way that the un-sintered material doesn�t, but then what do I know. I�d love to see results from a well controlled study on a simple system. | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 437 Registered: 9-2001 |
Art - Oh, I almost forgot: a while back I measured some mixtures of lampblack & PTFE over about 5 orders of magnitude of mixture concentration. This went from pure lampblack (as dark as I could get it) to pure PTFE and as close to pure as I could see any difference in the reflectance. I could probably find the data, if you could make some use of it. \o/ /_\ | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 436 Registered: 9-2001 |
Art - Peter Griffiths did a fair amount of that in the mid-IR, as I recall, and might have done some in the NIR, too, but I can't swear to that. I think Don and Peter wrote one or two papers about it (or maybe they just argued! :-) \o/ /_\ | ||
| Art Springsteen (artspring) Senior Member Username: artspring Post Number: 37 Registered: 2-2003 |
Hello all, This has been a fascinating discussion. Theory is a wonderful thing but like Karl, I'm an old experimentalist. And there is a question I have that I've never seen answered. Has there ever been a study of looking at putting controlled amounts of a particular absorber in a scattering media and looking at the linearity (or lack thereof) of the resultant mixtures? We do this sort of stuff all the time (ours being carbon black in PTFE) and we note linearities of log 1/R over a fairly wide range. But we don't see the same results for sintered versus un-sintered (but pressed to equal densities) carbon black in PTFE mixtures. And now we are looking at other systems containing neither PTFE nor carbon black (melamine in scattering media X, incipient Y in lactose, etc.) If there is literature on this, could someone steer me there. Failing that, would anyone want to look at a controlled study of these type of materials? We'd even provide the samples at no charge if we could set up some sort of round-robin study on this. While we've been doing this type of research for nigh on twenty years for internal use, we've not seen anyone else doing this sort of simple, two-component system. Or am I barking up a well marked tree??? | ||
| Donald J Dahm (djdahm) Senior Member Username: djdahm Post Number: 67 Registered: 2-2007 |
That's what I'm tempted to conclude. | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 435 Registered: 9-2001 |
Or poasibly Kortum derived them himself, for his book. Or if not for the book, then the book was his first opportunity to make his results public. \o/ /_\ | ||
| Donald J Dahm (djdahm) Senior Member Username: djdahm Post Number: 66 Registered: 2-2007 |
Howard: I have them as well. (In fact, quite a few of the old papers are now on line, inckuding Schuster's as well as Kubelka's.) What I'm saying that the Kubelka-Munk equation in the form we are used to seeing is not in Kubelka's paper's (or I'm blind). Somewhere between Kubelka and Kortum's writings it came on the scene, and I was trying to figure out where it first occurs. | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 434 Registered: 9-2001 |
Don - I have copies of the paper by Kubelka & Munk, and also the one by Kubelka, but your timing is not good for me to send them to you before next week. If you don't hear from me, send me a reminder. Fred undoubtedly also has copies, but I don't know whether his health permits him to pay attention to these matters. I haven't heard from him in the longest time, and neither has anybody else in the community, near as I can tell. I once proposed that somebody closer to Fred than I am, bring up the question of his turning over his archive of papers to someone else who can be a custodian for them, just because of eventualities like this; it may be that the time has come to do that, before they are lost. I know, for example, that he has copies of the original papers by Herschel, and who-knows-what -other treasures. It would be a shame for that collection to be lost. \o/ /_\ | ||
| Donald J Dahm (djdahm) Senior Member Username: djdahm Post Number: 65 Registered: 2-2007 |
Thank you gentlemen for the history lesson. On a related matter, I promised Jim Reeves that I would run down the original references to the Kubelka-Munk equation. I have run into a problem. What I (and many others) call the Kubelka-Munk equation is { (1-R)^2 / 2R = K/S }, while we refer to the left side (when R is the remission from an �infinitely thick� sample) as the Kubelka-Munk function, and symbolize it f(r): { f(r) = (1-R)^2 / 2R } Others call the whole expression above the Kubelka-Munk function. What some of these folks mean by the Kubelka-Munk equation is the differential equation: { dr(x)/ sdx = r^2(x) -2ar(x) +1 }. [See, for example; Loyalka and Riggs; Appl Spec 49(8), p1107 (1995).] Others include various solutions to the differential equations that has the two constants K and S in it, but they are not arranged in the familiar form. [See for example: Equation III-44 on p 244 of our book.] While Harry Hecht included the above form in his writings, I think he got it from the writings of Kortum. If anyone knows a reference before Kortum to the equation in the above form, I�d love to hear about it. [Kortum may have made the people he was quoting sound much more relevant than they did in the original. If I�ve got any of this stuff messed up, please set me straight.] Anyway, if we take the log of the above function, we get { log f(r) = log K � log S }. The function { log f(r) } was supposed to be the same as �the real absorption spectrum � determined by transmission measurements, except for the displacement by { -log S } in the ordinate direction� (pp 245 of our book). While I have spent much of the last 15 years trying to explain what was wrong with the Kubleka-Munk equation, I think we need to remind ourselves that the Kubelka-Munk function (whatever its limitations) has a theoretical significance that log(1/R) does not. | ||
| Karl Norris (knnirs) Senior Member Username: knnirs Post Number: 52 Registered: 8-2009 |
Don,you may not be aware of how we chose to make our first reflection measurements to study the possibility of using NIR to measure the moisture content of wheat flour in 1960. We were aware of the work by Collins on the characterization of the water bands in the IR region, and we had a news item about work in Germany on using NIR reflection to measure the moisture content of paper. We did not have a spectrometer for the NIR region so we built a very crude instrument for exploration. For our previous work on measuring the spectral properties of intact fruits we had developed a logarithmic photometer to handle the extreme dynamic range of the absorption of the fruit, and without thinking we used the same logarithmic photometer for our measurements on wheat flour. We did not see the reflection spectra, we saw Log(1/R) spectra. The scheme worked, and we obtained a non-linear regression to moisture content of wheat flour over the range from zero to 20 %. We knew little or nothing about the theory of spectroscopy on diffusely scattering samples, but our measurements worked using the difference at two wavelengths, 1940-2080 nm. The linearity was quite good over the important region from 12 to 15 % moisture. It is now nice that you and Howard offer understanding of the theory behind our experiment. | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 433 Registered: 9-2001 |
Don - yes and no. First of all, Karl and Phil are doing great in the department of outliving everybody, and as long as Phil has access to a supply of beer to keep himself fueled, I see no reason why he wouldn't live forever. On a related topic: has anybody heard from Fred lately? As for my "intellectualizing" history: again, yes and no. I certainly agree that the decision wasn't a simple one made either individually or by any concensus to use log(1/R), except for the fact that it was the transform that Karl used and recommended when discussing his work with the three companies that were willing to jump in on it at the time. Any further examination of potential advantages or disadvantages was done over a relatively long period of time, individually by the various companies, and by those few academicians and other government scientists who became interested; Fred McClure, Dave Wetzel, Woody Barton, plus Karl himself of course, being the main group of those, that come to mind, although later on there were others (Charles Hurburgh, Peter Griffiths, Gary Hieftje, Tomas Hirschfeld and not too many more for quite a while). In a way I had somewhat of a unique perspective on that, since I was one of the few people who was aware of what was happening in the public scientific community as well as inside a company, where most of the people working on the technology were not aware of events in the broader community until it was brought in-house. The companies, at least, were definitely not copying each other. In fact, the compeititve situation was such that if any company started to do something, the others would avoid doing that if at all possible, fearing they would be perceived as being a "copycat" company instead of as an industry leader by customers. So if more than one company did something, it was either because Karl recommended it or because it was so obviously advantageous that nobody dared to NOT do it. So in one sense, log(1/R) became the de facto standard by default, from Karl's use of it, and nobody could find a reason to not use it, either. But along the way, yes, the advantages started to be perceived, to provide additional justification for its use. I don't know how much investigation was done internally at the other companies, but at Technicon I know, because I was the one doing it, that we did try other transforms (including Kubela-Munk) and algorithms (including, for example, using Lagrange multipliers and other pretty far-out things). Of course, at that time, nobody knew and I certainly didn't understand how little difference those should make to the final results, but in retrospect, we now see why it hardly matters. All I did, therefore, was to put all that history into some sort of perspective, and try to make an organized, coherent, account of it. Another account of those early days, longer, more detailed and and better organized, was written by Bob Rosenthal and included in the commemorative book published for the 2006 IDRC, although that mostly deals with the commercial aspects of the events. So basically yes, right or wrong, we now use log(1/R) because we have not found anything better, nor does it seem likely that we will. That still does not mean we should stop trying. Like our discovery that the units of the analyte concentration matter, we may also learn that the proper transform of the optical data will matter, possibly for unexpected reasons. It won't necessarily provide us with "better" calibration performance, but it may, for example, make the calibration process easier and less troublesome. Easier, for example, to decide when you're overfitting. Or how to select factors and/or wavelengths. Etc. \o/ /_\ | ||
| Donald J Dahm (djdahm) Senior Member Username: djdahm Post Number: 64 Registered: 2-2007 |
Howard: ... I don�t think you would be faulted by the world if you always started by assuming that I don�t understand something. While you may outlive everybody else who was there at the time, and yours may become the official version, I suspect that the decision to go with { log)1/R) } was made on a far less rational basis than you suggest. I respectfully suggest that you have added intellect to the process in retrospect. ... Don As most old timers know, there is an alternate function for remission on the table. That is the so called Kubelka-Munk function. The folks from the DRIFTS side of the house tended to defend its use. You can still find plenty of references in text books (and maybe even our instrument suppliers literature) to it being analogous to Beer�s law in transmission, implying that it is a linear function of concentration. IT IS NOT! Since the light beam striking the �front� surface has never been subjected to absorption, the remission from the surface will always be present regardless of the absorption level in the sample. While this fact alone is enough to understand why the function is not linear with absorber concentration, the tantalizing assumption that removing it would make the function a straight line does not turn out to be true. Peter Griffiths et al found such a correction to be useful in linearizing response over ranges of concentration useful to them, but as we discuss on pp 109-1l1 of our book (which I feel free to plug since it was published by IMP), it only moves the range of linearity. Since we are dealing with inherently non-linear functions, we might as well go with the one that has the advantages that Howard outlines for { log(1/R) }. | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 432 Registered: 9-2001 |
Don - This posting was actually written yesterday, but I didn't get to checking it out and posting it until after Karl's message came in. I think Karl will be pleased to find that the theory supports his findings even more strongly than he may have expected. But my response actually started here, to address your comments. I'm not so sure the direction of the thread as far off the topic as it may seem on the surface. The original question was whether it's reasonable to expect to be able to subtract spectra. One of the key factors in addressing that question is whether any data transformations had been applied to the data and if so, which ones. Different data transformations obviously have different properties. One desirable property, that we would like to have in the transformations we use, is that it should be derivable from fundamental physical theory. I think we agree that at the present time such a transformation does not exist. Correction: if such a physical theory, and a transformation based on that theory, exists, we don't know what it is. I think the work you've done has made major progress toward development of such a theory, but we're not there yet. I think you don't disagree with that. Another desirable property that a data transformation should have is that it should provide a user doing quantitative NIR analysis with a simple disaster-recovery procedure. One data transformation that we know has that property is log(1/R). While derivable from theory only for the case of non-scattering samples, we find empirically that the use of the log(1/R) transformantion not only permits a simple disaster-recovery procedure, the same procedure also allows the user to correct his inctrument for drift and other known systematic differences between sample sets and sample subsets. Ideally we would like to have a single data transformation that encompasses both properties. At this time we don't know of any. In fact it is even possible that such a transformation exists, and one important reason we don't know about it is that it has never been tested for its potential capability of allowing for a simple disaster-recovery procedure, in an actual, realistic, application scenrio. My interpretation of that statement is that the disaster-recovery procedure should require no more experimental effort or mathematical computation than is required by the use of the log(1/R) data transformation, both for calibration and for diaster recovery. So while we may have gone a bit further afield that we would have expected to, I don't think we're completely out of the ballpark.Incidentally, I didn't write that post in order to be "awesome, brilliant and inspirational". I wrote it because I perceived, possibly incorrectly, that you didn't understand why the log(1/R) transformation was so universally used in NIR, and incorrectly attributed it to the approximation to non-scattering samples. While indeed, it started there, it caught on because of the practical utiliyt of it. Had a better transformation been developed early enough, I think that would have caught on. Now I don't know if even that would, becasuse of the inertia of so many years of practical useage of the log(1/R) transform. On the other hand, I also have some comments to make about the issue of why the "wrong" function works as well as it does. What you're saying is true enough, but in some ways there are similarities (I hesitate to say "analogies" because there may be too many differences to support this stronger term) to the differences between using the "right" and the "wrong" units for expressing the analyte concentrations, the results of which we have learned. In addition to your two reasons, there's another, underlying statistical reason, that most discussions of calibration behavior pay as much attention to over the years as they have to the question of the units for the analytes, although this one should be better known, since most people dealing with calibration know the underlying assumptions. But that's a good place to start. "Everyone" knows Gauss' rules for applying Least Squares calculations, but nobody seems to have thought them through and applied them to the question of chemometric calibrations. Three pertinent ones are here: A) The errors in the Y (dependent) variable are RIND (Random, Independent, Normally Distributed) B) There is no error in the X (independent) variables C) The model is linear in the coeffiicnts We will dispense with C by assuming that we are expressing the constituent concentrations in the appropriate units, so that the assumption holds. In practice, failure of that assumption to hold seems to be the largest source of error in most calibrations, but is mostly beside the point for our current discussion. There is much written, and much heat (and some light) generated over the questions surrounding the error in instrument readings, and other potential error sources in NIR analysis. Let's ignore all that, and also ignore the fact that there is not, never was, and never will be an experimental measurement made that is not subject to error, a fact that Gauss well knew despite his proclamation that independent variables must have zero error. So why, then, did he say that, and what did he really mean? And how is it that despite that statement, Least Squares calculations have been (successfully) performed in all sorts of applications in the 200+ years since then? And if we do ignore all that, what do we expect to happen? For starters, what Gauss's statement really means is that as long as the errors of the independent variables are "small enough" then we can treat the data as though those errors really are zero, because they will have no or negligible effect on the model produced from that data, and no or negligible effect on the values predicted from that model. How small is "small enough"? The third edition of Draper and Smith (see page 90 for the univariate case and page 231 for the multivariate case) provides exact epxressions for calculating the effect of a given amount of X-error, but a ball-park figure can be obtained by simply multiplying the error of a variable by the corresponding coefficient (for multivariate models, calculate the RMS of all those terms). If that product is small compared to the error of the Y variable, then the error is "small enough". As I said, there's a lot more heat than light generated on the subject, as well as the loss of who-knows-how-many trees for papers trying to figure it out, but in practice that has been the basis for the successful application of Least Squares for over 200 years. Since errors add according to their squares, an error in X corresponding to 10% of the Y-error (after multiplication by its coefficient) only increases the total error by 1%. So despite all the uproar over the issue, except in extreme cases models are relatively insensitive to errors in the X variables. The only "real" error in the X-data is the random electronic noise, which can mostly be shown to be very small comapred to the Y-value error (see, for example, Anal Chem.; 58(13); p.5814 (1986), where the electronic noise level is an order of magnitude below even the sampling error, in a "typical' calibration situation). So let's assume that we really have zero error in the X data, and go back to my previous queston: what should we expect to happen? Regression theory tells us that in this case, there is no error contributed by the X data (by definition) and therefore the only sources of differences between the actual (reference laboratory values) and the values form the model are due to the error in the Y-variable (the reference laboratory error) and the lack of fit of the model to the data (the source of which is failure to include all the pertinent variables). A key point, usually missed here, is that the errors in Y are constant, since those errors are fixed in the Y data when the reference laboratory values are measured. They are therefore independent of the model used, any lack of fit and also of any residual errors in the X-values (even ignoring the assumption that the X-errors are zero). It also means that the errors in the Y-value represent the limiting value for the differences between the reference laboratory results and the values from the model (assuming no overfitting, which can be defined for our current purposes as fitting the model to the errors, rather than to the data). Any statistician can tell you that if you select different sets of samples you will get a somewhat different model, due to the differences in the errors of the Y-values in the two sets. However, since each model effectively averages the effects of all the samples, the models each effectively model the average of each data set, which (as statistical theory also tells us) are all taken from the same ppopulation. The net effect is that the various models are all estimates of the same underlying population and therefore are more nearly the same than you might otherwise expect. Therefore in these circunstances, the only property that can change, and cause changes in the differences between the reference laboratory values and the values calculated from a given model are changes in the lack-of-fit. If in fact your data includes sufficient representation of all the necessary variations, then the lack-of-fit error will also approach zero. Then the only error left to cause a difference between predicted and actual values is the error of the Y-data, which, as we noted above, is a constant for any given set of Y values. Significantly, that does NOT depend on the functional form of the X-data. This is all idealized, of course. In reality, we've learned that using the wrong units for the concentration of the analyte introduces an inherent non-linearity into the relationship between the analyte values and the spectroscopy. True, we've only rigorously proven it for the case of clear liquids, but I believe that it can't be less worse in other types of samples, even powdered solids. I see no reason for the physics of the interaction of the light with the sample to be different between liquids and solids, even if the consequences of that interaction differ. That may make the situation more complicated, but not less, and so the effects that we've learned about in liquids will still apply. We've also seen that even in liquids the consequences of the use of the wrong units are much larger than any of the other errors from properties that we know about the samples, so I can't help but believe that the effects of differences between different models of the light interactions could be worse than what we've already dealt with. This being the case, since any chemometric modelling we do is already constrained to create a linear approximation to a much larger source of non-linearity than the difference between modes of light interactions cause, the modelling process can easily deal with the nonlinearities induced by the light-interation differences almost incidentally while dealing with the major nonlinearities induced by the unit errors. So how does that affect our main discussion? Once the calibration model has reached the point where it has dealt with the various sources of non-linearity, it has also inherently included all the information present in the data set that can help model the analyte values (in either the correct or the wrong units). What does that leave us with: X-variable error => 0 Lack of fit error => 0 Y-variable error = Lab value error Small variations due to electronic noise Therefore total error = Lab value error + electronic noise. Under these conditions it doesn't matter which function you use, not because it's unimportant, but because it's not the limiting factor in determining the error of the calibration. A serious mistake that has been made over the years has been the comparison of the total errors of calibrations to determine such underlying properties as to which data transform is "better", which functional model of reflection is "better" etc. In practice it was necessary to make such simplifications in order to prevent routine calibration exercises from each becoming a major research project. But when we want to exmaine the underlying properties of what affects a model and how it affects it, it becomes necessary to go back to fundamentals and examine them one at a time, setting up appropriate experiments that can allow us to extract the different effects. \o/ /_\ | ||
| Jos� Ram�n Cuesta (jrcuesta) Member Username: jrcuesta Post Number: 12 Registered: 11-2009 |
Dear Karl, Thanks a lot for this nice explanation that opens to me new ideas to interpret the spectra. I use it also quite often after sorting the spectra by constituent value and substrating the higher from the lower in order to interpret where the bands for that constituent can be. That helps me to interpret the loadins and the coeficients. Thanks again and best regards. Jos� Ram�n | ||
| Karl Norris (knnirs) Senior Member Username: knnirs Post Number: 51 Registered: 8-2009 |
 Thanks Don and Howard for the �banter�. I used a very strong phrase, to attract attention, but I did not explain why I subtract spectra. I subtract spectra to obtain information about the sample and the instrument used to make the measurement. I hope the image I am attaching can be observed. The top spectrum on the left shows the Log(1/R) spectra of two ground wheat preparations for the 1200 to 2400 nm region, but they are two different preparations drawn from a container of well-mixed ground wheat. The Reflection data were measured in 2001 with a Foss-NIRSystems model 6500 equipped with a spinning sample cup. This looks like one spectrum, but it includes one spectrum in red and a second in blue. If you zoom on this image you can see that the red spectrum has higher values at all wavelengths. The spectrum directly below is the difference spectrum subtracting the blue curve from the red curve, and note the difference varies with wavelength from about 0.0006 to about 0.0043. Note also the difference spectrum looks somewhat like the original spectra. These two preparations are not identical, and such preparations are never identical. My observation is that these two preparations of the same sample differ in scatter properties causing the red spectrum to have a longer optical pathlength than the blue spectrum. However, something else is happening, because the wavelength region above 2000 nm does not match the spectral character of the original spectra. Why ? I suggest instrument stray light is distorting the spectra for both preparations above 2000 nm The distortion is small enough that it is not detected in the Log(1/R) spectra, but it is there. The other observation on the difference spectrum is that it shows instrument noise above 2000 nm, and I suggest the sharp dip in the 1626 nm region also represents instrument noise, because I can not explain it otherwise. The two spectra (red and blue) on the upper right are also two preparations from a single container of ground wheat. These two spectra are so alike that it looks like a single spectrum, but the difference spectrum shows that they are not identical. My observation is that the difference in scatter properties of these two preparations is very small, but the red spectrum has a slightly longer optical path. Instrument noise is dominant in this spectrum making it difficult to interpret spectral changes, but I think I can see the effect of instrument stray light. Note that the peaks in the difference spectrum do not correlate with the absorption bands in the original spectra, which could be caused by small differences in the composition of ingredients in the two preparations. These observations are made on random a selection of two wheat samples from a total of 198 samples where two preparations were made on each sample fro a total of 396 spectra. My friends at NIRSystems may be unhappy that I suggest imperfections in their instrument developed more than 15 years ago to measure the composition of grains and oilseeds, but it was an excellent instrument as demonstrated by the next paragraph. The performance of this instrument in predicting the protein content of wheat is limited by the reference method precision. Many of you know that I like to brag about my derivative-ratio regression for calibrating on scattering samples. For these data I averaged the spectra for each of the 198 samples, and developed a single-term 2nd Der- ratio calibration for percent protein. My calibration provided an SEC of 0.095 % for samples varying from 9.0 t0 17.0 % protein, and this calibration predicted the protein content of the 396 non-averaged spectra with an error of 0.140 % protein. Please note that theory indicates that averaging random-noise data should reduce the error by the square root of 2, so I fit the theory. Please also note that the reported precision of the reference protein values was 0.109 % for wheat samples. | ||
| Donald J Dahm (djdahm) Senior Member Username: djdahm Post Number: 63 Registered: 2-2007 |
Yes, I read the post. It is awesome, brilliant, and inspirational!! It is also not really relevant in answering the question posed at the beginning of this string. When we deal with scattering samples, there are advantages for using the metric { log(1/R) }, but the function will not be linear with either concentration of absorber or thickness of sample. I can think of two reasons why it works as well as it does. First, over a short range, any monotone function can be fit reasonably well by another sloping in the same direction. Second, our chemometrics work (essentially) equally well regardless of the metric we employ. (Jim Reeves taught us that in his Hirschfield Award Address, and a related publication.) Given that there are factors that make things quite complex, there are two extremes that we understand very well. On is the case of no scatter, which is so beautifully simple that we apply it to everything in sight. For a sample for which one layer if a sample transmits a fraction �t� of the light, a sample having �n� such layers will transmit { t^n } of the light. The second is the case of no absorption (The assumption of radiation not diverging must be highlighted.) For a layer of a sample that remits a fraction �r�, and transmits a fraction �t�, the total light remitted from �n� layers will be { nr/(nr+t) }, while the total light transmitted is { t/( nr+t) }. Notice one is an exponential; the other is not. QED! | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 430 Registered: 9-2001 |
Don - I think what we're seeing is a resurgence of the dichotomy that has always existed in NIR: the difference between what we'd like to be able to do in the direction of connecting the methodology to the rest of Science, and the practical usage and limitations we encounter. I hope you received the explanation I recently wrote, in a different thread of this discussion board: http://www.impublications.com/cgi-bin/discus/show.cgi?5/11293 where I explained the practical reason for the use of the log(1/R) data transform of reflection data. As I explained there, however much time and effort we expend in trying to determine the correct model for explaining diffuse reflection phenomena, and making the necessary maeasurements in order to calculate that for any given sample, until we can convert that into an equivalently simple methodology (such as we can use with log(1/R)) for adoption into practical application, it won't enjoy widespread usage. Thus while the arguments over the behavior and benefits of the log(1/R) transform, as well as other potential ways of recasting the raw optical data, are interesting and important, we have to keep in mind the fact that it is the practical application of the theory to routine analysis that carries the day. That also does not mean that we should cease our attempts to improve and advance the theory, but it does mean that we have to be realistic about the benefits we can expect to achieve. \o/ /_\ | ||
| Donald J Dahm (djdahm) Senior Member Username: djdahm Post Number: 62 Registered: 2-2007 |
Howard: We should not discount the possibility that Karl did not literally mean to "Forget Theory!", but not to pay to much attention to our banter (which I hope the world understands is a source of enjoyment to both you and me. On a good day, I also find it a bit inspirational.) To those of you who don't know me personally: Karl is a gentleman, and my writing sometimes sounds like it's coming from a "jerk". However angry the words sound, I am not an angry man. I love life. I love my work. I am greatful to the NIR community for letting me be a part of it. I also hope to continue to rub theroy in your face. Don | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 427 Registered: 9-2001 |
Karl - I personally can't forget theory, and I believe that we, as a community, shouldn't. Without theory to tie the known facts together, Science degenerates into a disorganized collection of isolated facts. Experimenation is important in order to learn what the facts are, but without a theoretical framework to organize them, they provide no guidance as to what further facts are important to learn about. Do we have all the facts we need? NO. Do we have all the theory we need? NO. Is developing further theory difficult? YES. Is developing further theory important? YES. Who's going to do it? Nobody, if we don't. The fact that I agree with your statement that it's often possible to subtract reflection spectra (sometimes after a suitable empirical transformation) doesn't change any of the above. If anything, it only emphasizes the incompleteness of our current theoretical framework for describing the facts that exist. Don may grit his teeth at this, and I can't blame him, after the wonderful progress he has made in advancing the theory; nevertheless it still remains incomplete since it doesn't describe all the observable known effects. "Theory guides, experiment decides" is a more-or-less well-known mantra of science; that's how Science advances and I think it applies here. Both are needed and both are equally important. In physics 100 years ago, when obvervable effects could not be explained by existing conventional theory, new theory had to be developed to explain those effects, and that's why we now have quantum mechanics and relativity. Maybe we need something comparable, to explain the way light behaves in scattering media. If I understand Don's work correctly, he did in fact develop a new theory, that explains the behavior in the one-dimensional case although I'm not sure how complete that is, either, even for the one dimensional case. But it's obviously much more difficult to extend it to three dimensions. Does that mean we (again, as a community) shouldn't try? NO NO NO Does that mean that Don can ignore the facts that current theory can't explain? NO NO NO NO NO \o/ /_\ | ||
| Karl Norris (knnirs) Senior Member Username: knnirs Post Number: 50 Registered: 8-2009 |
Rugin, Forget the theory, I subtract spectra in log(1/T) and log(1/R) format to see the changes. If I subtract two spectra recorded one after the other, I see mostly instrument noise. To me the subtraction is a very useful tool. | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 426 Registered: 9-2001 |
Don - since Ruqin is an admitted newcomer to NIR spectroscopy, I didn't want to make any assumptions about what he might or night not have previously learned. Perhaps I should have emphasized the fact that my discussion was applicable mainly to clear liquid samples. However, what I had in mind was to point out that both Beer's Law and your recent work, which apply to different situations, are both models of the optical behavior of materials, which attempt to explain the behavior in those situations. However, the world does what it does regardless of whether we mere humans can explain it or not. Certainly your Representative Layer theory works better than Beer's Law to describe the optical behavior scattering samples, but I know that you're also the first to admit that it doesn't necessarily work for all situations. It was in that context that I said spectral subtraction works - - - when it works. In that same context, spectral subtraction will work when Representative Layer theory applies, and is applied. Using a correct conversion of the remittance value will, almost by definition, linearize the function and thereby allow manipulations of the data such as spectral subtraction to be valid methods of analysis. \o/ /_\ | ||
| Donald J Dahm (djdahm) Senior Member Username: djdahm Post Number: 60 Registered: 2-2007 |
Howard: I�m guessing that our �newcomer� understands how to do �referencing� so that it would give results equivalent to spectral subtraction. He says: �Theoretically, this way I could also get the spectra of the mixture subtracting A, and the result should be the same. However, the result turned out to be quite different.� He�s a PhD student. I think that somebody taught him in a way that led him to believe that his soil samples should �follow Beer�s law�, whether they used those words or not. At the risk of being inflammatory, I wonder who that fool was. Fifty years ago, the assumption that samples in remission should �follow Beer�s law� might have been a logical hypothesis for exploration. Apparently, there was a belief that if you did proper matrix referencing, you would get an undistorted spectrum of the analyte. Over time, careful experimentation showed that to be untrue. A classic experiment in this regard was reported in the theory chapter of the first edition of the �Handbook�. At that time, it may have been fair to say that we did not have a way to explain these things (note the similarity to your words), but we have a better theoretical basis now. In Chapter G of our book, we gave (what will seem �endless�) examples and showed how theory predicted the non-linearity which was observed. This negates the idea that spectral subtraction is a viable tool for remission samples for the general case. In that chapter, there is also an example or two which describe conditions when we would expect a linear response. I find your suggestion that spectral subtraction sometimes works for reasons that are not understood to be counterproductive. I would submit that it will probably be easy to understand the theoretical basis for the non-linearity in the samples that were mentioned, if one was in possession of the data. I wish I could convince you, as a highly respected member of the community that a starting point of �Beer�s law is worthless except in certain specific circumstances� is better than �Beer�s law is a reasonable approximation for most situations�. We should be surprised when spectral subtraction works for scattering samples, rather than disappointed when it doesn�t. But I fear the fools will keep on winning. | ||
| Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 424 Registered: 9-2001 |
I'm going to put my foot in my mouth by partially disagreeing with Don in the area where he's the acknowledged expert, and say this: despite the fact that there is no Beer's Law (or equivalent) for scattering samples, we do occasionally find that spectra can be subtracted. To my mind, this tells me that the physics "works" despite the fact that we do not have way to explain it. On the other hand, I agree with the rest of his comments, although I think he may not have explained his meaning in sufficient detail for a newcomer to follow. What I believe Don means is this: When we measure the spectrum of a sample against a "background" of any sort, what we are doing (or SHOULD be doing) is to compare the energies measured from the sample versus the background, in relative terms. Here we want to know whether the sample reflects (or transmits, or whatever) 50% of the energy that the background does, or 10%, or 90%, or whatever the relationship is between the sample and the background at a given wavelength. Ordinarily this is done by measuring the response of the instrument (which is assumed linear with respect to the actual optical signal) to the sample and to the background, and then compute the ratio of the two measurements. This computation results in a transmittance value, or a reflectance value, depending on the optical setup used when the measurement was made. These measurements cannot be meaningfully subtracted from each other, they must be divided one (sample) by the other (background). For a number of reasons, that I will not go into here, transmittance and reflectance values, once computed, are often transformed by taking the negative logarithm of those transmittance or reflectance values; in either case the resulting values are called Absorbances. Absorbances can be meaningfully subtracted one from another, since the subtraction of logarithms is equivalent to division of the original values. In your experiment, when you did the subtraction of spectrum A from the spectrum of a mixture, you should have been doing that on the absorbance values. If not, that is part of the cause of your difficulties, since the difference between two transmittance values is not easily related to physical effects. Similarly, if you scanned the samples and scanned A as the background, you should have converted the two spectra to absorbance values before subtracting. Otherwise you should have divided the spectrum of the mixture by the spectrum of A, not subtracted it. Another possible reason for getting poor results in your experiment is a notorious property of scattering samples, especially powdered solids: when you measure the spectra of such a sample multiple times, when the sample is disturbed between measurements, you do not get the same spectrum each time. It is a well-established fact that under those conditions, the spectrum will vary in a manner which is systematic across the spectrum, but random in magnitude and value, from spectrum to spectrum. You should establish, for your samples, the magnitude of this effect. You should do it for one or more "typical" mixtures as well as for the pure materials. It is easily done, simply by measuring the spectra several times, while shaking up or otherwise disturbing the samples between measurements. It is comomon to try to restore the sample surface to a uniform condition each time, before measuring the spectra, by pressing it down with a flat piece of glass or other material, and exerting a uniform pressure. In general, while this procedure will help minimize the spectral variations, some variation will still happen. \o/ /_\ | ||
| Donald J Dahm (djdahm) Senior Member Username: djdahm Post Number: 59 Registered: 2-2007 |
I assume that you are looking at Remission Spectra and that you have used log(1/R) as the metric. For spectral subtraction to work, the spectra must be additive using a linear metric, or as we say: �follow Beer�s law�. Non-scattering samples in transmission are the only samples that can be expected to �follow Beer�s law�. THERE�S NO BEER�S LAW FOR SCATTERING SAMPLES!!! | ||
| Ruqin Fan (rusie) New member Username: rusie Post Number: 1 Registered: 5-2011 |
Hello, I am a PhD student studying soil organic carbon using infrared spectroscopic technique. Recently I am confused with the following question.I got the spectra of a mixture of A and B, and then I subtracted spectra of A from spectra of this mixture. At the same time, I scanned the mixture with A as the background. Theoretically, this way I could also get the spectra of the mixture subtracting A, and the result should be the same. However, the result turned out to be quite different. I don't understand why. Anyone who knows about this please help me.Thanks a lot. |