| Author | Message | ||
     David W. Hopkins (Dhopkins) |
I will summarize the message I have developed over years of working with derivatives and presented most recently in a Short Course at IDRC-2004: How derivatives are calculated does make a difference. Derivatives remove many of the spectral effects that are interferences in diffuse reflection and transmission spectra, the variation in offset and underlying linear slope. The noise in derivatives depends upon how they are calculated, and there is a simple relationship that expresses the noise reduction for certain methods of calculating derivatives. Many methods of calculating derivatives can be expressed as convolution functions applied to the original digitized spectra, including the Savitzky-Golay and Segment-Gap methods. For such methods, the noise reduction is given by the RSSC, the Root Sum of Squares of the Coefficients: The noise in the derivative = RSSC * noise in the original spectrum. The RSSC is equal to the Index of Random Error, IRE, derived by Howard Mark (1991), Principles and Practice of Spectroscopic Calibration, p. 56. I prefer to call this quantity RSSC to emphasize the method of calculation, in line with current nomenclature such as RMSEP. The noise in spectra can be estimated by taking the difference of 2 successive scans of a stable sample, and calculating the standard deviation of the differences over a selected spectral range. The noise in the derivatives can always be reduced by selecting a higher segment or gap in the segment-gap method, or selecting a longer polynomial (keeping the same order of polynomial) in the Savitzky-Golay method, often while not materially affecting the shape of the derivatives. This effect ranges from a RSSC value of 1.414 for a simple 2-point forward difference 1st derivative to less than 0.03 for a 25-point quadratic fit Savitzky-Golay method. Derivatives also can be calculated by Fourier methods, as is frequently applied to interferometric data. In these methods, the RSSC method does not apply. However, the noise in the derivatives can be calculated from successive scans as indicated above. Can anyone share how the noise in the derivatives is affected in this case? I think that there are methods to minimize noise in the Fourier method too, but I do not have experience with the procedures. It is difficult to condense a presentation of several hours into a few paragraphs, but I hope that this is sufficient to elicit some questions and comments. Best regards, Dave | ||
| hlmark |
Dave - noise reduction in FTIR is generally accomplished by a fairly straightforward method: in an interferogram, the signal generally decreases as the the moving mirror moves further away from the point of zero retardation, while the noise remains constant. Therefore, by editing out data points corresponding to large values of the mirror movement, the noise is decreased with little loss of signal. The process of reducing noise by keeping all the data up to a given point, and deleting all data beyond that point (equivalent to multiplying that data by zero) is call "boxcar smoothing". There is not a free lunch here, though: you "pay" for the noise reduction with an accompanying decrease in spectral resolution (everything else the same, of couse, in particular, the measurement time), since spectral resolution depends on the total distance the interferometer mirror is moved, and zeroing out data beyond a given point is equivalent to having moved the mirror less. There is another (actually several other) way(s) to reduce noise. Multiplying the interferogram by ANY function that is unity at the point of zero retardation and decreases smoothly and monotonically as you go to points corresponding to larger retardations of the interferometer mirror will reduce the noise. This process is called "apodization" in the FTIR world. The faster and more strongly the function decreases, the more the noise will be reduced - and so will the resolution. There is also another accompanying side effect: using boxcar apodization causes artifacts to appear at the edges of sharp absorption bands. Other apodization functions also cause artifacts, but the nature and strength of the artifacts also increase as the apodization function decreases faster and more strongly - the boxcar is the limit on strength and speed of apodization, and therefore creates the strongest artifacts. Howard \o/ /_\ | ||
| W. Fred McClure (Mcclure) |
Dave, With respect to derivatives and the Fourier domain, what Howard did not tell you, is that when you move to Fourier space, you go from the wavelength domain to the frequency domain. Fourier did us a good turn here - as it turns out contributions to spectral reconstruction is arranged from low to high frequency. This makes it very nice in that the first Fourier pair is related directly to the DC component (or offset) in the spectra. You should review our (Tony Davies and Morimoto) earlier papers on spectral analyses in the fourier domain: 1. McClure, W. F. and A. M. C. Davies. 1994. More on derivatives: Part 1: Segments, gaps and ghosts. NIR news 4 (6): 12. 2. McClure, W. F. and A. M. C. Davies. 1994. More on derivatives: Part 3. Computing derivatives with Fourier coefficients. NIR news 5 (4): 14-15. 3. McClure, W. F. 1995. More on derivatives. Part 2: Band shifting and noise. NIR news 5 (1): 14-16. 4. Morimoto, S. and W. F. McClure. 1999. More on derivatives: Part 4. Resolving overlapping absorbance bands. J. NIRnews 10 (4): 10-12. Now none of our data were taken with an interferometer (or, an FTNIR instrument), the theory is exactly the same regardless of whether you are talking about the interferogram or the "the Fourier Spectrum" obtained by transformation from the wavelength domain. See: 1. McClure, W. F. 1983. Fourier transform techniques in the analysis of near infrared data. In Technicon's 4th International Symposium on Near Infrared Reflectance Analysis (NIRA) Symposium, ed. Anon., 1:18. Tarrytown, NY: Technicon, Inc. 2. McClure, W. F., A. Hamid, F. G. Giesbrecht and W. W. Weeks. 1984. Fourier analysis enhances NIR diffuse reflectance spectroscopy. Appl. Spectrosc. 38 (3): 322-328. 3. McClure, W. F. 1984. Fourier analysis of near infrared spectra. In Proc.of International Symposium On Near Infrared Spectroscopy, ed. D. Miskelly, D. P. Law, and Tony Clucas, 1:43-61. Melbourne, Australia: The Royal Australian Institute of Chemistry. 4. Davies, A. M. C., S. M. Ring, J. Franklin, A. Grant and W. F. McClure. 1985. Prospects for process control using Fourier transformed near infrared data. SPIE 553: 330-331. 5. Davies, A. M. C. and W. F. McClure. 1985. Near infrared analysis in the Fourier domain with special reference to process control. Analytical Proceedings 22: 321-322. 6. Davies, A. M. C. and W. F. McClure. 1986. Advantages and applications of Fourier transform near infrared spectroscopy. Proc. of the International NIR/NIT Conference (Budapest, Hungary) 1: 61-66. 7. McClure, W. F. and A. M. C. Davies. 1988. Fourier self-deconvolution in the analysis of near-infrared spectra of chemically complex samples. Mikrochim. Acta [Wien] I: 93-96. 8. Davies, A. M. C., H. V. Brichter, J. G. Franklin, S. M. Ring, A. Grant and W. F. McClure. 1988. The Application of Fourier-Transformed Near-Infrared Spectra to Quantative Analysis by Comparison of Similarity Indices. Mikrochim. Acta[WIEN-(VIENNA)] I: 61-64. 9. Hoy, R. M. and W. F. McClure. 1989. Fourier transform near-infrared spectrometry: Using interferograms to determine chemical composition. Appl. Spectrosc. 43 (6): 1102-1104. 10. Hoy, R. M. and W. F. McClure. 1990. Fourier Transform Near-Infrared Spectrometry: Using Interferograms to Determine Chemical Composition. In Near Infrared Spectroscopy: Proceedings of the 2nd International Conference, ed. M. Iwamoto and S. Kawano:371-376: Korin Publishing, Tokyo, Japan. 11. McClure, W. F. 1991. Fun with Fourier self-deconvolution. Spectrosc. World 3 (1): 28-30. 12. McClure, W. F. and A. M. C. Davies. 1992. Data compression by Fourier transforms. NIR news 3 (2): 10-11. Fundamentally, apodization is not a problem for broad-band absorptions (i.e. agricultural products). It only begins to be a problem for spectra with very narrow absorption bands. I was very disappointed to find that I could not used the first few Fourier coefficients to predict composition of polymers (furnished by Lois Weyer). In addition,generating derivatives from Fourier space has several advantages. First, generating the various derivatives there is simply a multiplication and a Fourier transformation - none of the complicated tables that are used in wavelength space are involved. It even provides for a 1.5 derivative (half way between a first and 2nd derivative) - for whatever it is worth. Yet, the same problems occur in Fourier space - viz. band shifting due to asymmetry, etc. Noise increases with increasing derivative order (unless you smooth), etc. However, a plus is that you do not lose ANY point on either end of the spectra as you do with boxcar or Savitsky (sp?). I will comment more on the noise problem, but later. Fred | ||
| hlmark |
Dave - Fred is giving you good advice. When you start to read literature on FT and related topics, you should also include Peter G's book on FTIR. He has some falrly extensive discussion on the theory of FTIR, including what are called the "trading rules" between noise, spectral resolution, artifacts and measurement time. Of course, there is also a very large amount of information about FT in the engineering literature; engineers are extremely concerned with the spectral properties of signals that are to be transmitted from one place to another. These include very detailed analyses of the noise behavior and S/N behavior of signals in both domains (the time domain (corresponding to an interferogram, or what Fred is calling the Fourier Spectrum) and the frequency domain, which is what we ordinarily call a spectrum). But most of that uses pretty advanced mathematics, way beyond what most of us would be comfortable with. Peter's book is good in that it is written for chemists, and is mostly non-mathematical. The downside of that is that it does not deal with many of the subjects, such as noise behavior and S/N behavior, quantitatively; it only discusses them in a relative qualitative manner. Computing a derivative in the Fourier domain, as Fred mentioned, involves a simple point-by-point multiplication; this is generically true of all FT operations: any convolution in the spectral domain becomes a multiplication in the Fourier domain, that's a fundamental property of the FT. You take the FT of the spectrum, the FT of the function you want to convolve the spectrum with, and mutiply the two transforms together point-by-point; the result is the FT of the convolution. So then all you need to do is compute the inverse transform of that. It seems complicated, but after the invention of the Cooley-Tukey Fast Fourier Transform (FFT) algorithm, it turned out that the FFT was so efficient that many convolutions were faster to compute (often by orders of magnitude) via the FFT route than by direct computation of the convolution. This made the FT extremely popular in the engineering world. In the case of a derivative, the convolution function (that you multiply the FT of your spectrum by in Fourier space), is a straight line, i.e., a function that is zero at the point of zero frequency in Fourier space, and increases in equal steps to a value of unity at the point corresponding to the highest frequency. As you can see, this is exactly the opposite behavior of a function that will reduce noise. You get a second derivative by multilying the FT of your spectrum by a parabola: again, zero at the point of zero frequency but increasing as the square of the data point number in Fourier space. And so forth. As far as I know, nobody in the chemistry/spectroscopy communities has quantitatively analyzed the noise behavior of that method of performing derivatives, although I'm sure it's well-known in the engineering world. Fred - I'm intrigued by your statement about the "1.5 derivative". Can you give me the citation as to where this is written up? Howard \o/ /_\ |