| Author | Message | ||
     Eric LALOUM (Unregistered Guest) Unregistered guest |
I generally don't like to raise sterile polemic on this kind of matters and there might be non scientifical reasons that prevent Howard to admit that SNV is definetely a non linear operation. For those people whose ego enable discussion, I want to add that a linear operation F should obey to the following law : F(aX+bY) = a F(X) + b F(Y) where a and b are scalar and X, Y vectors. It is this formula, which I intended to illustrate in my previous post. Derivative is a linear operation and SNV not. My question was not to know if there is a difference according to the sequence of pre-treatment: It should be an evidence. For those who want to have an example, just make a test with a simple spectrum like a straight line with a slope (the absorbance being a linear fonction of the wavelenght). If you apply 1st derivative in first, it leads to a constant spectrum and then SNV is even not applicable ; but you can apply SNV first and then derivative. I understand the comment of Dave regarding detrend, which is also a founded argument toward applying Derivative berfore SNV. Thanks, | ||
| David W. Hopkins (Dhopkins) New member Username: Dhopkins Post Number: 82 Registered: 10-2002 |
Howard, The wheat spectra results were demonstrably different, because the ratio of the results by the 2 orders was not constant. The zero crossings must be changed slightly, and the ratio of the values as zero is approached from the + or - directions were different, making the ratios different. Of course, the ratios go wild in both + and - directions as the results are divided by nearly zero values. I don't hold much confidence in the simple modeling, the real world of sample spectroscopy is where we operate. It is interesting to know what we expect theoretically, but it is clear that there is a lot more going on with real samples. I had in mind that the question was, do you get the same results when you do the operations in 2 different orders, and same meant identical, not proportional. Clearly they are not identical. The discussion has been interesting and enlightening. Best regards, Dave | ||
| hlmark (Unregistered Guest) Unregistered guest |
Eric - "linearity" in the mathematical sense is a well-defined property, the definition being "no variable or combination of variables being present in higher degree than the first", or something equivalent. The definition is important because linear functions have properties that non-linear functions may not have. Multiplication by a constant is a linear operation, because it doesn't change the degree of the resulting function, even if the space occupied extends beyond the original range of the data. Relationships between data points remain unchanged after multiplication by a constant (or any other linear transformation), e.g., angles between them, relative distances, etc. This is true regardless of the dimensionality (i.e., the number of variables) of the data. Thinking about it differently may be useful to you in helping your intuition make sense of how your data is behaving, but that is a private matter between you and your intuition. When discussing these matters, it's important to stick to the accepted, standard meanings of terms, in order to not increase confusion. There are already enough terms in the field where different people use the same term to mean different things, and the same term to mean different things - and sometimes they're the same people! We really need to stick to standard, accepted meanings of terms when they exist, so as not to sow further confusion. You should not be trying to change the definition used by the community. That said, your definition (generalizing from your explanation, which seems to be an example rather than a definition) seems to conform pretty closely to the standard definition, except that you call some cases "nonlinear", where considerations of linearity really don't apply. The fact that you get different answers when multiplying by different (somewhat arbitrarily chosen) numbers comes as no surprise, but the definition of "linearity" simply doesn't say anything about that, at all. Howard \o/ /_\ | ||
| Eric LALOUM (Unregistered Guest) Unregistered guest |
Dear All, I knew when posting, that this "linear" attribute would bring some discussion. In my mind it has a very strict meaning, but I agree that many different definitions can be given according to the situation. Here is "My" meaning : Let's consider, n spectra digitized in p wavelenghts. Each wavelenght is a variable/vector in R^n. Since the new variables constructed with derivation are linear combination of original ones, they span a subspace of the space in R^n spanned by original variables. Besides, (contrary to what I said in my first post) even Savitsky-Golay derivative, which is based on a moving window, is a perfectly linear operation. For SNV, the new variables that are generated (vectors in R^n) are not a linear combination of the original variables. So they don't span a subspace of the space of the original variables. It is very easy to verify, even with a small matrix... However SNV is effectively a linear operation applied to each spectrum, but since the scalar is diferent for each line (individual), it is not linear in "My meaning". Best, Eric | ||
| hlmark (Unregistered Guest) Unregistered guest |
Dave (Dave H., that is): Most of the things you say are correct, but as it is used the SNV is a linear operation, regardless of the fact that there is a nominal division operation performed. The reason for this is that once you compute the standard deviation of any spectrum, it is a constant for the subsequent operations. Therefore it can be factored out of the data and instead of saying that you are dividing the spectrum by the standard deviation (SD), it is equivalent to say that you are multiplying the spectrum by the inverse of the SD, which is a linear operation. Getting on to the comparison of the order of operations, I also did the exercise of trying it both ways. My "spectrum" was a single synthetic Gaussian band. I set up the X-scale so that there would be unit different between data points, so that the <delta>X term would be unity for all calculations and therefore not affect the magnitudes of the results. As you said, when the second derivative was done before the SNV, the magnitude of the result was approximately 1.5 at the peak of the second derivative curve, while doing it the other way gave a value of approximately two orders of magnitude less. However, correcting for the magnitude difference by multiplying by the ratio of the curves gave a value identical to 15 places in both results. So to the first approximation, the two orders of applying the operations give results that are the same except for a scaling factor (as you would expect for linear operations). What's not clear however, is whether or not, when applied to real spectral data, you would get a different correlation structure from the transformed data in the two cases, which would lead to models that differed by more than just the scaling factor. Howard \o/ /_\ | ||
| David W. Hopkins (Dhopkins) New member Username: Dhopkins Post Number: 81 Registered: 10-2002 |
David and Howard, There are several problems here. The first is, The SNV transform is decidedly non-linear, as it involves dividing the value each wavelength by the Standard Deviation over a selected range: Ai(SNV) = (Ai - avg(A))/SD(A) where the avg(A) and SD(A) are the average and stdev of the absorbance values over the selected range. For spectra with lots of peaks, this tends to make the bands appear at about +2 units and the baseline to be about - 2 units. Then when you take the 2Der, the peaks are lessened because the points are closely spaced, so that there is relatively small change per wavelength interval. This gives the SNV, 2Der results rather low amplitude. When you do the transformations in the opposite order, the low peaks of the 2Der are brought up to the +-2 scale, and so the amplitudes of the 2Der, SNV results are much greater than the other way. I did this with a set of wheat spectra, just to be sure. You can try it on real spectra for yourselves. I used Savitzky-Golay derivatives, but that should not be an issue. David, I don't understand why your single Gaussian did not follow this pattern too. Perhaps you chose a nice broad one, with lots of change per wavelength interval? Anyway, it does not make any sense to apply SNV to single bands, because that should take away all variation in the spectra, if there are no effects on the band width with variations in the band intensities. SNV should only be used when you have lots of bands in the region, and you assume that the major ones should be reasonably constant and should give a constant Stdev, so that the variation in the analyte band of interest is made more clear. That means that the analyte should not contribute in a great degree to the Stdev, and that care should be taken when deciding how to calculate the SNV. It is important to select a good working range of wavelengths for the SNV, and over which range the spectra should be corrected. With respect to the defining equation above, this means that the range of the index variable i need not be constrained to be the same range as used for the avg and stdev calculations. Some software packages do not give the user sufficient flexibility, particularly over this last item. I hope this is helpful. Best regards, Dave | ||
| David Mayes (Davem) New member Username: Davem Post Number: 1 Registered: 4-2006 |
I just did a simulation in DeLight, making a Gaussian curve with an added y^-4 function added. The resulting curve from snv followed by a 2nd derivative is identical if you do the operations in reverse order. David | ||
| hlmark (Unregistered Guest) Unregistered guest |
Dave - what I've read about detrend lately is that it also provides for removing a parabolic contribution to the baseline (such baseline as we have in NIR, anyway), so that should have an equivalent effect to second derivative. Or is this another case of multiple definitions (and meanings, and uses) for the same label? But my question was about your statement that the order of applying SNV and 2nd derivative makes a difference. Do you know why that should occur? Theoretically the two orders of application should be equivalent. \o/ /_\ | ||
| David W. Hopkins (Dhopkins) New member Username: Dhopkins Post Number: 80 Registered: 10-2002 |
Howard, Yes, it is the old observation, a first derivative removes all constant offsets from samples, and 2Der removes all linear slopes. Detrend is the fitting of a straight line to remove background slopes. They do the same function. However, I think 2Der is more effective, because it can remove any low rate of curvature as well. Regards, Dave | ||
| hlmark (Unregistered Guest) Unregistered guest |
Dave - any idea why that should happen? \o/ /_\ | ||
| David W. Hopkins (Dhopkins) New member Username: Dhopkins Post Number: 79 Registered: 10-2002 |
Eric, It is often forgotten that when SNV was introduced, it was always used following "detrend". It is important to do this, because it is not desirable to have various slope components inflating the standard deviations, to various degrees. As the 2Der acts as an effective detrend procedure, I would recommend doing the 2Der first, then SNV. The order does affect the results, and I think there is valid reason to perform the 2Der first. Best regards, Dave | ||
| hlmark (Unregistered Guest) Unregistered guest |
Eric - both SNV and derivatives (of a spectrum with respect to wavelength) are linear methods in the mathematical sense. Theoretically, then, it won't matter which order you do them in. However, applying a data transform to the spectra before executing the calibration algorithm will affect the correlation structure of the data, and therefore may well affect the performance of the algorithm, including the optimal dimensinality of the model. Linearity is only one of the characteristics of a data set that matters. Howard \o/ /_\ | ||
| Eric Laloum (Unregistered Guest) Unregistered guest |
Dear All, It's often useful to combine derivative with SNV processing for near infrared spectra. I wonder if the sequence (SNV or derivative first) has any importance ? Generally, I observe very few differences according to the sequence, but there is maybe some rational or theoretical work studying this ? Intuitively, I would say that it should be better to apply a non linear treatment as the last pre-processing especially before a linear method like PCA, PCR or PLS. I consider Derivative as linear because, the derivative spectrum lies in a subspace of the space spanned by raw spectra since new variables are differences of original variables. (Supposing that nonlinear effects induced by Savitsky-Golay derivative are minor). In that respect, PCA applied on raw data or derivative data, should be more or less equivalent, apart from the dimensionality. So to me, Derivative followed by SNV should be "better" than SNV + Derivative. Any Idea ? Thanks, |