| Author | Message | ||
     Tony Davies (td) Moderator Username: td Post Number: 145 Registered: 1-2001 |
Just for the Record! Please see my note (January 04) at the begining of this thread that according to Karl there should not be a derivative method associated with his name. Norris regression is what he will admit to! Best wishes Tony | ||
| Dongsheng Bu (dbu) Junior Member Username: dbu Post Number: 8 Registered: 6-2006 |
Dear Alexandra, Norris Gap Derivative is a special case of Gap-Segment Derivative with segment size = 1 (no smoothing). The Unscrambler updated Norris Gap Derivative in version 9.5 in collaboration with Dr. Karl Norris. Dr. Karl Norris has developed a powerful approach in which two distinct items are involved. The first is the Gap Derivative, the second is the "Norris Regression", which may or may not use the derivatives. The applications of the Gap Derivative are to improve the rejection of interfering absorbers. The "Norris Regression" is a regression procedure to remove the impact of varying path lengths among samples due to scatter effects. My personal view is that Norris Gap Derivative performs superior in the cases of optimizing gap-size and smoothing segment separately, such as Norris Regression. Savitzky-Golay Derivative has constraints on polynomial order/derivative order combination. Generally speaking, Norris Gap Derivative has no smoothing option. Someone wrongly treated Norris Gap Derivative the same as Gap-Segment derivative, in which smoothing is available. I didn't find significant pros and cons between Gap-Segment derivative and Savitzky-Golay Derivative, except that one can easily follow the number of unavailable points at two ends in Savitzky Derivative, while Gap-Segment derivative wouldn't. Dongsheng | ||
| David W. Hopkins (dhopkins) Senior Member Username: dhopkins Post Number: 101 Registered: 10-2002 |
Hi Alexandra, If you will send me your email address, I will send you a pdf file copy of a paper I wrote to explain the derivatives in detail. My conclusion is, you can obtain derivatives by either the Segment-Gap method or Savitzky-Golay that have the same resolution and averaging, so that you should use whatever method is available in the software you use. Some software has both methods. You need to optimize the parameters using either method. Usually this means selection of a convolution interval about the same size as the number of points in the half-band width of the sharpest band in the wavelength range in which you are working. I hope this helps you. By the way, I recommend that you change your profile so that your email address is not hidden. That way, you can receive responses directly from me or others. I know, this may seem like a bad security choice, but it has never been a problem for me. You can send me your email address directly, if you are still afraid to post it on the discussion group. Best regards, Dave | ||
| Alexandra Mayer (ampmayer) New member Username: ampmayer Post Number: 1 Registered: 2-2007 |
Hi I would like the algortihm or a good explaning reference of Karl Norris Gap derivaties. Has anyone experience in the difference pros and cons with the gap derivatives compared to the savizky-golay approach Thank you very much Alexandra Mayer Instituto Superior T�cnico Lisboa Portugal | ||
| Erik Skibsted |
Hi I would like the algortihm or a good explaning reference of Karl Norris Gap derivaties (isn't that the name?) Has anyone experience in the difference pros and cons with the gap derivatives compared to the savizky-golay approach Thank you very much Erik Skibsted University of Amsterdam / Novo Nordisk | ||
| David Russell (Russell) |
The best explanation that I have seen was in the manual for the NSAS software package (DOS software for NIRSystems instruments). A brief explanation also appears in a chapter on NIR in agriculture, but I don't recall the name of the volume. I suspect that Karl published it elsewhere. Perhaps Howard Mark knows and could jump in here. Dave Russell | ||
| DJDahm |
I hesitate to speak for Karl, noting that he is probably lurking out there reading this, and my old head does not do as well as it once did on details. I am writing this from memory and without access to my books in the office. First, I would like to recommend Dave Hopkins' article of about a year ago in "NIR news". He did a good job of explaining what was in the NSAS manual. As I recall, in the same issue, Karl has a description of his derivatives. I remember him saying that he wished he had known that Dave's article was coming out, because what is in the NSAS manual was not exactly the same as his method. My thought is that there is nothing all that special about how Karl calculates a derivative. The power is the "systematized empiricism" that his original algorithm embodied. Here is a low-tech explanation. If you have a symmetrical peak of a pure compound, it will have a derivative of zero at the maximum. If that wavelength is used to measure another component, there will be no contribution from the first. By changing the gap and segment used to calculate the derivative (or by changing to a second derivative), the place of zero contribution of an interfering compound can be moved. If you have a calibration sample set and turn it lose trying all combinations of gaps and segments, with luck, there will be a combination where the analyte has good contribution and all the interfering species have none. | ||
| hlmark |
I believe that Karl also wrote up a description of his derivative algorithm and published it in NIR News; unfortunately I don't remember which issue, it was a couple of years back. It's actually a little more extensive than Don's description, though. Here's a sketch of it, without all the details: Basically it is not just a derivative, it is the quotient of a derivative centered at one wavelength divided by another derivative centered at a different wavelength. The division process compensates for what is loosely called "particle size effect" and all the related phenomena that affect the scattering properties of the sample. Each derivative is computed using the "segemnt" parameter, which defines the range over which data is averaged, and the "gap" parameter which defines the spacing between segments that the difference approximation to the derivative is computed for. The segments, gaps, and wavelengths are varied independently for the numerator and denominator terms, and an algorithm is used to optimize each of these parameters. When Karl does a calibration, he doesn't depend slavishly on the computer results, but includes his own interpretation and background knowledge of the spectra and the samples, and how he expects the data to behave, when deciding which values of the parameters to use. Howard \o/ /_\ | ||
| lois weyer |
I liked Don's description. It got to the heart of the power of the approach of varying the derivative shapes. Howard's addition is just that. In the NSAS software, one had the choice of looking for a divisor term which would serve to do the corrections he mentioned. This is also powerful, but is not how I usually used the "Norris" derivatives. The derivative variations alone created excellent "single-term" robust calibrations. Also, one could add linear terms which improved calibrations even more. | ||
| NIRman |
A couple of years ago, Karl Norris sent me some information so I could talk about segment and gap size in the short course I teach. He critiqued my first draft, and I now have 7 PowerPoint slides which (I believe) explains the idea rather well. I've put them into a 65KB file and I'll send it to anyone who asks for it. If there's a way to attach it to this posting, let me know and I'll do it. Don Burns | ||
| Karl Norris |
I'm sorry I did not see this inquiry sooner, but hopefully I can clarify some points. For references I refer you to: NIRnews Vol.9(4) 3(1998), Vol.12 #3 6(2001), and Vol.13 #3 8(2002). The second reference is the most complete. First, there are two distinct items involved. The first is the gap derivative (sometimes called the Norris derivative by mistake), the second is the "Norris Regression", which may or may not use derivatives. The applications of the gap derivarive is as described by Dr. Dahm, and as used by Dr. Weyer to improve the rejection of interfering absorbers. The "Norris Regression" is a regression procedure to remove the effects of varying pathlengths among samples because of scatter effects. This is accomplished by incorporating a divisor into the regression term. The divisor can be the absorbance at another wavelength, a difference between the absorbance at two wavelengths, a first derivative, or a second derivative. The single wavelength divisor does not work well in many cases because that signal contains offset variations as well as multiplier variations, and we only wish to sense the multiplier signal. Hopefully you will study the second paper to learn how the regression is performed, and what an excellent job the regression procedure does on difficult samples. I will be glad to answer further questions. Karl | ||
| Tony Davies (Td) |
Thank you Karl, you came just in time to save me having to look up all the references. Just to emphasise the point. According to Karl there is no such thing as "Norris Derivatives" and the method use in Unscrambler for first derivatives should NOT be associated with his name! Message for Don. Yes it is quite easy to post files. Look at Documentation/Formatting to the left of this message. Please do it they sound really useful! Best wishes, Tony |