The best model for identification Log Out | Topics | Search
Moderators | Register | Edit Profile

NIR Discussion Forum » Bruce Campbell's List » General, All others » The best model for identification « Previous Next »

Author Message
Top of pagePrevious messageNext messageBottom of page Link to this message

Howard Mark (hlmark)
Senior Member
Username: hlmark

Post Number: 197
Registered: 9-2001
Posted on Monday, June 23, 2008 - 1:58 pm:   

I thought there was a recent thread where the recommendation of "round robin" interlaboratory studies came up, but I couldn't find it. So I'm posting on this thread as having the discussion nearest to the one I was looking for.

In any case, I wanted to notify anyone who is interested about these things, that ASTM does have a Standard Practice for conducting interlaboratory studies, I just received a notice of it's being updated, and so that reminded me to pass the info on:

http://www.astm.org/DATABASE.CART/WORKITEMS/WK20281.htm?A

For those not familiar with interlaboratory studies, they are historically one of the oldest methods, and to this day are still considered the "gold standard" for methods of evaluating the precision, accuracy and other characteristics of any type of chemical analysis. They were developed at a time when, and for use in situations where, there was no "reference value" available to compare a new method to; all comparisons had to be based on the internal properties of the data, so Statisticians worked together with Chemists to devise ways to tell how good a method of analysis was, when there was nothing to compare the results to. And they did a magnificent job.

So if anyone needs to know why and how "reference methods" got to be that way, here is where you can look.

BTW - if anyone can recall where the thread was that the question of interlaboratory studies came up, please copy this post to the appropriate thread.

\o/
/_\
Top of pagePrevious messageNext messageBottom of page Link to this message

Hector Casal (casalh)
Junior Member
Username: casalh

Post Number: 6
Registered: 1-2007
Posted on Tuesday, June 17, 2008 - 8:23 pm:   

Howard

You are absolutely right about ASTM, I should have mentioned it. Also, I should try to participate more in their activities as I benefit tremendously from ASTM documents. I also agree that it is way better than anything like 'government' decree.

Regarding what I would do to tackle material id models for a large number of materials. I would start by deriving a mask from the average spectrum of each material. The resulting data would be used for PCA for the whole lot of materials. Masks are very powerful, given a spectrum of a material that needs to be classified the mask often gives extremely good results. Masks are extremely useful when used with search algorithms.

Hector
Top of pagePrevious messageNext messageBottom of page Link to this message

Howard Mark (hlmark)
Senior Member
Username: hlmark

Post Number: 196
Registered: 9-2001
Posted on Tuesday, June 17, 2008 - 10:03 am:   

Hector - probably only the people involved are aware of this, but the ASTM (which, like USP, IEEE and a whole host of organizations in different fields, is an independent organization that creates standard practices by concensus of participating experts. Anyone who wishes to can have a say in the development of a standard practice. This is probably the second-best thing to a governmental agency doing it. Maybe even better, since the pool of potential experts is an open one.) is in the last stages of releasing a standard practice for validation of multivariate analytical methods. It also tells you what to do but not how to do it, so Hector's advice is good, since that will allow you to try different solutions until you find the one that is best for your products.

\o/
/_\
Top of pagePrevious messageNext messageBottom of page Link to this message

Mark Murphy (qamarkm1)
New member
Username: qamarkm1

Post Number: 4
Registered: 6-2008
Posted on Tuesday, June 17, 2008 - 9:47 am:   

Hector,

You are correct in that the user plays the most critical part in this whole process and that the construction of a suitable validation protocol is the best way forward.

My main concern is where to start with the validation protocol. In order to make things as straight-forward as possible, it would be best to apply the same set of chemometric models to all of the calibration libraries. We have several hundred raw materials to consider under the scope of this project and would like to try and identify a chemometric model that would apply to the majority of materials. As usual, it will not be possible to cover all aspects and, depending on the number found, these would be handled in a differnt manner.

Understandably the intepretation of the spectra should take all wavelengths into account as this will show the maximum number of differences between materials and allow for greater ease of application (as well as fewer outliers). My question was what would be the best chemometric treatment to apply that would be robust enough to handle such a large number of materials yet straight-forward enough so as not to cause complications either in the validation of the library or for possible future materials that may be added.

The main suggestions poses so far would suggest that PCA or SIMCA would be my best option.

Mark
Top of pagePrevious messageNext messageBottom of page Link to this message

Hector Casal (casalh)
New member
Username: casalh

Post Number: 5
Registered: 1-2007
Posted on Tuesday, June 17, 2008 - 9:03 am:   

The subject of material identification/confirmation for the pharma industry is very important as everybody has pointed out. There are no detaild instructions or recipes from authorities and maybe it is not possible to have them. I suspect that I am stating the obvious and I apologize if I am.

I think the most important part in this 'equation' is the user. The user (i.e., the pharmaceutical company) has the develop a very rigorous validation protocol. Think hard, carefully and long about what is the objective and how to test the different methods. Be careful not to develop a method based on the 'solvent' or on differences in baseline (we have all seen examples like these). If the company does not have an expert - hire one! It is too important a subject to be left to the istrument companies providing a one-week training course for a recently hired technician.

Once the user has a rigorous validation protocol have the instrument company agree to a verification period and if their instrument/method does not work, take the instrument back.

We have developed many methods and some of our own algorithms. Sometimes we find that M is enough, sometimes we have to also consider the scores or R, sometimes a discriminant based on distance or angle, etc.

Hector Casal
Top of pagePrevious messageNext messageBottom of page Link to this message

Howard Mark (hlmark)
Senior Member
Username: hlmark

Post Number: 195
Registered: 9-2001
Posted on Tuesday, June 17, 2008 - 8:35 am:   

Mark - don't worry about it. We always digress into interesting side discussions

\o/
/_\
Top of pagePrevious messageNext messageBottom of page Link to this message

Mark Murphy (qamarkm1)
New member
Username: qamarkm1

Post Number: 2
Registered: 6-2008
Posted on Tuesday, June 17, 2008 - 3:08 am:   

Oh dear... I seem to have opened a very large can of worms here!

Firstly, thank you all for your comments, they are very much appreciated.

My general application for the use of NIR is for the 100% identification of incoming raw materials for a pharmaceutical manufacturer in line with GMP requirements.

The main issue that I have is (as Tony has already stated) that each manufacturer says that their instrument/software does it better than the rest.

The respective Regulatory Authorities (ie: EMEA/FDA) have recognised that NIR technology is a major step forward for the qualitative and quantitative analysis of chemicals and drug components yet will not provide any direction on its use apart from stating that it must meet the standard validation requirements that they have.

While I can understand that these agencies need to remain objective by not picking one manufacturer over another, this then presents major issues for the non-expert, like myself, in deciding which manufacturer/instrument/software/mathematical-chemometric model is the most suitable for my needs.

I also think that the lack of concrete guidance from the Authorities also makes it difficult for the manufacturers to develop their products or to gear them for a specific market or application. Some manufacturers are still trying to sell dispersive instruments against a market full of Fourier Transform?

It appears that the general concensus from the manufacturers is the use of either PCA or Cluster Analysis but with the lack of guidance from the Authorities, I though it best to ask an expert or three!

Again, many thanks for your input!

Mark
Top of pagePrevious messageNext messageBottom of page Link to this message

Howard Mark (hlmark)
Senior Member
Username: hlmark

Post Number: 194
Registered: 9-2001
Posted on Monday, June 16, 2008 - 4:15 pm:   

Tony - you're no stranger to the IDRC, so you don't need me to intercede for you!! Of course, it's too late to make such major changes to this year's conference. I think you should talk to Fred, though; after all, he's still the Shootout chair, so maybe for the next conference something can be done.

I would say that if you want to have a qualitative data set for the Shootout, you might have to come up with it yourself, but if you do, I think Fred would be amenable to considering the new type of analysis for the Shooutout.

If you need my support on that, I'm willing to give it. Doesn't make anything automatic, though, there might be other submissions for shootout data that you'd have to compete against. But worry about that when it happens, for now decide what you want to do and go for it!

\o/
/_\
Top of pagePrevious messageNext messageBottom of page Link to this message

Tony Davies (td)
Moderator
Username: td

Post Number: 177
Registered: 1-2001
Posted on Monday, June 16, 2008 - 3:27 pm:   

Hello Howard,

99.99% agreement between us is much better than I could hope for! In fact I think we thrive on disagreement!
I am not sure I would want to be responsible for the quality control of a cancer drugs if the tolerances are that close, but I think we agree on two things:

1) It is really important to get correct identifications.
2) The present multiplicity of systems, most of which have to be set-up by non-experts, is far from an ideal state and is an invitation to errors. (I suggested that it is an accident waiting to happen).

You have certainly given me the encouragement I needed to try to press home our concerns. Can you do anything (through CNIRS?) on your side of the Atlantic?

I did have the thought that the �Shootout� has been too concerned with quantitative analysis but could we find a qualitative database that would be sufficiently large to be useful? Could we get it validated by IR? (Using IR experts �NOT chemometrics!). Perhaps there are ways? But we would also need to return the competition back to the original concept of testing methods/software not �experts� proficiency.

Best wishes,

Tony
Top of pagePrevious messageNext messageBottom of page Link to this message

Michael C Mound (mike)
Senior Member
Username: mike

Post Number: 56
Registered: 7-2007
Posted on Monday, June 16, 2008 - 3:13 pm:   

Howard, et al.,

Howard's remarks are very interesting to me, apart from the obvious, inasmuch as there is a sufficiency of hype from some quarters trumpeting the "100%" effectiveness of API in batch drug tablet or gelcap testing. Whether in those actual words, or phrases to that effect, the impression of error-free validations would seem to be under question, shootouts or no.

For those of us mortals who are more routinely involved with non-drug applications, we have always (speaking for myself, at least) regarded the pharma applications as being sine qua non with respect to reliability, and have held up these paragons as shining examples of NIRS.

How crestfallen we must now feel to find our heroes have feet of clay.

Say it ain't so, Virginia...

Thanks,

Mike
Top of pagePrevious messageNext messageBottom of page Link to this message

Howard Mark (hlmark)
Senior Member
Username: hlmark

Post Number: 193
Registered: 9-2001
Posted on Monday, June 16, 2008 - 2:37 pm:   

Tony - I agree with about 99.99% of what you're saying. The small area of disagreement has to do with the statement, "... a bit too much or little of the correct drug can hardly be life threatening!"

Some drugs, and anti-cancer drugs are probably the most prominent and well-known example, are extremely potent. For most anti-cancer drugs, in fact, the reason they work at all is because they are basically poisons, that's why they have such severe side-effects. The trick is to poison the cancer cells before the normal cells die off in too great quantity and kill the patient.

So the "window of efficacy" for those, as it's called, is usually very narrow: too much and you kill the patient with the "cure", not enough and the patient dies from the disease.

Similar conditions hold for some of the more potent painkillers; that's why abusers so often take "accidental" overdoses and die from them (to say nothing of "drugs of abuse").

So a bit too much can indeed kill you, for those drugs. I suppose it would be harder to make the case for too little, except that if you take too little painkiller when you need it, you might WISH you were dead!

But otherwise I agree with your comments. I would go even further. The CNIRS once tried, a long time ago, to mount a project to determine the correct answers for various algorithms on test data sets, in order find out the truth behind the claims of software manufacturers. Of course there was no cooperation from the manufacturers on this. To this day, we know that different packages give different results on the same data sets; but since nobody knows the "right" answer, nobody can make claims about the correctness or incorrectness of any particular implementation.

Most of us have forgotten, I think, that the original purpose of the "Software Shootout" was exactly to compare the software packages and figure out which gave the "right" answers, and by so doing, force the manufacturers to fix their software if necessary. It didn't work out that way; the "shootout" quickly morphed into a contest for who could get the best results from the given data set. Not a bad premise, but not the original purpose, either.

NIST has test data sets for many algorithms, with the correct answers known by concensus among top-notch computer scientists and statisticians. The only one that I know of that might even be close to being of interest to us is a set for multiple regression analysis, and as you point out, it's PLS that is dominant.

Su-chin Lo gave a talk at Pittcon about 5-7 years ago, where he compared results from several of the major program packages used for chemometrics. One of his main problems was the fact that different packages report different results, and reported them in different ways.

Where he could find corresponding results, he could make comparisons, but still couldn't get beyond "packages A, B and C give the same answer, while packages D and E give answers that are different (from A, B and C and from each other, if appropriate)".

If you could get some official agencies to take an interest, that would be great.

\o/
/_\
Top of pagePrevious messageNext messageBottom of page Link to this message

Tony Davies (td)
Moderator
Username: td

Post Number: 176
Registered: 1-2001
Posted on Monday, June 16, 2008 - 1:45 pm:   

Hello Mark and Howard(Mark),

I recently spoke at a meeting in London, UK organised by the Joint Pharmaceutical Analysis Group (this a Royal Pharmaceutical Society/Royal Society of Chemistry collaboration) when I warned that the present system for NIR identification was sailing into dangerous waters.

I am very concerned that the present lack of directions enables almost every instrument company to push their own (patented?) software as being the best available. Most of these programs have never been tested by external, independent chemometricians and their robustness is unknown. While people are happy for quantitative analysis to be quite tightly regulated (and PLS is dominant); there seems to be an idea that identification is less important. To me the opposite must be true. The wrong drug might be lethal while a bit too much or little of the correct drug can hardly be life threatening!

I suggested that it is time that for us to initiate a debate to encourage the authorities to commission some studies. I cannot claim that I received any support (but that is not unusual!).

I think that the present answer to Mark�s question is that no one knows but we all have opinions! Personally I like canonical variates analysis, it has been around for some time. If you want a more modern method then I would go for SIMCA which is available in quite a few software packages.

Best wishes,

Tony
Top of pagePrevious messageNext messageBottom of page Link to this message

Howard Mark (hlmark)
Senior Member
Username: hlmark

Post Number: 192
Registered: 9-2001
Posted on Monday, June 16, 2008 - 11:53 am:   

Mark - Without any a priori information about the application, my recommendation is usually Mahalanobis Distance, with or without doing a PCA first.

But then, I'm prejudiced!!

\o/
/_\
Top of pagePrevious messageNext messageBottom of page Link to this message

Mark Murphy (qamarkm1)
New member
Username: qamarkm1

Post Number: 1
Registered: 6-2008
Posted on Monday, June 16, 2008 - 11:27 am:   

What do members of the forum consider to be the best mathematical/chemometric model to use for the identification of incoming raw materials?

The ICH initially published guidelines several years ago which recommended combined use of wavelength correlation of the 2nd derivitave spectra and Maximum Wavelength Distance, both applied on the whole NIR range.

Now it appears that the ICH are backing off of such a detailed statement within the guidelines and it seems that it is being left up to the user to decide. The USP currently do not have a published stance on this at present.

Many of the current market manufacturers are using/recommending Principle Component Ananlysis and Cluster Analysis but I am interested in what long term users have been using and would recommend.

Many Thanks.

Mark

Add Your Message Here
Posting is currently disabled in this topic. Contact your discussion moderator for more information.