Public Protocols? Burying the lede on the TEST Act

Not to be confused with the Test Act.
(via Luminarium)

4 Democratic members of Congress recently co-sponsored the TEST (Trial and Experimental Studies Transparency) Act, which is intended to expand the scope of mandatory registration of clinical trials. Coverage so far has been light, and mainly consists of uncritical recycling of the press release put out by congressman Markey’s office.

Which is unfortunate, because nowhere in that release is there a single mention of the bill’s most controversial feature: publication of clinical trial "supporting documents", including the patient’s Informed Consent Form (ICF) and, incredibly, the entire protocol (including any and all subsequent amendments to the protocol).

How Rep. Markey and colleagues managed to put out a 1,000-word press release without mentioning this detail is nothing short of remarkable. Is the intent to try to sneak this through?

Full public posting of every clinical trial protocol would represent an enormous shift in how R&D is conducted in this country (and, therefore, in the entire world). It would radically alter the dynamics of how pharmaceutical companies operate by ripping out a giant chunk of every company’s proprietary investment – essentially, confiscating and nationalizing their intellectual property. 

Maybe, ultimately, that would be a good thing.  But that’s by no means clear ... and quite likely not true. Either way, however, this is not the kind of thing you bury in legislation and hope no one notices.

[Full text of the bill is here (PDF).]

[UPDATE May 17, 2013: Apparently, the irony of not being transparent with the contents of your transparency law was just too delicious to pass up, as Markey and his co-sponsors reintroduced the bill yesterday. Once again, the updated press release makes no mention of the protocol requirement.]

How Not to Report Clinical Trial Data: a Clear Example

I know it’s not even August yet, but I think we can close the nominations for "Worst Trial Metric of the Year".  The hands-down winner is Pharmalot, for the thoughtless publication of this article reviewing "Deaths During Clinical Trials" per year in India.  We’ll call it the Pharmalot Death Count, or PDC, and its easy to explain – it's just the total number of patients who died while enrolled in any clinical trial, regardless of cause, and reported as though it were an actual meaningful number.

(To make this even more execrable, Pharmalot actually calls this "Deaths attributed to clinical trials" in his opening sentence, although the actual data has exactly nothing to do with the attribution of the death.)

In fairness, Pharmalot is really only sharing the honors with a group of sensationalistic journalists in India who have jumped on these numbers.  But it has a much wider readership within the research community, and could have at least attempted to critically assess the data before repeating it (along with criticism from "experts").

The number of things wrong with this metric is a bit overwhelming.  I’m not even sure where to start.  Some of the obvious issues here:

1. No separation of trial-related versus non-trial-related.  Some effort is made to explain that there may be difficulty in determining whether a particular death was related to the study drug or not.  However, that obscures the fact that the PDC lumps together all deaths, whether they took an experimental medication or not. That means the PDC includes:

• Patients in control arms receiving standard of care and/or placebo, who died during the course of their trial.
• Patients whose deaths were entirely unrelated to their illness (eg, automobile accident victims)

2. No base rates.  When a raw death total is presented, a number of obvious questions should come to mind:  how many patients were in the trials?  How many deaths were there in patients with similar diseases who were not in trials?  The PDC doesn’t care about that kind of context

3. No sensitivity to trial design.  Many late-stage cancer clinical trials use Overall Survival (OS) as their primary endpoint – patients are literally in the trial until they die.  This isn’t considered unethical; it’s considered the gold standard of evidence in oncology.  If we ran shorter, less thorough trials, we could greatly reduce the PDC – would that be good for anyone?

Case Study: Zelboraf

FDA: "Highly effective, more personalized therapy"

PDC: "199 deaths attributed to Zelboraf trial!"

There is a fair body of evidence that participants in clinical trials fare about the same as (or possibly a bit better than) similar patients receiving standard of care therapy.  However, much of that evidence was accumulated in western countries: it is a fair question to ask if patients in India and other countries receive a similar benefit.  The PDC, however, adds nothing to our ability to answer that question.

So, for publicizing a metric that has zero utility, and using it to cast aspersions on the ethics of researchers, we congratulate Pharmalot and the PDC.

Friday Flailing: Medical Gasses and the Law of the Excluded Middle

Buried

Aristotle never actually said "principium tertii
exclusi", mostly because he didn't speak Latin.

under the mass of attention and conversations surrounding the ACA last week, the FDA Safety and Innovation Act (FDASIA) contained a number of provisions that will have lasting effects in the pharmaceutical industry.  What little notice it did get tended to focus on PDUFA reauthorization and the establishment of fees for new generic drug applications (GDUFA).

(Tangent: other parts of the act are well worth looking into: NORD is happy about new incentives for rare disease research, the SWHR is happy about expanded reporting on sex and race in clinical trials, and antibiotic drug makers are happy about extra periods of market exclusivity.  A very good summary is available on the FDA Law Blog.)

So no one’s paid any attention to the Medical Gasses Safety Act, which formally defines medical gasses and even gives them their own Advisory Committee and user fees (I guess those will be MGUFAs?)

The Act’s opening definition is a bit of an eyebrow-raiser:

(2) The term ‘medical gas’ means a drug that is--

‘(A) manufactured or stored in a liquefied, non-liquefied, or cryogenic state; and
‘(B) is administered as a gas.

I’m clearly missing something here, because as far as I can tell, everything is either liquefied or non-liquefied.   This doesn’t seem to lend a lot of clarity to the definition.  And then, what to make of the third option?  How can there be a third option?  It’s been years since my college logic class, but I still remember the Law of the Excluded Middle – everything is either P or not-P. 

I was going to send an inquiry through to Congressman Leonard Lance (R-NJ), the bill’s original author, but his website regrets to inform me that he is “unable to reply to any email from constituents outside of the district.”

So I will remain trapped in Logical Limbo.  Enjoy your weekend.

Faster Trials are Better Trials

[Note: this post is an excerpt from a longer presentation I made at the DIA Clinical Data Quality Summit, April 24, 2012, entitled Delight the Sites: The Effect of Site/Sponsor Relationships on Site Performance.]

When considering clinical data collected from sites, what is the relationship between these two factors?

• Quantity: the number of patients enrolled by the site
• Quality: the rate of data issues per enrolled patient

When I pose this question to study managers and CRAs, I usually hear that they believe there is an inverse relationship at work. Specifically, most will tell me that high-enrolling sites run a great risk of getting "sloppy" with their data, and that they will sometimes need to caution sites to slow down in order to better focus on accurate data collection and reporting.

Obviously, this has serious implications for those of us in the business of accelerating clinical trials. If getting studies done faster comes at the expense of clinical data quality, then the value of the entire enterprise is called into question. As regulatory authorities take an increasingly skeptical attitude towards missing, inconsistent, and inaccurate data, we must strive to make data collection better, and absolutely cannot afford to risk making it worse.

As a result, we've started to look closely at a variety of data quality metrics to understand how they relate to the pace of patient recruitment. The results, while still preliminary, are encouraging.

Here is a plot of a large, recently-completed trial. Each point represents an individual research site, mapped by both speed (enrollment rate) and quality (protocol deviations). If faster enrolling caused data quality problems, we would expect to see a cluster of sites in the upper right quadrant (lots of patients, lots of deviations).

Instead, we see almost the opposite. Our sites with the fastest accrual produced, in general, higher quality data. Slow sites had a large variance, with not much relation to quality: some did well, but some of the worst offenders were among the slowest enrollers.

There are probably a number of reasons for this trend. I believe the two major factors at work here are:

1. Focus. Having more patients in a particular study gives sites a powerful incentive to focus more time and effort into the conduct of that study.
2. Practice. We get better at most things through practice and repetition. Enrolling more patients may help our site staff develop a much greater mastery of the study protocol.

The bottom line is very promising: accelerating your trial’s enrollment may have the added benefit of improving the overall quality of your data.

We will continue to explore the relationship between enrollment and various quality metrics, and I hope to be able to share more soon.

Changing the Rules, Ever So Slightly, For Rare Diseases

At the end of last year, US Reps Cliff Stearns (R-FL) and Ed Towns (D-NY) introduced the Unlocking Lifesaving Treatments for Rare-Diseases Act (ULTRA for short). Despite what its bold name might imply (and unlike many recent congressional healthcare bills), ULTRA is actually a modest and carefully-though-out piece of legislation.

The main thrust of ULTRA is to enable developers of drugs for rare diseases to take advantage of the FDA’s existing Accelerated Approval pathway. Accelerated Approval reduces the initial burden of proof for manufacturers to bring a drug to market by conducting smaller clinical trials that measure a drug’s efficacy against “surrogate” endpoints – that is, endpoints that do not directly measure the disease, but rather other factors that are associated with the disease. This can greatly reduce the time and cost of clinical trials.

To qualify for Accelerated Approval, however, trials for a new drug needs to meet two conditions:

• The drug must be studied for treatment of a serious disease, with unmet medical need


• There must be clinical evidence that improving the surrogate endpoint is reasonably likely to predict real benefit for those with the disease

ULTRA does not change the first criterion, only the second. For rare diseases, there is often not robust clinical evidence to support surrogate endpoints, so the bill alters the language slightly to permit the FDA to accept “reasonable scientific data that support and qualify the relevance of the surrogate endpoint”. In essence, the burden to prove the validity of the surrogate has been relaxed, permitting their use in pivotal trials, and using a surrogate may reduce the number of patients needed for a trial by as much as 50-75%.

Accelerated Approval still requires the drug manufacturer to complete full trials to more firmly establish the drug’s efficacy – it just allows the drug to be available on the market while those full trials are being conducted. ULTRA does not change this requirement for drugs of rare diseases, so in the end it is not lowering the standard for these drugs at all.

Obviously, anything can happen to a bill as it wends its way through congress. But as it is currently written, ULTRA is a highly rational, well-targeted adjustment to current law that should quickly show benefits for patients with rare diseases, and deserves quick action and passage.

(Further reading: the FDA Law Blog has an excellent review of the proposed act.)

DDMAC to weigh in on trial design?

The FDA Law Blog has an incredibly interesting entry regarding last week's Untitled Letter from the FDA's Division of Drug Marketing, Advertising, and Communications (DDMAC) to Novartis.

The letter, regarding a detail aid for Novartis's Focalin XR, accuses Novartis of making "unsubstantiated superiority claims" for Focalin XR in comparison to Concerta. What is interesting -- and completely new as far as I can tell -- is that the claims DDMAC are taking exception to are, in fact, primary endpoints of two controlled clinical trials:

Treatment for ADHD consists of symptom relief over an extended time period; thus, ADHD medications must control disease symptoms over the entire treatment course. However, the referenced clinical studies only focused on one specific time point (2 hours post-dose) as the primary efficacy measure in the treatment course of Focalin XR and Concerta. By focusing on the 2 hour post-dose time point, the studies did not account for the different pharmacokinetic profiles and subsequent efficacy profiles associated with Focalin XR and Concerta over the entire treatment course.

So, in essence, DDMAC appears to be taking exception to the trial design, not to Novartis's interpretation of the trial results. This would seem to be a dramatic change in scope.

I am not familiar with the trials in question -- I will post an update with more information shortly. Of special interest to me would be to understand: Were these pivotal trials that played a role in Focalin XR's approval? If so, did the FDA review them in a Special Protocol Assessment (and therefore are two distinct branches of FDA providing divergent opinions on these endpoints)?

Accelerated Approvals are Too Fast, Except When They're Too Slow

A great article in Medscape reports on two unrelated articles on the FDA’s process for granting (and following up on) Accelerated Approvals of oncology drugs.

First, a very solid review of all oncology drugs approved through the accelerated process since 1992 is in the latest journal of the National Cancer Institute. The review, written by FDA personnel, is in general concerned with the slow pace of confirmatory Phase 3 trials – over a third (18 of 47) have not yet been completed, and even the ones that have completed have taken considerable time. The authors consider process changes and fines as viable means for the FDA to encourage timely completion.

Second, over at the New England Journal of Medicine, Dr Bruce Chabner has a perspective piece that looks at the flip side: he argues that some compounds should be considered even earlier for accelerated approval, using the example of Plexxikon’s much-heralded PLX4032, which showed an amazing 80% response rate in Metastatic Melanoma (albeit in a very small sample of 38 patients).

I would argue that we are just now starting to get enough experience to have a very good conversation about accelerated approval and how to improve it -- still, less than 50 data points (47 approved indications) means that we need to remind ourselves that we're still mostly in the land of anecdote. However, it may be time to ask: how much does delay truly cost us in terms of our overall health? What is the cost of delayed approval (how many patients may potentially suffer from lack of access), and correspondingly what is the cost of premature approval and/or delayed confirmation (how many patients are exposed to ineffective and toxic treatments)?

The good news, to me, is that we're finally starting to collect enough information to make a rational estimate of these questions.