When trials are cut short, who benefits?

Clinical trials stopped early for benefit often grab headlines but it pays to look closely at the fine print. Does speeding a potentially effective drug to market benefit patients or does it benefit drug companies by shortening the road to FDA approval?

Left trying to discern the details are internists, who face increasing pressure to understand how issues of overestimation and bias will play out in their practice among their patients.

“Look beyond the hype at clinical trials stopped early for benefit, particularly when the effects are too good to be true,” said Victor M. Montori, ACP Member, associate professor of medicine at Mayo Clinic in Rochester, Minn. “One should be worried about applying [those results] in practice.”

Dr. Montori first addressed the issue in “Randomized trials stopped early for benefit,” published in the Nov. 2, 2005 Journal of the American Medical Association. The study reviewed 143 randomized clinical trials stopped early for benefit, 92 of which were published in five “high-impact” medical journals, the New England Journal of Medicine, Lancet, JAMA, Annals of Internal Medicine and the British Medical Journal.

Stopped trials were usually industry-funded drug trials addressing such issues as acute coronary syndromes or lung cancer. The trials were stopped after recruiting about 64% of the planned sample after a median of 13 months follow-up and one interim analysis. Dr. Montori also noted that these trials don't always report why they were stopped early and show inordinately large treatment effects, which may lead to increased publicity, which in turn may influence investigators, journals and funding agencies.

As more and more of these trials are published in top journals and reach the ears of patients, internists have to dig deeper into how the trials were conducted—something busy internists don't always have time to do, Dr. Montori and others said. A key concern is that these results trickle into clinical practice with overstated efficacy and without having all potential effects.

However, others contend that it would be unethical to continue these trials because it would subject some people to inferior treatment. And, stopping early may prove beneficial in the end even when the published data overstated the benefit.

The notion of carrying research out to the end to get all the data doesn't always hold water, said Steven Goodman, MD, PhD, associate editor and senior statistical editor for Annals, and professor of oncology, pediatrics, epidemiology and biostatistics at Johns Hopkins. For example, if someone gets a therapy because of a mortality benefit, it wouldn't necessarily matter if that benefit was 10% instead of 20%, he said.

Taken from a different perspective, Dr. Goodman said it could be that the study with a clear benefit was planned originally to go on too long. He also stressed that the larger estimate from a trial stopped early also will be more imprecise. So if the margin of error is reported it becomes clear that there is more uncertainty about the size of the benefit when a trial is stopped earlier.

“The issue is when a study is designed not thinking about the information that is needed for clinicians and patients to consider these interventions, but because they want the cheapest, fastest, shortest study to get the results so they can go to the FDA for approval,” said Dr. Montori. “Speed becomes the objective, not clarity and completeness of information. That's the fundamental challenge.”

Why it's happening

Critics say the following factors may contribute to the increase in the number of trials being stopped early for benefit:

Limited safeguards. Beware of studies that don't have safeguards to avoid making decisions about data accumulating over time. For example, a study that looks at data after every fifth or tenth patient was enrolled would likely find overinflated positive results.

Media attention. Dramatic results grab headlines. Others don't, for example, if a treatment increases the chance of surviving a heart attack by 1 in 10,000.

Investors. Investors' push for results can pressure investigators to stop the trial and look at the data, according to Robert A. Harrington, MD, professor of medicine, division of cardiology, and director, Duke Clinical Research Institute, Durham, N.C.

Stopping a trial

However, not everyone is convinced outside forces are having an adverse effect on trials.

“While there are examples of areas where sponsors have inappropriate influence on publishing or reporting, I believe it is far less likely that the sponsor can compromise the data monitoring,” said Dr. Goodman.

One person doesn't stop a trial. Rather, many people are involved, noted Dr. Goodman, who authored an article in Annals that found that the average of estimates from trials that use statistical stopping guidelines differ little from the true value (“Stopping at Nothing? Some Dilemmas of Data Monitoring in Clinical Trials,” June 19, 2007.)

Dr. Montori acknowledged the importance of data safety monitoring boards, which are independent groups that monitor the safety of the trial, and that they may actually prevent stopping trials early. But he added that members of those boards, until recently, have had limited information about the potential downsides of stopping trials too early.

It's easy to second guess the data monitoring board's judgment to stop a trial on the chance a drug's effects might prove more dangerous in the future, Dr. Goodman noted. But he cautioned that each situation has to be weighed on its own merits to decide if by getting data out earlier, people can benefit. In those cases, he said, further follow-up can be handled through observational and outcome studies.

“This is what experienced trialists and data monitors deal with,” he said, “reading the tea leaves as the signals slowly evolve.”

In fact, some trials have stopped early with marked benefits, such as some early HIV treatment trials, and others have been stopped early because of harms related to what were standard therapies, such as anti-arrhythmic drugs, noted Jeremy Sugarman, MD, professor of bioethics and medicine at Johns Hopkins.

However, he noted that researchers need a compelling rationale to stop a trial early since you may learn more in longer trials.

For now, Gordon Guyatt, FACP, professor in the department of clinical epidemiology and biostatics at McMaster University in Ontario, Canada, said he realizes he's in the minority in believing that all trials should continue to the end.

“I don't think we should stop trials early for benefit at all,” he said. “Short of that, take as few looks as possible and look as late as possible.” Negative consequences may emerge as trials continue, and systematic reviews cast further insight.

The role of journals

Some say the drive to publish in top journals, and editors' willingness to publish these studies, is also a factor (also known as publication bias).

“Investigators and patients want to be part of a study that's a winner, that shows a positive effect,” said Dr. Montori who wrote articles on the topic that appeared in both JAMA and Annals. “Plus, researchers and sponsors who will profit from the published findings … are a strong force saying, ‘Don't mess it up. Report now.’”

Dr. Goodman added, “This is about the general appetite for novelty and good science. This is something the journals struggle with.”

Good science?

Critics say stopping a trial early without enough data unfairly puts the onus on physicians to understand all the subtleties of the issue, and then explain them to their patients (see sidebar “Talking to patients about clinical trials”).

“We haven't done a good job with keeping the practitioner in mind when putting out this information,” said Dr. Harrington at Duke.

Physicians need to understand stopping rules, which prompt investigators to stop a trial because an intervention has unrecognized potential harms or benefits larger than expected, making it unethical to continue one way or another.

Next, how many events are enough? Dr. Montori said a best guess is somewhere between 200 and 400, with earlier termination acceptable when the harms are already known or when harms and costs are known to be negligible.

Dr. Montori remembers one study of perioperative beta blockers (N Engl J Med. 1999 Dec 9;341(24):1789-94) that stopped early when an interim analysis showed a 100% reduction in risk of heart attacks with only nine events in the trial and 80% reduction in deaths with only 11 events in the trial with 20 primary endpoint events total. Perioperative beta-blockers became a recommended intervention in guidelines and a quality improvement target for hospitals.

A later study of more than 8,000 patients taken to completion confirmed that beta-blockers could reduce the risk of heart attacks by 30%, but also doubled the risk of disabling strokes and increased the risk of death by about 30%.

Very few studies absolutely settle a question and physicians always see incomplete information in the medical literature, Dr. Goodman said. He recommended that doctors look at studies combined, not just one study, to make better informed decisions for their patients. He advised particularly noting the limitations sections of articles.

Also look at the methods section and the last paragraph labeled “statistical analysis,” what many doctors consider “a big yawn,” advised Brian Haynes, MACP, PhD, editor, ACP Journal Club PLUS, professor of clinical epidemiology and medicine at McMaster. “Look at what they were looking for and what they found and if there's an incredible difference, don't believe it.”

Dr. Guyatt recommended being skeptical when looking at any drug's first trial. “Find evidence-based sources that pay less attention to what's hot off the press. Do not jump quickly at what is new, especially if it's expensive and disseminated with great enthusiasm,” he said.

Going forward

Steps should be taken to lessen the impact of trials stopped early. Editorial writers can point out study limitations to readers, Dr. Montori said. Already, some data monitoring committees are setting up policies to avoid introducing bias while satisfying requirements of institutional review boards or funding agencies for having stopping rules in place.

For now, busy practitioners need to critically examine the evidence and, in the case of early stopping, be extra critical, Dr. Haynes advised. “I see this as the dark side of progress.”