Should clinicians be sceptical about the reported findings in oncology randomised clinical trials?

8th June 2023

A recent analysis suggests changes to primary endpoints occur frequently in randomised clinical trials in oncology, representing a risk of suboptimal – or even potentially harmful – patient care, together with outcome reporting bias. Rod Tucker investigates what this means for clinicians.

Evidence-based clinical practice is a fundamental paradigm of modern medicine and of huge importance in areas such as oncology where treatments have the potential to be curative. Consequently, there is an expectation that the data derived from randomised clinical trials (RCTs) – which reflect the highest level of scientific evidence – are complete, accurate and unbiased.

Equally important in RCTs is the primary endpoint. This represents the most important outcome and serves to assess the main objective of a trial. When designing a randomised trial, a crucial principle is that the endpoints, especially the primary endpoint, is set in advance. Failure to do so, can introduce bias but, more worryingly, creates opportunities for manipulation.

A lack of transparency

The importance of setting the primary outcome prior to commencing a study and not deviating from the original protocol, was first highlighted in 1990 by Jay Siegel, a physician and research scientist working for the FDA in the US. Siegel discussed how the published description of a study could differ from the original protocol – for instance, changes to the primary endpoint, sample size or statistical tests.

Thus, in a double-blind RCT, if researchers discovered once masked allocation was revealed, that the results didn’t suit their needs, it might be possible to ‘improve’ the findings through manipulation of the data. Siegel suggested that this might happen by reporting on fewer patients or even focusing on the findings from a particular subgroup. In the absence of transparency, he felt that both the pre-publication reviewers and those reading the final published article would not be aware of any retrospective protocol modifications and thus be unable judge the reliability of the final results.

This lack of transparency in clinical trial reporting, exemplified by inadequate access to trial protocols and selective reporting in the final published results, has been suspected for many years, but there was a lack of proof that the practice was used.

This suspicion was confirmed in a 2004 analysis by researchers from Oxford. In an analysis of 102 trials, the researchers uncovered that 50% of efficacy and 65% of harm outcomes per trial were incompletely reported. In addition, when the published articles were compared to the original protocol, 62% of trials had at least one primary outcome that was changed, introduced, or omitted.

Later work simply re-affirmed this finding. In a 2015 systematic review of registered and published outcomes in randomised trials, researchers found that discrepancies between registered and published outcomes of clinical trials were common. Furthermore, this occurred regardless of either the body funding the study or the journal in which the articles were published.

Frequency of primary endpoint changes

But did the use of selective reporting influence trial outcomes? In other words, how often did researchers alter the original primary outcomes to ‘fit’ the available data and consequently only publish positive findings?

This was the question considered in a recent study published in JAMA Network Open by a multi-disciplinary team of US researchers. Focusing on RCTs in oncology, the team examined the frequency with which changes to the primary endpoint were made (i.e. deviations from the original protocol) and, more importantly, if these changes were reported by the authors of a published study.

In addition, the team went one step further and also explored whether there was a relationship between changes to the primary endpoint and the trial outcomes. They examined possible changes using three different methods: the history of tracked changes on ClinicalTrials.gov; self-reported changes noted in the article; and changes reported within the protocol, including all available protocol documents.

With data from the inception of ClinicalTrials.gov to February 2020, researchers uncovered 755 trials, of which 19.2% (145 trials) had a primary endpoint change using at least one of the three detection methods. Somewhat concerning was how among these 145 trials, 70.3% failed to disclose the change in the primary endpoint within the published manuscript. In further analysis, the researchers found that changes to the primary outcome were significantly associated with trial positivity (odds ratio, OR = 1.86, 95% CI 1.25 – 2.82, p = 0.003). While the study did not provide ‘proof’ of selective reporting, the findings did suggest that researchers had changed the outcome of interest to, as Siegel suggested 30 years earlier, ‘improve’ the results.

There are, of course, perfectly legitimate reasons for making changes to a primary endpoint in a study. For example, a high rate of discontinuation or slow accrual of suitable patients would make it difficult to fully assess the impact of a particular intervention. It is also possible that, as the trial continues, emerging evidence from other studies may suggest a more appropriate endpoint. Nevertheless, it is incumbent on trialists to communicate such changes. Based on the current evidence, this does not appear to be a common practice.

Could RCT reporting standards be improved?

Clinical trialists have, for many years, had access to the Consolidated Standards of Reporting Trials (CONSORT) guidance, which was first introduced in 1996. This provided a checklist and flow diagram that authors should use for reporting a randomised clinical trial. Updated in 2010, the statement was further revised in 2022, providing 17 outcome-specific items that should be addressed in all published clinical trial reports, designed to increase transparency and hopefully to minimise the risk of selective non-reporting of trial results.

Although revising primary endpoints after commencement of a clinical is not unreasonable, especially in light of new and relevant scientific knowledge, the practice should remain uncommon. A further difficulty is that many high impact journals, while endorsing CONSORT, fail to ensure compliance with the guidance, as revealed in a recent analysis, with many rejecting correction letters documenting study shortcomings.

Whether reporting of changes to primary endpoints or other outcomes will be mandatory in the near future remains uncertain. As a result, with nearly a fifth of trials failing to document such changes, it seems likely that selective reporting will continue to plague trials. At the present time, it seems that full transparency is an ambition as opposed to a reality. Clinicians therefore have every right to remain somewhat sceptical about the reported findings in oncology randomised clinical trials.