D. Umbricht
Corresponding Author: Daniel Umbricht, MD, Distinguished Medical Director, Neuroscience and Rare Diseases, Roche Pharma Research & Early Development, F. Hoffmann-La Roche Ltd, 4070 Basel, Switzerland, Phone +41 61 688 3043, daniel.umbricht@roche.com
J Prev Alz Dis 2020;4(7):299-300
Published online June 1, 2020, http://dx.doi.org/10.14283/jpad.2020.34
Dear Editor,
In the viewpoint paper by Aisen and Raman (1) the authors argue against the use of interim analyses in trials of compounds aiming at disease modification in Alzheimer’s disease (AD). I agree with the caution the authors advise when using interim analyses. However, they seem to do it for the wrong reason and fail to identify the bigger issues at hand. As evidence they cite two recent instances in which interim analyses led to declaration of futility of the trials and consequent conclusions and decisions which were not supported by analyses conducted later on in complete or larger datasets. Example one concerns the compound aducanumab where an initial decision to stop its development was later reversed after additional post-hoc analyses revealed a potential signal. The second example involves TTP448 or azeliragon where a phase 2 trial was stopped based on futility analyses but subsequent analysis of all data suggested efficacy.
Closer examination of the examples the authors provide actually demonstrate the opposite of their claims. In the case of aducanumab the interim analysis indicated that the study EMERGE was ‘trending’, whereas ENGAGE was not (http://investors.biogen.com/static-files/ddd45672-9c7e-4c99-8a06-3b557697c06f). The final analysis confirmed these results with EMERGE showing a signal, while ENGAGE did not. However, the decision to stop further development was based on a pooled analysis, whereas the final analysis was done separately for each study. If the two studies had been pooled, the final analysis would have been negative. In addition, major changes to dosing levels were made shortly before the interim analysis which ultimately seemed critical for the reversal of the interim findings but obviously violated what I would call “the symmetry” requirement (see below). The apparent inconsistencies between the interim analysis and results of post-hoc analyses which led Biogen to a reversal of its interpretation of the data, cannot be used as an argument against futility analyses. First of all despite the symmetry violations, the futility analysis actually predicted the final results both for the pooled data and the individual studies accurately. Secondly, the reversal of Biogen’s position was based primarily on the analysis of a subgroup of patients not similarly represented in the interim analysis population. For these reasons, the issue is not the futility analysis but the fact that basic principles and assumptions for the correct use of an interim analysis were violated. As the authors correctly point out, the results of an interim analysis are only valid for predictions if the conditions of the study before and after the futility analyses are the same (symmetry requirement). So doing an interim analysis on a pooled data set but the final analysis on separate samples violates this principle. Similarly, changing dosing strategies, allocation of subjects to different doses and new post-hoc analyses of subgroups (all implemented in the aducanumab trials) that are not addressed by the futility analysis do as well. If such critical parameters or analysis plans are changed during or after the study, the results of any futility analysis are invalid and must not be used for decision making. In the case of aducanumab these principles were not adhered to.
In the case of TTP448 or azeliragon clear conclusions cannot be drawn as the published information describes (2) the futility analysis as ‘using descriptive statistics only’ indicating that conditional probabilities of success were not calculated which may have led to different decisions. Interestingly, the positive results of the final analysis of the phase 2 study led to a large phase 3 trial called STEADFAST. In April 2018, it was announced that STEADFAST had failed to meet its co-primary endpoints and was terminated early in June 2018. Based on these results one may conclude that the futility analysis was ‘correct’ and many resources could have been saved but were wasted on an ineffective drug.
The examples provided by the authors point to a bigger issue, namely the requirement to apply clear thinking and principles, and methodological rigor when planning the conduct and analysis of pivotal studies that include a futility analysis. If the symmetry requirement due to dose escalations or post-hoc subgroup analysis is violated predictions based on results of a futility analysis are invalid. Simply put, the problem is fuzzy thinking and lack of methodological rigor, not the interim analysis per se.
The authors correctly point out the staggering cost of an incorrect decision which can cut both ways: It can deprive patients of effective treatments or allocation of valuable resources to ineffective drugs – resources not available for the development of novel, potentially effective drugs. The challenge to drug development is finding the best path between timely termination of ineffective drugs to save resources and full exploration of effective compounds. Futility analysis, when correctly done, do offer valuable information that can help in making this tough decision.
References
1. Aisen, P.S. and R. Raman, Futility Analyses in Alzheimer’s Disease (AD) Clinical Trials: A Risky Business. J Prev Alz Dis 2020;3(7):195-196.
2. Sabbagh, M.N., et al., TTP488: From futile to fast track. Alzheimer’s and Dementia, 2015. 11(7): p. P290.