Any life scientist who has written a meta-analysis or systematic review of preclinical or clinical studies is likely capable of complaining at length and in detail about the lack of standardization, the myriad ways in which small differences in study design can sabotage any attempt at comparison, the critical details missing from published study methods, the misleading interpretations and summaries observed in the abstracts of a sizable minority of papers, and so forth. From some perspectives the literature is a mess, as you might expect when thousands of people with little incentive to conform to any one viewpoint set forth to pursue their own ideas about how to run a study.
Paper-length complaints about this situation can be found in the literature. You might recall a recent discussion of the harms done by non-standard controls in life span studies, for example. In today's preprint, researchers work through a database of preclinical studies in order and outline all of the problems that they encountered in trying to build any sort of broader body of understanding from a field in which every scientific group takes a different approach when it comes to assessing the effects of interventions on aging. Beyond standardization, rigor is clearly also an issue.
There is increasing interest in interventions targeting the aging process. The "geroscience hypothesis" posits that a shared pathophysiology of aging shapes most chronic diseases and interventions targeting aging will confer larger health benefits than those targeting any individual disease. Research into such anti-aging interventions has grown substantially, including trials repurposing commonly used drugs such as metformin. Given the possible substantial health benefits of slowing aging, the quality of preclinical studies in this area may be especially important. However, alongside the challenges translating results from one species to another, model organism studies have a long history of shortcomings and design flaws.
Here, we systematically analyzed 667 studies from DrugAge, a curated database of preclinical experiments investigating the effects of interventions on aging and lifespan in non-human animals. We aimed to evaluate the quality of reporting and methodological rigor of this literature, assess the distribution of observed effect sizes, and probe for the presence of diverse biases. We also investigated how these features changed over time. We found significant shortcomings in reporting of crucial design features such as randomization and blinding, as well as large variation in reporting quality and effects across species. Non-mammal findings typically did not translate to mammals. For 36 compounds with both mammal and non-mammal experiments, only eight showed a significant lifespan increase in both non-mammals and mammals; the number of experiments and sample sizes for these results were limited. These results are exploratory, and the numbers are small, but they raise hesitation about the direct translation of these results to more complex organisms such as humans.
Furthermore, previous work has suggested that some interventions may have different effects if started late in an organism's lifespan rather than early, and there is significant interest in discovering interventions that slow aging in older adults. In our assessment, we found that most preclinical experiments started the anti-aging intervention early in the organism's lifespan, often prior to sexual maturity, when key senescence mechanisms may lack relevance. Although we did not find a significant difference in the effect of interventions between early and late start experiments, the sparsity of late start results makes this comparison uncertain. Our study clearly highlights the paucity of late start experiments in the literature, a deficit of evidence that needs to be remedied.
View the full article at FightAging
