Technology could be used to identify published scientific papers that have the greatest potential for real world impact – such as new medical treatments – according to UCL and UCLH research.
It means scientific work with the best chance of being translated into improvements in patient care could be recognised ahead of time.
The impact of scientific findings can be judged by whether they are later included in patents, guidelines or policy documents. But at present it is difficult to predict in advance which discoveries will go on to have such impact.
Objective predictions are currently based on simple bibliographic metrics such as the number of times other researchers cite a published article. But the predictive power of this approach has not been previously quantified.
The team at UCL and UCLH led by Professor Parashkev Nachev (UCL Queen Square Institute of Neurology) compared this traditional approach to predicting future impact with use of deep learning models that take account of detailed textual information found within article titles and abstracts, alongside bibliographic data. The team analysed 43.4 million biomedical journal articles from 1990 to 2019.
Findings published in the journal Patterns showed that the best of these deep learning models significantly outperformed traditional, citation-based models at predicting the future inclusion of a scientific article in a patent, guideline or policy document. These models could also predict which scientific articles would later be linked to the awarding of a Nobel prize without being specifically trained on the task.
Dr Amy Nelson, the first author of the study, said: “The value of medical research ultimately lies in the patient benefit it brings. Simple metrics of scientific productivity, such as citations, cannot be expected to distinguish publications that deliver real-world impact from those that only promise it. If we choose to rely on objective metrics to evaluate medical research and guide its future direction, then AI-based models of research content, not just dissemination, are likely to be more effective.”
