Can AI Write Scientific Review Articles?

Scientific literature reviews are a critical part of advancing fields of study: They provide a current state of the union through comprehensive analysis of existing research, and they identify gaps in knowledge where future studies might focus. Writing a well-done review article is a many-splendored thing, however.

Researchers often comb through reams of scholarly works. They must select studies that aren’t outdated, yet avoid recency bias. Then comes the intensive work of assessing studies’ quality, extracting relevant data from works that make the cut, analyzing data to glean insights, and writing a cogent narrative that sums up the past while looking to the future. Research synthesis is a field of study unto itself, and even excellent scientists may not write excellent literature reviews.

Enter artificial intelligence. As in so many industries, a crop of startups has emerged to leverage AI to speed, simplify, and revolutionize the scientific literature review process. Many of these startups position themselves as AI search engines centered on scholarly research—each with differentiating product features and target audiences.

Elicit invites searchers to “analyze research papers at superhuman speed” and highlights its use by expert researchers at institutions like Google, NASA, and The World Bank. Scite says it has built the largest citation database by continually monitoring 200 million scholarly sources, and it offers “smart citations” that categorize takeaways into supporting or contrasting evidence. Consensus features a homepage demo that seems aimed at helping laypeople gain a more robust understanding of a given question, explaining the product as “Google Scholar meets ChatGPT” and offering a consensus meter that sums up major takeaways. These are but a few of many.

But can AI replace high-quality, systematic scientific literature review?

Experts on research synthesis tend to agree these AI models are currently great-to-excellent at performing qualitative analyses—in other words, creating a narrative summary of scientific literature. Where they’re not so good is the more complex quantitative layer that makes a review truly systematic. This quantitative synthesis typically involves statistical methods such as meta-analysis, which analyzes numerical data across multiple studies to draw more robust conclusions.

“AI models can be almost 100 percent as good as humans at summarizing the key points and writing a fluid argument,” says Joshua Polanin, co-founder of the Methods of Synthesis and Integration Center (MOSAIC) at the American Institutes for Research. “But we’re not even 20 percent of the way there on quantitative synthesis,” he says. “Real meta-analysis follows a strict process in how you search for studies and quantify results. These numbers are the basis for evidence-based conclusions. AI is not close to being able to do that.”