All articles tagged with #peer review
An AI-assisted analysis of public peer-review reports for 8,000 Nature Communications papers (2017–2024) finds that papers subjected to tougher criticism and larger revision costs tend to be more highly cited in the following three years. While the quality of reviewer comments correlates with impact, how constructive the feedback is does not, suggesting that rigorous review can both reflect ambitious work and help raise its future influence.
Retraction Watch interviews Aaron Brown about his book Wrong Number, a critique of how the media hypes influential studies that can be illogical or misleading; Brown argues for journalist skepticism and cautions against outsourcing scrutiny to statisticians, advocates for post-publication peer review and full transparency (data, code, holdout samples), and suggests independent testing institutions to vet ideas, with a nuanced view on when to disclose uncertain findings and how to treat government data to improve the reliability of published research.
ArXiv will ban authors for a year if there is incontrovertible evidence that LLM-generated content in a submission hasn’t been checked (e.g., hallucinated references or misleading meta-comments), with future submissions requiring acceptance at a reputable peer‑reviewed venue. This strengthens efforts to curb AI-generated “slop” and builds on prior policies restricting AI-heavy content to peer‑reviewed publications; bans are subject to moderator review and author appeals.
The BMJ group’s retraction of nearly an entire guest-edited special issue on cancer immunotherapies highlights broader worries about guest-edited publishing: rapid, high-volume special issues driven by APCs can weaken peer review, inflate researchers’ CVs, and undermine research integrity, prompting calls for tighter editorial oversight and reforms in author-pay models and funding practices.
Retraction Watch’s weekend digest notes that large language models aren’t the core issue in science publishing, reports that offering cash to spot errors doesn’t work, and spotlights vaping studies with numerous flaws and few retractions, while also outlining ongoing investigations and policy discussions around scientific integrity and publishing practices.
Nature published a peer‑reviewed update on Sakana AI’s AI Scientist, a system that uses LLMs to search literature, generate hypotheses, run experiments, and draft papers. The tool submitted three original AI‑generated papers to a leading ML conference, with one accepted, but the authors tempered claims of fully automating science and included an automated reviewer. They stress AI should assist human scientists, while flagging risks like originality dilution as autonomous research advances.
Researchers present The AI Scientist, an end-to-end autonomous system that ideates, codes, runs experiments, analyzes results, writes full manuscripts, and even conducts its own peer review to generate new ML papers. It operates in template-based and template-free modes, relies on foundation models and agentic tree search, and includes The Automated Reviewer to gauge paper quality. In tests, a fully AI-generated manuscript reached workshop peer review at ICLR (though not top-tier publication), and overall quality improves with more compute and better models, while the work also highlights ethical risks and the need for responsible norms as autonomous scientific systems mature.
The 2026 ICML used hidden watermarks in submitted papers to detect AI-generated peer reviews; 497 papers (about 2%) were rejected for violations of the conference's LLM-use policy, and organizers introduced two review streams—one permitting limited LLM use and one banning it—to protect trust in the peer-review process.
Tyler Cowen asks for ideas on how AI should transform academic journals; commenters propose AI-led triage and grading, disclosures of AI use, open-access reforms, machine-readable submissions with supporting artifacts, transparent AI pipelines, auditing, and new funding models, while skeptics caution that AI cannot fully replace human reviewers or authorship, highlighting both opportunities and risks in rethinking scholarly publishing.
A chemistry researcher had 35 papers retracted within 24 months for a mix of issues including major errors in analyses, compromised peer review, image-related problems, and citation manipulation. Most retractions appeared in Elsevier- and Royal Society of Chemistry–published journals, placing the scientist on Retraction Watch’s leaderboard. Coauthors have defended the work as a matter of presentation rather than fraudulent data.
A pediatric journal admitted that 138 published case reports were fictional, created under a confidentiality program to protect patients; at least 61 of these cases have been cited as fact in other journals, prompting corrections, scrutiny of potential retractions, and concern over how misinformation spread, including the controversial 'baby boy blue' opioid-in-breast milk case. Editors say future reports will clearly label cases as fictional, but copies and citations in databases like PubMed Central and Semantic Scholar have already propagated the error.
Science sleuths outline five practical checks to spot dubious papers: vet the references for relevance and possible fake or self-citing patterns; verify authors and affiliations (and ORCID IDs); examine figures and images for manipulation; evaluate the science itself for formulaic, boring, or implausible findings often produced by paper mills; and read the abstract for clarity and consistency, using community resources like PubPeer and Retraction Watch to corroborate concerns.
Nature Machine Intelligence reports a large-scale randomized study showing that automated, LLM-generated feedback via the Review Feedback Agent improves peer review quality and engagement. At ICLR 2025, over 20,000 reviews were analyzed; 27% of reviewers who received AI feedback updated their reviews, incorporating more than 12,000 suggested edits. Blind evaluations found revised reviews more informative, and the intervention increased writing length (about 80 extra words for updaters) with longer author and reviewer rebuttals. The study suggests carefully designed LLM feedback can make reviews more specific and actionable while boosting reviewer–author engagement; data and open-source code are available.
A five-LLM AI coach, called Review Feedback Agent, was developed to help peer reviewers deliver more specific, constructive, and less toxic feedback. When tested on thousands of existing reviews, it frequently suggested actionable ways to improve comments. It remains unclear whether this improves the quality or impact of the papers being reviewed, requiring further study.
AI is evolving faster than the systems that evaluate it, so much published research can be outdated or misleading by the time it appears; studies based on older models risk misinforming on health and therapy, highlighting power imbalances favoring AI companies and the need for faster, more independent peer review.