How to Detect Fabricated References in Large Language Model Outputs

Imagine writing a research paper, citing a study that sounds perfectly legitimate-author, journal, date, DOI-all there. You trust it. Your professor trusts it. The journal accepts it. Then you find out: it never existed. That’s not a glitch. It’s happening right now, in real academic papers, and it’s getting worse.

Large Language Models (LLMs) like GPT-4o, Claude, and Gemini are being used to draft literature reviews, summarize papers, and even generate reference lists. But here’s the problem: they make up citations. Not accidentally. Not rarely. Consistently. And often, they do it so well that even experts can’t tell the difference without checking.

What Are Fabricated References?

Fabricated references-also called "ghost references"-are citations that look real but point to non-existent papers, books, or studies. They have plausible author names, journal titles, volume numbers, and even DOIs. But when you search for them in databases like Crossref, PubMed, or Google Scholar, you get nothing. Not a 404 error. Not "not found." Just silence. Because the reference was never real to begin with.

These aren’t mistakes from a poorly trained model. They’re a direct result of how LLMs work. These models don’t store facts like a library. They store patterns. When you ask for a citation about "the effects of mindfulness on body dysmorphic disorder," the model doesn’t pull a real paper. It reconstructs one from fragments it’s seen before: "Smith, 2022, Journal of Clinical Psychology," but the author doesn’t exist. The journal doesn’t exist. The DOI is a random string that leads nowhere.

The Scale of the Problem

The numbers don’t lie. A November 2025 study in JMIR Mental Health analyzed 176 citations generated by GPT-4o across six simulated literature reviews. Of those, 35 (19.9%) were completely fake. Another 64 (45.4%) had serious errors-wrong DOIs, incorrect journal names, mismatched volumes. That’s 65% of all citations being unreliable.

And it’s worse in some areas. For topics like binge eating disorder, 28% of citations were fabricated. For body dysmorphic disorder, it was 29%. But for major depressive disorder? Only 6%. Why? Because there’s more published research on depression. The model has more real data to work with. When it’s unsure-when the topic is niche or emerging-it invents.

Earlier models were even worse. GPT-3.5 had a 55% fabrication rate. GPT-4 cut it to 18%. GPT-4o? Back up to nearly 20%. Progress stalled. The problem isn’t getting solved by better models-it’s getting more dangerous because people trust them more.

How Fake Citations Slip Into Published Papers

In January 2026, GPTZero scanned every paper accepted to NeurIPS 2025, one of the most selective AI conferences in the world. They found over 100 fabricated citations across 51 papers. These weren’t rejected drafts. These were accepted, peer-reviewed, and published. The reviewers didn’t catch them.

Why? Because checking citations is tedious. Reviewers focus on methodology, results, and novelty. They assume references are real. And why wouldn’t they? The citations look correct. They follow APA format. They have plausible titles. They’re embedded in coherent paragraphs.

But here’s the scary part: AI search engines are making it worse. If you search for a fake citation in Perplexity, Bing AI, or even Google’s AI Overviews, the system doesn’t say "no results." It invents a summary. "According to Smith (2022), mindfulness reduces symptoms in 78% of patients with body dysmorphic disorder." Then you cite it. Then someone else cites your paper. Then another AI trains on your paper. Now the fake citation is part of the training data. It becomes more convincing. It spreads.

One fabricated reference to "Prof. Williamson" was cited 43 times on Google Scholar-each time by a different paper, each time generated by AI. No such person exists. No such study exists. But now, it’s in the record.

Why This Isn’t Just a "Student Mistake"

You might think this only affects undergrads using ChatGPT to write essays. It doesn’t. It’s in peer-reviewed journals. It’s in clinical guidelines. It’s in policy briefs. A doctor might read a paper citing a fabricated study on a new therapy for anxiety. They might recommend it to a patient. The patient might stop their real medication. That’s not hypothetical. It’s a real risk.

And it’s not just about false information. It’s about broken trust. Science depends on cumulative knowledge. Every paper builds on the ones before it. If even 10% of citations are fake, the entire structure starts to crumble. You can’t verify a claim if the source you’re supposed to check doesn’t exist.

How to Detect Fabricated References

You can’t rely on intuition. You can’t rely on formatting. You have to check. Here’s how:

1. Use trusted databases-Search each citation in Crossref, OpenAlex, or PubMed. Don’t use Google Scholar alone. It indexes preprints, retracted papers, and fake entries.
2. Check the DOI-Go to https://doi.org/ and paste the DOI. If it doesn’t resolve, the citation is fake. Many fabricated DOIs are just random numbers.
3. Look for consistency-Does the journal name match the publisher? Is the volume and issue number plausible? Is the year before the journal was founded? These are red flags.
4. Use detection tools-CERCA is a free, open-source tool that checks references locally. It queries Crossref, OpenAlex, and Zenodo, and flags mismatches. It doesn’t upload your PDF. It runs on your machine. Privacy first.
5. Play "Dead Reference"-It’s a free game created by researchers to train people to spot fake citations. You’re shown 10 references. 5 are real. 5 are AI-generated. Try to tell them apart. It’s harder than you think.

Don’t assume a citation is real because it’s in a reputable journal. Don’t assume it’s real because it looks professional. Don’t assume it’s real because your AI tool says so.

What Institutions Need to Do

Universities and journals aren’t ready. Most have no policy on LLM-generated references. Here’s what needs to change:

• Require verification-Every paper using AI-generated references must include a verification log: a screenshot or list showing each citation was checked in Crossref or OpenAlex.
• Train researchers-Graduate programs need mandatory modules on AI citation risks. Not just "don’t use AI," but "here’s how to use it safely."
• Adopt detection tools-Journals should integrate CERCA-like tools into their submission systems. If a paper has 5+ suspicious references, flag it.
• Update citation parsers-Tools like Cermine need to handle all citation styles (APA, MLA, Vancouver, IEEE) and detect formatting anomalies that AI often messes up.

The problem isn’t AI. The problem is treating AI like a librarian. It’s not. It’s a storyteller. And it’s really good at making up stories that sound real.

It’s Not Going Away

Some researchers think better prompts will fix this. "Just ask for real sources only." No. The model doesn’t know what’s real. It only knows what patterns look plausible. Even if you say "only cite real papers," it will still generate fake ones. It’s not lying. It’s guessing.

Some think we’ll ban AI in academia. That won’t work. It’s too useful. Too fast. Too cheap. The answer isn’t to stop using AI. It’s to build systems that force human verification at every step.

The future of research depends on trust. If we stop verifying citations, we stop trusting science. And once that’s gone, it’s hard to get back.

What You Can Do Today

• Never copy-paste a citation from an AI without checking it.
• Use CERCA or manually verify every reference in Crossref.
• Teach others how to spot fake citations. Share "Dead Reference" with your lab or class.
• If you’re a reviewer, ask: "Did you verify these references?"
• If you’re an editor, demand verification logs.

AI won’t stop making up citations. But you can stop accepting them.