Detecting AI Hallucinations in Generated Code
A USENIX Security 2025 study found 19.7% of packages recommended by 16 LLMs across 576,000 code samples don't exist — and attackers are registering them first.
Deep dives, practical guides, and incident analyses from engineers who build Safeguard. No fluff, no vendor FUD — just what you need to ship secure software.
A USENIX Security 2025 study found 19.7% of packages recommended by 16 LLMs across 576,000 code samples don't exist — and attackers are registering them first.
Copilot-generated code carried vulnerabilities in ~40% of cases in a 2021 NYU study. Static scanning alone cannot catch the runtime-only bug classes LLMs introduce.
Nearly 40% of GitHub Copilot's suggested programs contain exploitable vulnerabilities, and 19.7% of AI-generated code samples reference packages that don't exist.
NYU found 40% of Copilot-generated code contained exploitable flaws; Samsung banned ChatGPT after three leaks in under 20 days. The productivity math still isn't simple.
OWASP's 2025 LLM Top 10 and MITRE ATLAS both treat adversarial testing and posture scanning as separate disciplines — most AI programs still run only one.
One 2025 benchmark found an LLM filter cut Semgrep's false positives by 88.6% — while a separate study found GPT-4 alone flagging vulnerabilities was wrong more often than right.
A 2021 NYU study found roughly 40% of Copilot completions on security-relevant prompts contained exploitable flaws. Here's a field guide to catching them.
Developers using AI coding assistants wrote less secure code in 4 of 5 tasks in a 2023 Stanford study — and were more confident it was safe.
GitHub has disclosed that Copilot suggestions match training-set code verbatim about 1% of the time — and a class action over it is still being argued in 2026.
Weekly insights on software supply chain security, delivered to your inbox.
Your first fix PR is minutes away.
No sales call required, even your agent can complete the purchase over MCP.