A fair scanner test has known answers (ground truth), counts both missed issues and false alarms, gives every tool the same inputs and settings, and can be repeated by someone else. If any of those is missing, the result is a demonstration, not a measurement.
Ground truth comes first
Ground truth is the list of vulnerabilities you know are really present, and the places you know are safe. Without it you cannot say whether a finding is correct. You can only say a tool found something.
There are three common ways to get it.
- Use a purpose-built application where weaknesses were placed deliberately. The OWASP Benchmark project states its intent that all vulnerabilities deliberately included in and scored by the Benchmark are actually exploitable.
- Inject known bugs into a realistic codebase. NIST's SATE VI Classic Track did this: the team said it would only analyze warnings related to the injected bugs.
- Use your own code and have two engineers independently confirm each finding.
Each has trade-offs. Purpose-built apps are clean but synthetic. Injected bugs are closer to real code but costly to prepare. Your own code is the most relevant but the slowest to label.
Count both kinds of error
A tool can miss a real issue (a false negative) or flag a harmless one (a false positive). Report both, per vulnerability category, since a tool may be strong in one category and weak in another. The next post in this series covers precision and recall.
Same inputs, same rules
Fix the code version, the build, the language versions, and any configuration. If one tool gets a tuned profile and another runs on defaults, say so, or tune all of them with the same effort. Record tool versions and rule update dates.
Repeat the run
Run each tool at least three times and compare. Variation between runs is itself a finding. If a tool produces different results on identical inputs, you need to know before you rely on it in a pipeline.
Document the method
Write down the inputs, the labels, the settings, the dates, and who did the labeling. Someone else should be able to follow it. This is also the best defense against later arguments about whether the test was fair.
Try it yourself
Click through it at your own pace, or open it full screen.
Open a finding and look at what you would need to label it correct or incorrect. That is your ground truth work.
Where Safeguard fits
Safeguard offers SBOM and dependency scanning, reachability analysis and zero-day discovery. A free scan at gold.safeguard.sh/scan can be one entry in a test like this.
Quick check
Can I skip ground truth if I only have a small codebase?
No. A small codebase makes labeling faster. Even a handful of confirmed findings and confirmed non-issues is better than none.
Sources
Last reviewed: 19 October 2026.
