How do you measure application security success without gaming it?

Measure application security by outcomes you can defend: how much of your software is covered by checks, how fast you fix the issues that are exploitable, how mature your practices are against a model, and how many serious problems escape to production. Raw finding counts mostly measure how hard you looked.

Why do finding counts mislead?

A new scanner can double your open findings overnight with no change in real risk. A team that stops scanning looks better than one that scans well. Counts also hide severity and context: a thousand unreachable low severity items matter less than one exposed critical flaw.

Metrics shape behavior. If teams are rated on the number of open issues, they will close them by marking them as accepted. So pick measures that are hard to improve without improving security.

Which measures hold up?

Coverage. What share of repositories, services and deployed artifacts run your checks, and how recently? Coverage is a denominator problem. It needs an inventory, which is itself a useful finding when it is wrong.

Time to remediate what matters. Measure from discovery to fix, split by priority class. "What matters" needs a rule. Two public signals help. CISA's Known Exploited Vulnerabilities Catalog lists flaws seen exploited in real attacks. The Exploit Prediction Scoring System from FIRST estimates the probability that a published CVE will be exploited in the wild in the next 30 days, published daily for every CVE. Using both gives a reasoned top tier, and your own reachability and exposure data narrows it further.

Escapes. Count serious issues found after release, by a customer, a researcher, a penetration test or an incident, and ask which earlier check should have caught each. This is the nearest thing to a true error rate.

Maturity. OWASP SAMM, the Software Assurance Maturity Model, offers a way to assess where you are and set targets. It has 15 security practices in five business functions, each with three maturity levels, and it says explicitly that not every organization needs the maximum level in every practice. Use it to pick targets and track movement, not to chase a score.

Practice adoption. NIST's Secure Software Development Framework (SP 800-218) lists practices such as testing executable code and recording and triaging each discovered issue in the team's workflow. A simple adoption measure is the share of teams that can show those records.

What should you avoid?

Avoid single composite scores that blend unlike things. Avoid targets set on the metric alone, such as "reduce open findings by 30 percent," without a rule for how. Avoid comparing teams on raw counts when their codebases differ in size and age. And avoid reporting a trend without stating when tools or inventories changed, since that breaks the line.

A constructed example

A company with twelve engineering teams reports four numbers each quarter. Coverage: 83 percent of production services run dependency and code scanning on every merge, up from 60. Median days from discovery to fix for issues that are both reachable and on the KEV list or high on EPSS: six, down from nineteen. Escapes: two serious issues found after release, both traced to a service that was not yet in the inventory. SAMM target: move one practice, secure build, from level one to level two by year end. These numbers are small and uncomfortable, which is the point. Each one has an owner and a next action.

How to check or apply this

1. Build the inventory first: repositories, services, deployed artifacts and owners.
2. Define the "must fix fast" class with a written rule, using exploitation signals and exposure.
3. Track discovery to fix time for that class, and report medians and the oldest items.
4. Log every escape with the check that missed it, and fix the check.
5. Pick a maturity model, assess once, and set a target for two or three practices.
6. Note any tool or inventory change in the report so trends stay honest.
7. Review the measures yearly and retire any that people have learned to game.

Safeguard's reachability analysis and policy gates can supply the "is it exploitable here" and "was a rule enforced" data points for items two and four.

Quick check

Sources

• OWASP SAMM, About
• NIST SP 800-218, Secure Software Development Framework
• FIRST, Exploit Prediction Scoring System
• CISA Known Exploited Vulnerabilities Catalog

Last reviewed: 18 October 2026.