How to evaluate a license compliance tool in a 2 week trial

To evaluate a license compliance tool, run it on three of your own repositories, compare its output to a hand-checked sample, and test the policy and exception workflow end to end. Judge it on accuracy against your code, not on the demo data the vendor chose.

A two week trial is enough if you plan it. Here is a day by day shape and a scorecard.

What are you actually testing?

A license tool answers four questions: what components do I use, direct and transitive; what license does each one carry; does that license break my policy; and what do I do about exceptions. Each question needs its own test.

The SPDX License List helps here. It is a catalog of commonly used licenses and exceptions, and each entry has a short identifier such as MIT or Apache-2.0 and a permanent URL. A tool that reports SPDX identifiers gives your legal and engineering teams a shared vocabulary. Ask whether it can also express compound cases like a license WITH an exception, since SPDX uses that syntax.

Week one: accuracy

Pick the sample. Choose three repositories: one clean and small, one large and old, and one with a mix of languages. Include one where you already know a tricky license.

Hand check twenty components. Pick twenty at random, including at least five transitive ones. Look up each license from the package source. This is your answer key.

Run the tool and compare. Score each component as correct, wrong, missing or reported as unknown. Count how many unknowns the tool left and whether it says why.

Test transitive depth. A tool that only reads your manifest misses what your dependencies pull in. Check that nested dependencies appear.

Test the lockfile. Compare what the tool reports with what your build actually installs.

Week two: policy and workflow

Write your policy as rules. For example: allow a list of permissive licenses, flag licenses that need legal review, block a few that your business cannot accept. See whether the tool can express that without engineering hacks.

Trigger a violation. Add a component with a flagged license on a branch. Watch where the tool tells you: in the pull request, a dashboard, an email? Check the message is clear.

Handle an exception. Approve one exception with a reason and an expiry date. Confirm it appears in the record and lapses.

Export evidence. Ask for a report in SPDX or another standard form. Hand it to someone in legal and ask whether they can use it.

Check speed and noise. Time scans and count alerts that nobody would act on.

Scorecard

Rate each from zero to three:

1. Accuracy on the answer key.
2. Transitive coverage.
3. Handling of unknowns.
4. Policy expressiveness.
5. Exception workflow.
6. Where alerts show up for developers.
7. Export quality.
8. Where the scan runs and what data leaves your network.

Add up the scores, but read the notes. A tool that fails item one should not win on item seven.

What does Safeguard add?

Safeguard produces component inventories with license information and policy checks in the same place as vulnerability findings, and the CLI can run on your machine with the local runner. Use the same scorecard on us.

What to do

1. Choose three repositories and build a twenty component answer key.
2. Run each candidate and score accuracy.
3. Write your policy and test a violation and an exception.
4. Export a report and show it to legal.
5. Pick on the scorecard and write down why.

Quick check

Sources

• SPDX License List

Last reviewed: 18 October 2026.