How to Evaluate an SBOM Generator for Embedded Firmware

Does it find code with no manifest?

Most firmware dependencies were copied into the tree by a person. There's no package.json, no lockfile, often no version file. Tools that work by reading manifests, which includes Syft and Trivy in their default modes, will return little or nothing for these projects. That isn't a bug in those tools; it's what they were designed for.

Look for identification of vendored libraries such as FreeRTOS, lwIP, Mbed TLS or FatFs from the source itself. Check how much the tool can discover beyond what a package manager declared.

Does it get the version right?

A component name without a version is useless for vulnerability matching. Vendored code makes versions hard: the version may live in a header macro, a changelog, or nowhere at all. Silicon vendors also ship modified forks, so "FreeRTOS 10.4.3" in an ST or NXP SDK may not be byte-identical to upstream.

Compare reported versions with your ground truth. Ask how the tool handles missing version information, uncertainty, and modified code, and check whether its evidence supports the version it reports.

Does it report what ships, or what's in the repo?

A vendor SDK can contain hundreds of components. Your build compiles perhaps a dozen, and the linker discards unused parts of those. An SBOM of the repository over-reports, which means your team triages vulnerabilities in code that never reaches a device. An SBOM of the binary alone under-reports, because compiled C keeps very little identifying information.

Ask whether the tool distinguishes between code present in the tree, code compiled, and code linked into the image. Check the reported components against your build to understand whether the SBOM describes the repository or the firmware you ship.

Does it handle static linking and toolchain libraries?

Firmware is statically linked. Once the linker merges everything into one image, component boundaries are gone, and binary scanners struggle. There's also code in your image that never appears in your source tree: the C runtime (newlib, or the IAR runtime library), libgcc, and startup code from the toolchain. These have versions, licenses, and occasionally vulnerabilities.

Check whether statically linked libraries and toolchain runtime components appear in the SBOM. Ask what build or linker information the tool needs to identify code that comes from outside your source tree.

Does it work with your actual build?

Ask what the tool needs from you. Some require a specific build system (Zephyr's west spdx and Yocto's create-spdx are good, and only work inside those ecosystems). Some require a rebuild with a wrapper around the compiler. Some need a cloud upload of source or binaries, which rules them out for defense and many medical teams.

Evaluate the tool with your actual IAR, Keil, Make or CMake build. Understand the integration effort, any required build changes, and whether it can run offline if your team needs that. Check source and binary upload requirements against your constraints early.

Can you see why it made each identification?

Every tool in this space uses heuristics, and heuristics make mistakes. What matters is whether you can check them. For each component, you should be able to see the evidence: which files, which compile line, which hash or string matched, and how confident the tool is. An auditor will ask you the same question.

Inspect the evidence for individual components, including uncertain identifications. Ask how confidence is communicated and whether your team can review and correct the results.

Do the identifiers match vulnerability databases?

An SBOM is only as useful as its identifiers. Embedded components are poorly covered: many have no CPE in the NVD, PURLs are inconsistent for code that isn't in a package registry, and vendor forks don't map cleanly to upstream entries. Run the SBOM through a vulnerability scanner and look at both directions: known CVEs that were missed, and false matches.

Review the CPE and PURL mappings against known components and vulnerabilities. Ask how the tool handles components without an established mapping, and whether corrections persist across subsequent runs.

Does it capture enough for the SBOM to be made compliant?

Regulators expect each component to carry specific data. The 2021 NTIA minimum elements asked for supplier, name, version, unique identifiers, dependency relationships, SBOM author, and timestamp. CISA's 2026 update adds component hashes, licenses, the generating tool's name, and the generation context, and FDA, EU CRA and BSI TR-03183 requirements go further. Don't expect a generator to fill all of that in at scan time. Full enrichment during code scanning is hard, and for embedded code it's often impossible: a vendored file carries no supplier record, and a library copied into your tree has no registry entry to look up. Supplier names, normalized licenses, and end-of-life data come from outside the code, and are better added afterwards by an SBOM management platform.

What the generator must do is capture everything that only the source and the build can provide, because no downstream tool can recover it later: hashes, the license and copyright text actually found in the source, accurate names and versions, PURL and CPE identifiers where they can be determined, dependency relationships, and the tool name and generation context. Test both halves. Check what the generator captured, then load the SBOM into the management platform you plan to use, run its enrichment, and score the result with a tool such as sbomqs.

Is the output reproducible and standards-valid?

Build the same commit twice and diff the SBOMs. They should be identical apart from timestamps. Then validate the output against the CycloneDX or SPDX schema, and score it with an open source quality tool such as sbomqs. If you're shipping under FDA 524B, the EU CRA or BSI TR-03183, check the required fields now, not the week before submission.

Check for consistent component data across runs, valid output, and the fields your team requires. Confirm that the SBOM can be consumed by the other tools in your workflow.

How is pricing structured?

Have the pricing conversation upfront. Commercial generators may charge by SBOM, repository, or scan run. Ask what counts as a billable unit and estimate the cost using your expected number of projects, build variants, and CI runs. A price that looks reasonable during a proof of concept may look different when the tool runs on every build.

Open-source generators are generally free to use, though setup and maintenance still take engineering time. If an open-source tool meets your requirements and works well in your environment, use it.

What is the evaluation timeline?

Set the timeline before you start, and hold every tool to it. A reasonable plan is one to three weeks in total: a first SBOM within a day or two of installation, a week to run the tests above against your ground truth, and a week to try a second project with a different toolchain or target, since a tool that was tuned for the first project by the vendor's engineers will often stumble on the second. Track who did the work. An evaluation that only succeeds with the vendor driving is a services engagement, not a tool.