Part 1 of this series ended with a star. Cisco’s Antares-1B, running on my laptop, points a beam into a repository and says “look here”, and roughly one time in three it’s pointing at the right file. Cheap, private, fast.

It also ended with an uncomfortable truth: “look here” is not “this is real”.

That gap is not a rounding error. It is the entire distance between a demo and a security program. And it’s not a gap that a bigger model closes on its own. Ask the best LLM you can buy to “find the vulnerabilities in this code” and you’ll get a confident, well-formatted answer. What you won’t get is any way to know which of those claims were checked, whether you’ll get the same answer next Tuesday, or why a given finding exists six months from now when an auditor asks.

So this post is about the forge. I took the Foundry security harness specification, a design document Cisco published (not a tool), built a working implementation of it, and then did the thing that matters: I pointed it at real code. Not a toy app with planted bugs. Two large, popular AI/ML platforms, an OWASP training application, and three well-known Python libraries with published CVEs.

Here’s what happened, what it means for you, and how to do it on your own code. Everything below comes from stored runs in my Foundry repo.

The one-sentence version

Foundry doesn’t make the model smarter. It makes the model’s output something you can build a process on.

That sounds like a downgrade until you’ve tried to operate an LLM security tool in production. The model was never the bottleneck. Trust was.

Foundry is a governance layer around whatever model you already use. The pipeline is four roles, small enough to read in an afternoon:

• An Indexer breaks the code into function-level units.
• A Detector asks the model which security rules plausibly apply to each unit, plus a light exploratory pass for anything no rule describes.
• A Triager assigns each candidate a verdict with structured citations: file, line, and the exact quoted text.
• A Reporter gives every finding a stable identity, orders by severity, and renders the result.

Around all four sits an Orchestrator with a hard token budget, a declared stop condition, an append-only provenance log, and a store that deduplicates.

Two design decisions carry most of the weight.

Detection is inclusive; the decision is strict. The Detector is told to prefer recall: propose anything plausible. All the discipline lives downstream. That separation lets one model be thorough and trustworthy at the same time, instead of making you choose.

The evidence gate is not a second opinion. It never asks the model “are you sure?” It takes every citation and resolves it against the bytes on disk: the file must exist inside the target, the line must be in range, and the quoted text must actually appear there. A finding is only labeled “true-positive” if its citations survive. Anything else is demoted to “needs-review”, or rejected and shown to nobody.

A model cannot argue its way past a line number that doesn’t exist.

Real world, stop one: Langflow

The first target was Langflow 1.7.3, a popular low-code platform for building LLM agents and pipelines. I chose it because it’s realistic: a large, first-party Python codebase with a real API surface, real auth logic, and real history. Scope was the entire v1 API: 289 function-level units across 25 modules.

And I gave myself a hard rule for this experiment: no strawman baseline.

It would be easy to fake a win here. Point a crippled LLM prompt at the code, let the fancy system beat it, declare victory. That proves nothing. So the baseline got the same model, the same code, and a fair, disciplined senior-AppSec prompt demanding exactly the rigor the harness enforces: structured findings, exact line citations, verbatim quotes, no invented references, no duplicates. I wrote the methodology down before the run.

What the numbers say

Three honest readings of that table.

Foundry surfaced more than twice the findings. Same model, same code. Not because it’s cleverer, but because a single agent turn curates a short headline list and drops the long tail, while the harness sweeps every one of the 289 units under a declared budget. A vulnerability in module 23 of 25 matters as much as one in module 1. Completeness is a property of the harness, not the model.

Citation accuracy tied. A capable model asked in good faith for rigor largely delivered it. But look at why the tie is less comforting than it seems: the baseline’s clean citations are a property of this model, on this scope, with this prompt. Swap in a weaker model or a sloppier prompt and it breaks silently. Foundry’s 100% is a guarantee by construction, because the same gate runs every time.

The confidence tier is the real difference. The baseline called 14 of its 15 findings “true-positive”. Nothing enforced that. It’s a hope that the model volunteers its own doubt. Foundry’s line between confirmed and needs-review was drawn by a mechanical check, not by the model’s mood.

What one governed finding looks like

The most serious thing the run found was a critical code-injection bug. Langflow takes a flow input’s display name, which a user controls, and pastes it into a string of Python source that it then executes with exec(). The only sanitization is lowercasing and swapping spaces for underscores, which does nothing about parentheses, colons, or newlines.

A plain agent can find that. The interesting part is what surrounds it:

• Three cited lines showing user input becoming source code, source code becoming a function, and the function being executed. The gate read all three in the real file before the finding was allowed to be called confirmed.
• A permanent identity built from file, weakness class, and symbol, deliberately not the line number, because line numbers move.
• A pointer into the provenance log, back to the exact detector call that proposed it and the exact triage call that confirmed it, with token usage attached. Somebody who wasn’t in the room can reconstruct why this finding exists, six months later.

The same run also confirmed two critical path traversals in the knowledge-base API, where user input can perform arbitrary directory deletion.

Run it twice

Here’s the failure mode that quietly kills LLM security tooling. You run it Monday, triage the output, run it Friday on the next commit, and get a differently worded wall of text you now have to diff by hand. After the second time, the team stops running it.

So I ran the full pipeline twice, with independent model calls. Across both runs the store held 122 finding-records that collapsed to 67 distinct fingerprints. 56 recurred and landed on their existing identity instead of duplicating: roughly 90% overlap despite model non-determinism. Findings carry the same fingerprint in both runs.

That second run stands in for your next CI run. What you want from it is the delta (new, recurring, gone), computed rather than eyeballed. One agent turn has no identity between runs, so there’s nothing to compute.

Bigger scope, bigger gap

I also ran a narrower 12-module, 107-unit slice of Langflow. At that size, the single-shot baseline kept up on breadth. The raw counts looked like a tie, and only the governance properties separated the two. At the full 25-module scope, the baseline simply couldn’t hold everything in one turn and curated about 15 headline items.

That’s the most useful thing I learned from Langflow: the harness’s value grows with scope. Which is exactly why the next stop was bigger.

Real world, stop two: MLflow

A fair skeptic would ask whether I’d quietly tailored the harness to Langflow. So I pointed it at MLflow 2.9.0, a different and much larger codebase, and changed only the config file: target path, scope globs, output directory. The engine and the 13-rule corpus stayed the same.

Two things stand out.

The findings landed where history says they should. 34 path-traversal findings concentrated in MLflow’s artifact storage layer (the local, FTP, SFTP, HDFS, and Databricks artifact repositories) and its file and URI utilities. That’s the same neighborhood as MLflow’s real path-traversal CVEs, CVE-2023-6909 and CVE-2023-6977. The rule that found path traversal in Langflow’s knowledge-base API found it in MLflow’s artifact repositories, unmodified.

Most of what surfaced sat in “needs-review”, and that’s the tier doing its job. On a bigger, unfamiliar codebase, the gate was far more conservative about what it would let the model call confirmed. A plain agent would have handed you 123 items with the same confident label. Foundry handed you 13 it could prove line by line and 110 leads, and told you which were which. For a team deciding where to spend reviewer hours, that distinction is the whole point.

And the governance held at 3x scale: evidence gate, provenance, bounded stop, all behaving exactly as on Langflow. A single agent turn cannot hold 830 units in one context window. The harness doesn’t need to.

Real world, stop three: an OWASP training app

PyGoat is OWASP’s intentionally vulnerable Django application. It’s small, around 1 MB, and it’s honest application code: it calls pickle.loads(), eval(), subprocess.Popen(shell=True), and raw SQL on user input. That’s exactly the pattern-level weakness a security rule corpus is built for.

A full sweep of its 180 units in about six minutes surfaced 45 findings: 42 confirmed, 3 needs-review, 14 rated critical, across 11 weakness classes. Hardcoded secrets (a JWT signed with a literal string, plaintext credentials), shell injection, eval() on POST input, MD5 password hashing, raw SQL built by string concatenation, pickle.loads() on a cookie, SSRF. Every confirmed finding passed the evidence gate.

How Cisco customers and partners can use this

Everything I’ve shown is in a public repo. Here’s how to put it to work.

If you own code, run it on your repo

The repo’s build-your-own guide walks you through it, but the shape is simple, and every step is a config change, not a code change:

1. Pin your target. Add your repository and a fixed tag or commit to the fetch script, so every run is reproducible and every finding refers to a known tree.
2. Scope your attack surface first. Set the include globs to the code that handles external input: API routes, handlers, views, file and URL utilities. Exclude tests and migrations. Write down why you chose that scope. You’ll widen it later.
3. State your goals in plain language. “Focus on tenant isolation and authorization, injection is secondary” or “We handle PII, prioritize data exposure and secret logging.” That text is injected into both the Detector and Triager prompts, so it genuinely steers the evaluation.
4. Set your budget. Pick a model and a hard token cap. The governor stops cleanly at the cap. You’ll never get a surprise bill.
5. Bring your own rules. The 13 shipped rules are Python and framework-generic YAML. Add your house standards the same way and the Triager checks them with citations instead of vibes. The engine is language-agnostic; only the function extractor is Python-specific.
6. Run, score, repeat. Run the harness, optionally run the fair baseline and score the two against each other, then run again on the next commit and triage only the delta.

My Langflow numbers will give you a rough cost model: a full 289-unit sweep was about 2M tokens, and PyGoat’s 180 units were about 420K.

If you’re building a security program: Build your own from the spec

My repo is a reference implementation: a teacher, not a product. The durable asset is the Foundry specification itself: about 130 requirements and three dozen deliberate [NEEDS CLARIFICATION] decisions that it leaves to you, such as your datastore, your role decomposition, your stop rule, and your severity scheme.

Run the spec through a tool like spec-kit, answer those decisions for your context, and use my repo’s customization and architecture docs as one worked answer to each of them. The trust properties (gate, fingerprints, provenance, bounded stop) are what you carry over.

If you’re a partner: Sell evidence, not opinions

Most “AI-powered code review” engagements end in a PDF of model output that the customer has to take on faith.

A Foundry-style engagement ends with something very different: every confirmed finding tied to quoted source lines that were machine-checked, an append-only log of every model call behind every verdict, a declared scope and budget agreed up front, and stable finding IDs, so next quarter’s re-assessment shows the customer a measurable delta instead of a fresh wall of text. That’s a recurring, auditable service, and it runs on whatever model the customer is already approved to use.

What I am not claiming

Foundry is not a smarter bug-finder. Same model. It surfaces more because it enumerates exhaustively.

Foundry is not cheaper. About 10x the tokens of a single agent turn on Langflow. The spend buys coverage, the gate, and deduplication, under a hard cap. That’s the trade, stated as a trade.

No “zero false positives, zero misses”. Those are negatives you can’t prove without perfect ground truth. Here, “true-positive” means verified and worth a reviewer’s time, and “needs-review” means a lead.

Nothing here is marked “exploited”. Proving exploitability needs a live testbed, and the spec rightly makes that a separate role I didn’t build. The architecture doc names that omission and its cost.

And when is a plain agent the right answer? A one-off look by one engineer at a small, familiar piece of code. It’s far cheaper and genuinely useful. Just use it. Foundry earns its cost when findings have to feed a process: an issue tracker that mustn’t drown in duplicates, reviewers who need to know what was checked, runs that repeat as code changes.

Next: The file that keeps filling up

There’s one artifact I’ve barely mentioned, and it turned out to be the most valuable thing the harness produces.

Every time the Detector’s exploratory pass finds something that no rule in the corpus describes, and the Triager agrees it’s real, Foundry writes it to a file. On Langflow, that file caught an endpoint that restricts itself to “local connections only”, and decides what “local” means by trusting a header any attacker can set. On MLflow, it caught an unchecked index into a user-supplied path that ends up choosing which run a privileged API call acts on.

Those are not just findings. They’re lessons: patterns the system didn’t know to look for, learned from real code.

Meanwhile, AI coding assistants are writing new code faster than any of us can review it, and they don’t know those lessons either. In Part 3, I’ll show you how Foundry turns each lesson into a rule that lives inside the developer’s IDE, so the next time an AI assistant is about to write that bug, it doesn’t.

See you next week!

Everything here is reproducible from github.com/juliogomez/foundry: the stored runs for every target, the provenance logs, the rule-gaps, and the scored comparison.