Static Analysis of LLM Code Output Before Execution

Static analysis works on code without executing it. The dominant technique for security is taint tracking: designate untrusted input entry points as sources, designate sensitive operations as sinks, trace data flows between them through the abstract syntax tree and control-flow graph. Tools like Fortify, FindBugs, and CodeQL operate on this foundation. They are deterministic and auditable, and they are also dependent on handcrafted rule sets that domain experts must continuously maintain.

That maintenance dependency is where the trouble starts. Source-sink specifications are rarely complete. An uncatalogued sink, an unusual API wrapper, a custom framework that obscures the data flow: any of these can let a real vulnerability pass unnoticed. More commonly, incomplete specifications produce the opposite problem: excessive false positives that erode analyst trust until scanner output becomes indistinguishable from noise. Teams paged on phantom vulnerabilities long enough learn to tune them out, and that learned dismissal compounds quietly until the scanner is, functionally, decorative.

The structural blind spots have been documented for over two decades, and they are not fixable deficiencies; they are intrinsic to reasoning about code without executing it. Race conditions depend on thread interleaving that has no analog in an abstract syntax tree. Logic flaws gated on specific dynamic inputs are invisible to a tool that never runs the code. Memory errors requiring a particular allocation sequence to trigger will surface in a static trace only under narrow conditions.

LLMs add a qualitative wrinkle on top of these existing quantitative gaps. Generated code tends to embed assumptions about calling context, library version, or input shape that pattern-only scanners are not equipped to reason about. An LLM may produce structurally correct code, code that clears every rule the scanner knows, while encoding a semantic flaw that only materializes when the surrounding system behaves in a particular way. The scanner sees clean code. The flaw is in the assumptions the code makes about the world around it.

It is also worth considering a benchmark mismatch that is rarely discussed openly. Existing vulnerability detection benchmarks skew heavily toward Python for code generation tasks and C/C++ for vulnerability detection, with divergent distributions of weakness types across languages. A scanner that looks authoritative in a benchmark paper may have quietly degraded recall on the languages actually arriving in your CI pipeline. The numbers in the paper and the numbers in your pipeline are not the same numbers, and the difference is rarely documented clearly enough to inform tool selection.

Static analysis is a necessary gate, not a sufficient one. Both things are true simultaneously, and conflating them in either direction produces a pipeline that fails in predictable ways.

How LLM-Augmented Static Analysis Closes the Gaps That Rule-Based Scanners Leave Open

The most consequential development in applied static analysis since roughly 2023 is the neuro-symbolic hybrid: a deterministic program analysis engine paired with an LLM reasoner, where each component compensates for the other's characteristic failure mode. LLM reasoning alone hallucinates; it will confidently flag a vulnerability that does not exist because it pattern-matched on surface features. Static analysis alone misses context; it cannot infer whether an unlisted API is a sink without being told. Pairing them is not a theoretical gesture, and the published results are specific enough to be useful.

AdaTaint, presented at ACM CISAI 2025, addresses the root cause of false positives directly. Rather than relying on hand-maintained source-sink lists, it adaptively infers specifications from the code under analysis. Evaluated against CodeQL, Joern, and LLM-only pipelines, AdaTaint reduced false positives by 43.7% on average while simultaneously improving recall by 11.2%. That combination is rarer than it sounds. Most approaches that reduce false positives do so by becoming more conservative, which means they also miss more real vulnerabilities. AdaTaint avoids that tradeoff because the LLM's suggestions are grounded in program facts rather than surface pattern matching.

IRIS, from MIT, offers the clearest single comparison between pure static analysis and a hybrid approach. CodeQL alone detected 27 vulnerabilities in the benchmark set. IRIS with GPT-4 detected 55, more than double, while improving on CodeQL's false discovery rate by 5 percentage points. IRIS also surfaced 4 previously unknown vulnerabilities that no existing tool had flagged. These are not marginal refinements.

LLMxCPG, published at USENIX Security 2025, approaches the problem from a cost-control angle that matters in production. Using code property graph-guided slicing, the system reduces the code passed to the LLM reasoner by 67 to 90% while preserving vulnerability-relevant context, achieving F1-score improvements of 15 to 40 percentage points over prior baselines at a fraction of the token cost. For pipelines running continuous analysis on high-volume code generation, the cost profile changes the economic conversation.

QLPro, published in mid-2025, introduces a triple-voting mechanism to reduce single-model confidence errors. Evaluated on 62 confirmed vulnerabilities across 10 open-source Java projects, CodeQL found 24. QLPro found substantially more through LLM-assisted taint classification, and the voting mechanism specifically addresses the case where one model is confidently wrong; requiring agreement across three reduces that failure mode considerably.

The question of whether to choose between SAST and LLM analysis is itself a false one. The integration problem is how to anchor the LLM's semantic reasoning in program facts that prevent hallucination, while extending the static engine's rule coverage with the LLM's ability to reason about unlisted sinks and sources. These four systems each take a different path toward that integration, and none of them has fully solved it.

The Patch Problem: Why Static Analysis Also Needs to Run After LLM-Generated Fixes

Most teams treat static analysis as a front-door filter: the LLM generates code, the scanner evaluates it, flagged code routes back to the LLM for a fix, the fix enters the pipeline. That model assumes a fix is a resolution. The data complicates that assumption considerably.

Research on LLM-generated patches finds that they introduce new security vulnerabilities in approximately 9.5% of cases while successfully resolving the original issue. One in ten patches that "work" in the narrow sense of eliminating the flagged finding also introduce something the scanner had no prior cause to examine. CVE-2025-34291, an authentication bypass in the Langflow platform that permitted arbitrary Python code execution, illustrates the category concretely: a change intended to improve security created a new critical exposure. That raises an important question: if the patch is itself a source of risk, what does it mean for a pipeline to treat patching as a terminal step? A pipeline that does so is producing a guarantee it cannot actually keep.

The Feedback-Driven Security Patching framework, FDSP, published in late 2025, proposes a structural answer. Rather than treating the scanner as a one-time gate, FDSP feeds scanner findings back to the LLM as prompt context for iterative refinement. Evaluated with Bandit, the approach reduced vulnerabilities by up to 33%; evaluated with CodeQL, a stricter scanner with higher recall requirements, the reduction was 12%, smaller but still meaningful. The key architectural insight is that the scanner functions as a feedback signal driving patch quality, rather than a pass-fail checkpoint.

The implication is operationally inconvenient but structurally clear: every AI-generated change, not just the initial generation but every subsequent patch, must re-enter the static analysis gate before proceeding. The pipeline is a loop. It closes only when the code passes clean, rather than when the original finding is resolved.

Which Commercial SAST Tools Handle LLM-Generated Code in 2025, and What Each One Covers

No single tool covers everything. The productive question is which combination fits the pipeline, and that depends on language distribution, compliance requirements, and where in the development cycle scanning occurs.

CodeQL is the de facto research baseline. Its taint analysis is rigorous, its query language is open and extensible, and its results are reproducible enough that multiple research papers use it as the comparison point. The recall gaps documented by IRIS and QLPro are real; treat CodeQL as a complement to hybrid approaches rather than a standalone answer.

Semgrep Code made its Gartner Magic Quadrant debut in 2025, with its multimodal analysis approach showing meaningful progress on business-logic vulnerabilities that sink-level pattern matching cannot reach, a category where most SAST tools remain structurally weak.

Checkmarx One has held a Gartner Magic Quadrant Leader position for seven consecutive years. Its broad language coverage and compliance posture matter most for regulated industries running AI-generated code at volume.

Snyk Code's most relevant feature for AI-generated code pipelines is real-time IDE scanning. Catching LLM output before it reaches the pipeline, at the moment of generation, is a different and complementary control from scanning in CI.

Veracode's legacy strength is binary and compiled-language analysis. When LLM-generated code is compiled before scanning, binary analysis coverage becomes relevant in ways that source-only scanners cannot address.

SonarQube offers a self-hosted, open-source option that lowers the barrier for teams integrating scanning into CI without a commercial contract. It lacks the highest recall in the field, but its accessibility makes it a realistic starting point for organizations building toward more sophisticated layering.

One gap none of these tools fully closes is business logic vulnerabilities. Understanding whether code does the wrong thing from the system's perspective requires intent-level knowledge of the whole codebase that sink-level pattern matching cannot supply. This is a structural limitation of the approach, not a deficiency unique to any particular product. The 27-versus-55 detection comparison between CodeQL and IRIS is not an argument against CodeQL; it is an argument against treating any single scanner as the complete answer.

What Static Analysis Cannot Catch, and How Execution-Time Controls Pick Up the Remainder

Static analysis operates on abstract representations of code. It cannot observe runtime state, timing relationships, or input values that only materialize during execution. Several vulnerability classes evade static inspection as a direct consequence.

Race conditions and concurrency bugs depend on thread interleaving that has no analog in an abstract syntax tree. Logic flaws gated on specific dynamic inputs are invisible to a tool that never runs the code. Memory errors requiring a particular allocation sequence will appear in a static trace only under narrow conditions. Business logic violations (those requiring real understanding of what the system is supposed to do) are out of reach for any scanner reasoning only about code structure.

A clean static analysis report means the code has passed a necessary filter. It does not mean the code is safe to execute in a shared environment. But what if that distinction collapses under schedule pressure, or when teams inherit pipelines they did not build and do not fully understand? It does, regularly, and the consequences are rarely visible until something breaks.

The correct framing is a division of labor. Static analysis is the pre-execution filter. A sandboxed execution environment is the runtime control that contains the blast radius when something the scanner missed is triggered. The two layers address different threat surfaces; neither makes the other redundant, and neither is sufficient without the other.

Why the Execution Environment for LLM-Generated Code Needs to Be Purpose-Built, Not Repurposed

The failure pattern that appears most often in production goes like this: a team runs static analysis, receives a passing result, and executes in a generic container, at which point they discover that generic containers were designed for trusted application workloads with known behavior. That is a categorically different threat model from what they are actually running.

Generic containers lack per-agent isolation strong enough to prevent cross-sandbox escapes when the code inside is actively untrusted. They were not built for stateful execution that survives crashes and allows long-running agent work to resume. Their provisioning speed was not optimized for pipelines spinning up a new sandbox per code execution. Compliance posture, where it exists at all, is typically retrofitted rather than architectural.

The isolation architecture options carry real tradeoffs, and none of them is obviously correct for every workload. Firecracker microVMs give each sandbox its own Linux kernel, the strongest available security boundary, with boot times around 150 milliseconds. gVisor interposes a user-space kernel that intercepts syscalls before they reach the host kernel, achieving sub-second cold starts at the cost of software rather than hardware-enforced isolation. Hardened OCI containers with seccomp profiles and namespace isolation provision fastest but reduce the kernel attack surface less than microVMs do. The threat model determines the appropriate tradeoff; there is no universal answer.

Purpose-built environments designed explicitly for autonomous AI systems, rather than developer sandboxes or generic container workloads, can offer provisioning times in the 27 to 90 millisecond range to keep agents responsive when sandboxes are created per code execution. Stateful design means agents can continue long-running work across the generate-analyze-execute cycle without losing context. Customer-managed compute keeps data within the organization's own cloud boundary, Docker-native compatibility means teams bring existing images and tooling without rewriting their stack, and SOC 2, HIPAA, and GDPR compliance are built into the architecture rather than retrofitted. Open-source codebases are inspectable by the teams that depend on them.

The argument for purpose-built runtime infrastructure is not that generic containers are universally inadequate. It is that the threat model for untrusted, machine-written code at agent scale is materially different from what generic containers were designed against, and tooling designed for the right threat model performs better against it.

How to Assemble a Pre-Execution Pipeline That Actually Holds in Production

The pipeline is a loop, not a linear conveyor. Every AI-generated change, including every patch generated in response to a scanner finding, re-enters the gate before proceeding. Treating it as a conveyor is the most common structural mistake in pipelines I have reviewed, and it tends to be invisible until something breaks.

Stage 1: Static analysis at generation time. Run a rule-based scanner, CodeQL, Semgrep, or Bandit depending on language, immediately on raw LLM output. Establish severity tiers and enforce them: block on critical findings, queue for review on medium, log low-severity findings for trend analysis. That last step matters more than it might appear. Low-severity findings tracked over time reveal whether the LLM's generation patterns are drifting toward a particular vulnerability class. The trend is the signal; the individual finding is noise.

Stage 2: LLM-augmented taint analysis. Apply a hybrid tool to the same output for source-sink coverage the rule engine cannot infer. Use CPG-guided slicing to pass only vulnerability-relevant code slices to the LLM reasoner rather than full files; this controls token cost and latency at the scale where continuous analysis on high-volume generation becomes economically constrained.

Stage 3: The patch loop. For flagged code, route to an LLM for remediation with the specific scanner finding surfaced as prompt context, then re-run both Stage 1 and Stage 2 before proceeding. Skipping re-scan after patching, given the 9.5% new-vulnerability rate on LLM patches, is a risk the data does not support accepting. The FDSP feedback architecture makes this tractable: scanner output becomes the prompt signal rather than a separate process.

Stage 4: Isolated execution. Code that clears the gate executes inside a sandboxed environment built for untrusted code, not a shared container. Isolation contains the residual risk from runtime-only vulnerability classes that static analysis cannot see. Stateful sandboxes allow long-running agent work to continue across the generate-analyze-execute cycle without state loss.

Monitoring layer. Treat scanner alert trends as a signal about the LLM's generation patterns. A spike in a particular CWE class is a prompt-engineering or model-selection problem, not merely a triage queue. The scanner is producing data about the generator; using that data to inform the generation side is an obvious leverage point that most teams leave untouched.

The gap this pipeline does not close is business logic flaws requiring intent-level understanding of the whole system. That is the current frontier in both research and tooling, and knowing where the gap sits matters practically: it tells you where human review still earns its cost, and where to focus the next iteration of the control layer. The average cost of a data breach reached several million dollars in 2024, a notable increase over the prior year, and open-source software vulnerability reports rose sharply year-over-year in the same period. Those figures were already climbing before LLMs began generating production code at scale. The pipeline question is whether to build this layer before or after something breaks.