Why Every Patch Runs in a Sandbox Before Reaching Your Code

The Trust Problem

When an AI generates code changes and pushes them to your repository, trust is the fundamental barrier. Developers need to know:

1. Does this patch actually fix the error?

2. Does it introduce new errors?

3. Does it change behavior in unexpected ways?

4. Is it safe — no malicious code, no credential exposure, no destructive operations?

WarpFix answers all four questions before any code reaches your repository, using a multi-layered validation pipeline.

Layer 1: Static Safety Checks

Before any patch runs in a sandbox, it passes through static safety rules:

Forbidden patterns: The patch cannot contain destructive operations (rm -rf, DROP TABLE), credential access (process.env.SECRET), or file system operations outside the project directory.

Forbidden file changes: Lock files (package-lock.json, yarn.lock), environment files (.env), CI configuration, and security-sensitive files cannot be modified by automated patches.

Size limits: Patches are capped at 200 changed lines. Larger changes indicate the LLM may be rewriting files rather than making targeted fixes.

Scope validation: The patch must only modify files in the src/ directory (or equivalent source directories). Test files, configuration files, and documentation cannot be changed without explicit user approval.

These checks are fast (under 10ms) and catch the most obvious bad patches before they consume sandbox resources.

Layer 2: Sandbox Execution

Patches that pass static checks are applied in an isolated Docker container:

Environment Setup

The sandbox mirrors your repository's CI environment as closely as possible:

- Same Node.js/Python/Go/Rust version (detected from .nvmrc, pyproject.toml, go.mod, or rust-toolchain.toml)

- Same package manager and lockfile

- Same environment variables (secrets are replaced with safe test values)

- Same OS base image

Execution Flow

1. Clone the repository at the exact commit where CI failed

2. Apply the generated patch

3. Install dependencies (using cached layers for speed)

4. Run the specific failing CI command

5. Capture exit code, stdout, and stderr

6. Compare with the original failure output

Success Criteria

A patch passes sandbox validation if:

- The previously failing command now exits with code 0

- No new test failures appear (test count is equal or higher)

- No new compiler/linter errors are introduced

- Execution time is within 2x of the baseline (to catch infinite loops or performance regressions)

Failure Handling

If the sandbox fails, WarpFix does not give up. It:

1. Analyzes the new error output

2. Attempts a second fix that addresses both the original and new errors

3. If the retry also fails, it opens a PR with the analysis (not the patch) and flags it for human review

Layer 3: Confidence Scoring

Even patches that pass the sandbox receive a confidence score that determines how they are presented:

• 95-100: Auto-merge eligible (if the organization has opted in)
• 85-94: PR opened with "recommended to merge" label
• 60-84: PR opened with "review required" label
• Below 60: Comment-only mode — analysis posted but no PR opened

The confidence score considers:

- Whether a fingerprint match existed (higher confidence for known patterns)

- Sandbox execution results

- Patch complexity (fewer changes = higher confidence)

- Historical acceptance rate for this error type in this repository

What We Have Caught

In production, our sandbox has prevented several categories of problematic patches:

Patches that fix the error but break other tests: The LLM might change a function signature to fix a type error, but this breaks callers in other files. The sandbox catches this because it runs the full test suite, not just the failing test.

Patches that mask errors instead of fixing them: For example, wrapping a failing assertion in a try-catch. The sandbox detects that the test passes but the underlying behavior is wrong (via assertion count checks).

Patches with unintended side effects: A dependency version bump that fixes one issue but introduces a breaking change in another module. The sandbox catches this through integration tests.

Performance

Sandbox validation adds 30-90 seconds to the repair pipeline, depending on repository size and test suite duration. We optimize this with:

• Cached Docker layers: Dependencies are pre-installed and cached, so only the patch application and test execution happen fresh
• Parallel validation: For repositories with independent test suites, we can run validations concurrently
• Early exit: If a critical safety check fails, we skip the full sandbox run

The 30-90 second investment is worthwhile: it is the difference between "AI generated a patch" and "AI generated a verified fix." Users trust WarpFix because they know every patch has been tested before it reaches their PR queue.