GPT-5.5 · three scaffolds compared at same model · single attempt, no execution feedback, memory layer on throughout · official harness scoring
Details ↓122/122 Polyglot tasks (Python + Go + JS) · 93.9% HumanEval+ · full JARVIS CODE stack, nothing dropped
Details ↓No public benchmark measures long-haul conversation survival. We're building one. RFC in progress · github.com/longhaul-bench
Details ↓JARVIS CODE carries a persistent memory layer (JLC) under the agent. The obvious worry: does carrying it slow the model down? This page is the answer — standard coding benchmarks run end-to-end through JARVIS CODE, scored by official harnesses, never by the agent. Setup for every run: chat = GPT-5.5 (flat-rate subscription route), encoder = gpt-5.4-mini, one used consumer laptop (i5-8500, 8 GB RAM).
The benchmark current-generation agents are measured on: 500 real GitHub issues from twelve Python codebases (django, sympy, astropy…). The agent receives the repository at the faulty commit plus the issue text, and must produce a working patch. Verdicts come from the official SWE-bench harness — containerized runs against hidden tests the agent never sees.
Same brain, three bodies. The only comparison that isolates the scaffold is holding the model constant — so this chart is GPT-5.5 only, run through three different harnesses:
| Scaffold running GPT-5.5 | % resolved | Conditions |
|---|---|---|
| mini-SWE-agent run by Vals AI · Jul 2026 | 82.6 | independent · bash-only minimal agent (the official SWE-bench reference agent) · n=500 |
| OpenHands OpenHands Index · Apr 2026 | 78.2 | most-used open-source agent · test-running steps + critic/retry infra · reasoning_effort=high · n=500 · public tarball, verified by us |
| JARVIS Code this work | 75.6 | single attemptno test execution during generationused laptopsubscription flat-ratememory layer oneffort: mediumself-measured |
Read the three rows as an ablation: same model, same 500 tasks, different scaffolds. mini-SWE-agent gives the model a bare bash loop — including the freedom to run repo tests. OpenHands adds a full agent SDK with test-running steps and critic/retry infrastructure (its public per-instance artifacts are downloadable; we verified the 391/500 ourselves). JARVIS Code ran blind one-shots — the laptop carries no per-repo test environments, so the agent never executed a single repo test during generation, with the full memory layer riding along the whole time. One more stated asymmetry: OpenHands ran at reasoning_effort=high (its own tarball metadata); our turns rode the route-default medium — a second recoverable lever alongside the execution loop. Landing 2.6 points under the most-used open-source agent while giving up execution feedback entirely is the claim this page exists to make: the spread is the price of the feedback loop — not of the memory layer. And it's a floor, not a ceiling: an execution loop is a roadmap item, forgetting isn't. (Excluded: an "88.7% Verified" figure for GPT-5.5 circulates on aggregator sites but traces to no primary source — OpenAI's own launch reports SWE-Bench Pro 58.6% and no Verified number at all, having deprecated Verified reporting in Feb 2026. Field context, different models: the current absolute Verified leader is Claude Mythos 5 at 95.5%, vendor-reported 5-trial average.)
Run integrity: 500/500 generated (33 h unattended on the laptop), 498 patches applied cleanly, 2 empty patches counted as failures — nothing excluded. Scoring: official swebench 4.1.0 harness, 498/498 evaluated, 0 errors, per-instance logs archived. Failed-instance IDs are public in the evidence pack.
These older benchmarks are in every frontier model's training data, so they can't rank modern models — the board froze in late 2024. What they can prove is scaffold-neutrality: run through the full JARVIS CODE stack, memory layer and all, the model loses nothing.
HumanEval+ next to the strongest entries ever published on the official EvalPlus leaderboard:
| Model | base | + extra tests | Source |
|---|---|---|---|
| JARVIS Code + GPT-5.5 this work | 98.2 | 93.9 | this run · agentic |
| o1-preview Sept 2024 | 96.3 | 89.0 | EvalPlus leaderboard |
| GPT-4o Aug 2024 | 92.7 | 87.2 | EvalPlus leaderboard |
| Qwen2.5-Coder-32B | 92.1 | 87.2 | EvalPlus leaderboard |
| Claude 3.5 Sonnet June 2024 | 87.2 | 81.7 | EvalPlus leaderboard |
Polyglot verdicts come from a runner-owned clean test pass (pytest / go test / npm test) after the agent exits, with test files hash-checked (SHA-256) against tampering. HumanEval+ scored by the official evalplus harness; 0 evalplus references in any solution (scanned); the 10 misses are public. Polyglot covers the Python/Go/JS slices (122 of 225 six-language tasks) — C++, Java, Rust not yet included; full-set comparisons don't apply.
Every benchmark above hands the agent a fresh world per task — and that's exactly what real work never does. Real work is one conversation that lasts weeks. There, agents don't crash; they quietly forget (context compaction drops details) or the cost curve explodes. No public benchmark measures survival across a long-haul conversation. So we're building one.
LongHaul-Bench — "LongHaul measures agents, not foundation models." Seed-generated synthetic worlds plant facts made of invented tokens (no world-knowledge shortcut), then probe them at growing distances. The script's event timeline is the answer key — deterministic scoring, no LLM judge. Every agent brings its own memory strategy: compaction, RAG, notes, databases — all legal. Only the answer key is off-limits. Cost is measured alongside accuracy: a per-turn token ledger makes the leaderboard two-axis — remember more, and show what it cost.
Constitution: this is not a JARVIS benchmark — it's agent-neutral by design, with an open harness anyone can run. Launch baselines: Claude Code, Codex CLI, and a bare-LLM full-replay control — published even where we lose. Public instance + held-out seeds (memorizing the public script scores zero on a fresh seed — planted tokens change, structure doesn't). Status: RFC in progress · github.com/longhaul-bench
Same rules for every benchmark on this page:
| Rule | What it means |
|---|---|
| Agent never scores itself | Verdicts come from official harnesses (swebench, evalplus) or runner-owned clean test runs after the agent exits. |
| Single attempt, single trial | One agentic try per task, one run per benchmark. Vendor flagship numbers are often multi-trial averages — stated where known. |
| No execution feedback on SWE-bench | The agent never ran repo tests during generation — every patch is a blind one-shot. Top harnesses iterate against tests; ours didn't. Stated on the chart. |
| Tamper checks | Benchmark test files are hash-verified; solutions scanned for benchmark-data access; SWE patches filtered against test-file edits. |
| Nothing excluded | Empty or failed generations count as failures. All failed-instance IDs are public. |
| Contamination stated | HumanEval (2021) and Exercism are in every model's training data — those scores prove scaffold-neutrality, not model intelligence. SWE-bench Verified is where the current-generation comparison lives. |
| Hardware honesty | Generation ran on one used consumer laptop (i5-8500, 8 GB RAM) over a flat-rate subscription — no fleet, no per-token API spend. |
The runners (bench/) ship in the repo, and every run's results, logs, and per-instance reports are archived. Ask your AI: