Run on September 28, 2026
73 of 89
82.0% of tasks solved
- One attempt per task
- Standard network setup
- Graded by each task's own tests
Benchmarks
OxCode solved 73 of 89 tasks on Terminal-Bench 2.1 and resolved 410 of 500 instances on SWE-bench Verified, with one attempt for each. Each result comes with the conditions it was run under and a public record you can check.
Run on September 25, 2026
410 of 500
82.0% of instances resolved
- One attempt per instance
- Closed book
- Graded by the official SWE-bench harness
How this compares to the leaderboard
The official Terminal-Bench leaderboard averages several attempts for each task. Our run made one attempt for each task, so it is a different measurement and is not a leaderboard entry.
The chart sets our result next to the entries on the leaderboard, so you can see the difference for yourself.
- GPT-6 Astra (high), Codex87.4% ± 1.8
- Fable 5 (xhigh), Claude Code83.8% ± 2.3
- GPT-5.5 (xhigh), Codex83.2% ± 2.2
- OxCode, one attempt per task, run by us82.0% ± 8.0
- Fable 5 (high), Terminus 280.5% ± 2.3
- Grok 4.5 (high), Cursor CLI79.3% ± 2.9
- Opus 4.8 (high), Claude Code78.9% ± 2.6
- GPT-5.6 Terra (max), Codex78.4% ± 2.5
- GPT-5.5 (xhigh), Terminus 278.0% ± 2.4
- Muse Spark 1.1 (xhigh), mini-SWE-agent76.2% ± 2.4
- GPT-5.6 Luna (max), Codex75.7% ± 2.6
- Sonnet 5 (high), Claude Code74.6% ± 3.2
- Gemini 3 Pro (high), Terminus 273.9% ± 2.5
- Opus 4.7 (max), Claude Code68.9% ± 2.8
- Opus 4.7 (max), Terminus 266.1% ± 2.7
- Gemini 3 Pro (high), Gemini CLI65.8% ± 2.7
- Gemini 3.1 Pro (high), Gemini CLI65.8% ± 3.3
- Gemini 3.1 Pro (high), Terminus 265.6% ± 3.2
- GLM-5.1 (max), Claude Code58.6% ± 2.4
Show the same data as a table
| Entry | Score | Plus or minus |
|---|---|---|
| GPT-6 Astra (high), Codex | 87.4% | 1.8 |
| Fable 5 (xhigh), Claude Code | 83.8% | 2.3 |
| GPT-5.5 (xhigh), Codex | 83.2% | 2.2 |
| OxCode, one attempt per task, run by us | 82.0% | 8.0 |
| Fable 5 (high), Terminus 2 | 80.5% | 2.3 |
| Grok 4.5 (high), Cursor CLI | 79.3% | 2.9 |
| Opus 4.8 (high), Claude Code | 78.9% | 2.6 |
| GPT-5.6 Terra (max), Codex | 78.4% | 2.5 |
| GPT-5.5 (xhigh), Terminus 2 | 78.0% | 2.4 |
| Muse Spark 1.1 (xhigh), mini-SWE-agent | 76.2% | 2.4 |
| GPT-5.6 Luna (max), Codex | 75.7% | 2.6 |
| Sonnet 5 (high), Claude Code | 74.6% | 3.2 |
| Gemini 3 Pro (high), Terminus 2 | 73.9% | 2.5 |
| Opus 4.7 (max), Claude Code | 68.9% | 2.8 |
| Opus 4.7 (max), Terminus 2 | 66.1% | 2.7 |
| Gemini 3 Pro (high), Gemini CLI | 65.8% | 2.7 |
| Gemini 3.1 Pro (high), Gemini CLI | 65.8% | 3.3 |
| Gemini 3.1 Pro (high), Terminus 2 | 65.6% | 3.2 |
| GLM-5.1 (max), Claude Code | 58.6% | 2.4 |
Terminal-Bench 2.1
73 of 89 tasks solved (82.0%). Run on September 28, 2026.
Conditions
- Attempts
- One per task. The public leaderboard averages several attempts per task, so this is not the same measurement as a leaderboard entry.
- Network
- The benchmark's standard setup. Harbor, the official runner for Terminal-Bench 2.1, gives each task's container internet by default, because many tasks have to install the software they need. We ran with that default.
- Graded by
- Each task's own hidden tests, run by the benchmark's verifier. A task counts as solved only on a reward of 1.0.
How the run worked
- The agent
- The OxCode CLI in single-loop mode: one conversation, in which OxCode works until it decides the task is done and calls finish. That is the engine behind Agent mode from version 0.8.0. It was a development build from September 28, 2026, using one model for every call, through OxCode's own model gateway.
- Permissions
- Full access inside each task's container, so the agent was not stopped for approvals. OxCode's safety checks stayed on.
- The harness
- Harbor 0.22.0, the runner for Terminal-Bench 2.1, with OxCode installed as the agent. 8 tasks ran at a time.
- What it saw
- The task's own instructions, handed to the CLI in a file. Nothing from the task's tests.
- Time
- Terminal-Bench sets a time limit for each task. OxCode gave itself a deadline just under it, to leave room to wrap up. The whole run took 2 hours 19 minutes and made 3,281 model calls.
How the attempts ended
| Outcome | Tasks | What it means |
|---|---|---|
| Solved | 73 | The hidden tests passed. |
| Tests failed | 9 | The agent called finish, and the hidden tests failed. |
| Out of time | 6 | OxCode's deadline ended the attempt, and the work was not passing. |
| Other | 1 | Labelled request_timeout in the record. See the note on torch-pipeline-parallelism. |
| Total | 89 |
13 attempts reached the deadline. 7 of those had already got the work passing, so they count as solved.
Notes on specific tasks
- torch-pipeline-parallelism
- The record labels this attempt request_timeout, and the label is wrong. OxCode's own halt after five consecutive failed edits ended it, not a timeout. It is a loss either way.
- qemu-alpine-ssh and qemu-startup
- These two could not be graded. On the day of the run, the tests' own setup step (apt-get install -y curl) failed, because Debian no longer serves the package files its index listed. The tests never ran, for any agent we ran, so no agent could pass these two that day. They count as not solved and stay in the total.
What this result does not include
- Several attempts per task. This is one attempt each, and a run with several attempts would measure something different.
- A model name. The agent is OxCode as a whole.
- Agent transcripts.
SWE-bench Verified
410 of 500 instances resolved (82.0%). Run on September 25, 2026.
Conditions
- Attempts
- One per instance, no retries, no instance re-run.
- Excluded
- None. Every one of the 500 is in the total.
- Network
- Closed book: no internet access during the agent phase.
- Graded by
- The official SWE-bench evaluation harness, not by us.
How the run worked
- The agent
- The OxCode CLI, a development build from late September 2026, with OxCode's own model routing.
- Where it ran
- Inside each instance's own evaluation image, on the repository at the commit before the fix. Each instance had up to 40 minutes, and 12 ran at a time. The whole run took 5 hours 50 minutes.
- What it saw
- A fixed preamble and the issue text, and nothing else. The gold patch, the test patch and the list of tests that decide the score never reach the agent.
- How it was scored
- Once the run had finished, its patches went to the official harness. That harness applies each patch in a fresh container and runs the project's tests. A patch counts only if the harness says the issue is resolved. We do not score anything ourselves.
- Closed book
- Before the run, a probe ran inside the same image the agent used. It confirmed that the agent had no route to the internet, to package indexes or to git remotes, and that the image held no commits from after the fix. The only open route was to OxCode's own model gateway.
How the attempts ended
| Outcome | Instances | What it means |
|---|---|---|
| Resolved | 410 | The official harness says the issue is resolved. |
| Not resolved | 90 | This includes 3 empty patches, and 5 instances where the harness could not run the project's tests. |
| Total | 500 |
What this result does not include
- The benchmark's answers. The gold patches and test patches stay where SWE-bench publishes them.
- Agent transcripts. They contain the full problem statements and files from the instance repositories.
- A model name. The agent is OxCode as a whole.
Check it yourself
Every file below is in the public evaluations repository. The tasks and their tests are published by the benchmark owners, not by us.
Terminal-Bench 2.1
Every task listed as solved can be run again with your own agent, or checked against Terminal-Bench's own tests.
README.mdThe full write-up of this run.results.tsvOne row per task: the task name, whether it was solved, how the attempt ended, how many model calls the agent made, and how many seconds it worked.
SWE-bench Verified
These files are enough to grade the result again with the official harness.
README.mdThe full write-up of this run.predictions.jsonlThe patch OxCode produced for each of the 500 instances, in the harness's own input format.report.jsonThe official harness's report for those patches, unedited.closed-book-probe.txtThe network check run inside the benchmark image before the run.
To grade the patches yourself with the official harness:
pip install swebench
python -m swebench.harness.run_evaluation \
--dataset_name SWE-bench/SWE-bench_Verified \
--predictions_path predictions.jsonl \
--max_workers 8 \
--run_id oxcode-2026-09-25Your report should list the same 410 resolved instances as ours.
One field in predictions.jsonl was changed for publication. model_name_or_path reads oxcode instead of an internal configuration label. Every model_patch is byte-identical to what was graded. The SHA-256 of all 500 patches concatenated in file order is:
38c63d993cdd1d92b24a77f1a433ebeec32c4846117632da670bf9819e018bceWhat's next
82.0% is our first published Terminal-Bench 2.1 run, not our ceiling. We're improving OxCode's agent harness for more and better results, and every result we state here will be in the evaluations repository, with the files to check it.
If something here does not add up, open an issue there.