When will SWE-bench be solved? | Manifold

When will SWE-bench be solved?

Basic

5

Ṁ28

2030

7%

2024

32%

2025

26%

2026

7%

2027

7%

2028

7%

2029

15%

later

Resolves to the first year in which https://www.swebench.com posts a 90% resolved score in the “verified” category. The current score is 38.80% by Amazon Q Developer Agent.

This question is managed and resolved by Manifold.

#Technical AI Timelines

Get

1,000

and

3.00

Sort by:

bought Ṁ5 YES

Relatively significant chance of never (which resolves at later). Frontier LLMs are just going to get more and more contaminated with this benchmark's test data - at some point authors might just stop tracking

Related questions

What will be the best performance on SWE-bench Verified by December 31st 2025?

Codebuff solves at least 40% of issues on SWE-Bench by March 31, 2025

AI resolves at least X% on SWE-bench WITH assistance, by 2028?

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2027?

What will be the highest score achieved on SWE-Bench Verified in 2025?

AI resolves at least X% on SWE-bench without any assistance, by 2028?

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2026?

Related questions

What will be the best performance on SWE-bench Verified by December 31st 2025?

What will be the highest score achieved on SWE-Bench Verified in 2025?

Codebuff solves at least 40% of issues on SWE-Bench by March 31, 2025

AI resolves at least X% on SWE-bench without any assistance, by 2028?

AI resolves at least X% on SWE-bench WITH assistance, by 2028?

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2026?

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2027?

© Manifold Markets, Inc.•Terms + Mana-only Terms•Privacy•Rules