Will an AI SWE model score higher than 50% on SWE-bench in 2024?
Plus
16
Ṁ470Dec 31
20%
chance
1D
1W
1M
ALL
This question is managed and resolved by Manifold.
Get
1,000and
3.00
1,0003.00Sort by:
Traders (I can't tell which mention to use) -- how do you feel about changing this to be SWE-bench Verified explicitly?
https://www.swebench.com/ -- explanation of the differences here:
SWE-bench Lite is a subset of SWE-bench that's been curated to make evaluation less costly and more accessible.
SWE-bench Verified is a human annotator filtered subset that has been deemed to have a ceiling of 100% resolution rate.
If traders by majority do not want this change, we'll leave it at SWE-bench Full (which does not have a 100% resolution ceiling). And to make it fairer, it should be a majority of people voting NO.
Related questions
Related questions
Will an AI achieve >85% performance on the FrontierMath benchmark before 2028?
61% chance
80% on SWE-Bench Verified by Jan 1 2025
10% chance
Will an AI agent system be able to score at least 40% on level 3 tasks in the GAIA benchmark before 2025.
37% chance
AI resolves at least X% on SWE-bench assistance, by 2025?
AI resolves at least X% on SWE-bench WITH assistance, by 2028?
What will be the best score on the SWE-Bench (unassisted) benchmark before 2025?
39% chance
Will OpenAI models achieve ≥90% on SimpleBench by the end of 2025?
40% chance
Will any model get above human level on the Simple Bench benchmark before September 1st, 2025.
55% chance
Will an AI achieve >85% performance on the FrontierMath benchmark before 2027?
31% chance
Will an AI be capable of achieving a perfect score on the Putnam exam before 2028?
65% chance