“An AI singularity or intelligence explosion never happened.”

Removed a typo, this sentence uses to be

"As AI became safer as it got more powerful without much human effort outside of some RLHFing"

This question got me thinking about optimal formats for this, so I'm trying a weird one where all trades are cancelled after a week but then it re-resolves in 2060.

/Joshua/if-humanity-avoids-ai-doom-until-20

“AutoGPT6 escaped, started doing something, was caught and vivisected. This was enough of a warning shot to create AI CERN, which miraculously succeeded."

I am sufficiently pessimistic about humanity's ability to coordinate that I think most surviving worlds in 2100 are ones in which we are just lucky and it turns out that the relevant technology is much harder to invent than we think it is. Specifically, we might be lucky and:

A) The next AI breakthrough on the order of Transformers simply never arrives. LLMs keep getting better, but no amount of additional training data makes them a superintelligence.

or

B) It turns out there are no superweapons. Nanotechnology just doesn't work how we currently expect it to, engineering super-viruses turns out to be impossible, etc. Without any easy way to kill us all instantly, AI decides to work with us instead.

I am sure someone could phrase these better than me, but they are what I'm hoping for. I still think we should be desperately trying to coordinate moratorium treaties and develop human intelligence augmentation etc, but I doubt we pull those off.

@Joshua I’d like to second (B) especially. I work in nanoscience and I’m shocked by how seriously people take Eric Drexler’s ideas (I hesitate to say pseudoscience, but they’re certainly not very rigorous). I just don’t think it’s plausible that even a superintelligence could figure out how to engineer self-replicating nanobots and the like.

Previous questions like this to mine for answers:

The new answer format you're using is better, of course.

@Joshua Thanks. I totally missed this despite doing some searching. Maybe I'll close the market if the overlap is too large.

@Jono3h Okay, that being a "pick one" question makes it pretty unappealing. My market stays live!

@Jono3h Don't close it! An unlinked multichoice is much better than those old linked parimutual markets.

@Joshua Oh this also exists which is a duplicate of EY's market but with the current linked format that allows shorting:

So I would expect it to perhaps have more accurate percentages than EY's original, even though it has fewer traders. Probably also worth including the description?

My reasoning is that in general, large groups of people mostly make big changes in response to disasters.

Most of my probability mass is on things like energy scarcity, climate issues, etc. that just make AI research unfeasible.

Also significant is a failed takeover, causing everyone to understand the risk more viscerally. But that's hard to estimate. It's hard to imagine an AI causing significant enough damage without also just winning.

If it turns out to be too hard to make creative agents, then we survive for free. I wouldn't count on it but possibly it's true.

reposting because its a good question, please submit!

this is a great question, but is too far beyond my planning horizon to expect useful resolution. no bet.

@L 2030 or 2040 feel more bettable

I can make a copy of this for 2040, though I expect similar results (and these questions cost me 30% of my entire capital to create)
Gimme bucks or make one and I'll link it.