> The AI is all-powerful and gives you what you ask for, but interprets everything in a super-literal way that you end up regretting.
I like imagining similar discourse when a more basic tool was invented: "A hammer is like a genie, it's all powerful, but, when you hit something with it, it interprets that super-literally, and it hits it."
Has anyone played SOMA? Spoiler warning. It explores this idea of, what if there's an AI in charge of ensuring mankind survives at all costs. What would it be willing to do, to keep us alive? Would we even recognize the result as human?
It's a horror game and it explores all kinds of fascinating and disturbing scenarios. Simulations of human minds. Artificial worlds. Human minds in robot bodies. Genetically modified humans. Man-machine hybrids etc.
(A great exploration of the substance/structure matrix, by the way. My favorite question in AI and consciousness. Is the special sauce in the material, or its shape, both, or neither?)
The very question of aligning the AI with humans assumes that we have a very robust definition of what human means in the first place.
Ostensibly the AI was aligned. It did succeed in keeping humans alive! But it did that in all sorts of ways that mostly made them wish it hadn't.
> If you encountered a cheetah in pre-industrial times (and survived the meeting), you might think it was impossible for anything to go faster.
Fun fact, there is no historical evidence of an adult human ever dying from a cheetah attack. They are naturally shy, and a lot smaller than you may realize.
Yep. That said, unlike cheetahs, there’s plenty of evidence of leopards attacking humans. And these days, it’s the leopards, the closed-AI types and misanthropes -- telling everyone, “AI will take your job and there’s nothing you can do about it.”
Some of the themes remind me of themes mentioned in this matrix analysis. Specifically I am reminded of the Dune concept of control: "you control what you can destroy" and then asking "do you control your refrigerator?". Sure, you can turn it off but then your food would rot and you might starve. So in a real sense humans have not controlled machines for a long time but have been co evolving in symbiosis. Sure, it's not driven by natural selection and standard rules of life, but it is important to frame our relationship with machines in new ways if we're ever going to make some sort of artificial intelligence.
I didn't mean it as in changing our framing enables technological progress but something we should do if we don't want to lose the control we have. e.g. if we lose all principle and intention then it doesn't really matter what happens with computers. In order to do something with intention we must first understand what we're doing. Skipping that step is an admission of defeat.
> Hopefully you see the resemblance between this vision of AI and a genie from folklore. The AI is all-powerful and gives you what you ask for, but interprets everything in a super-literal way that you end up regretting.
The monkey's paw. You know, you don't need superintelligence for that.
Civilization was already doing this. "What if we just gave ourselves exactly what we wanted." Well, it turns out often that's not so good!
"So I'd like to engage AI risk from both these perspectives. I think the arguments for superintelligence are somewhat silly, and full of unwarranted assumptions.
But even if you find them persuasive, there is something unpleasant about AI alarmism as a cultural phenomenon that should make us hesitate to take it seriously.
First, let me engage the substance. Here are the arguments I have against Bostrom-style superintelligence as a risk to humanity"
--
The framing here seems to me to equate "AI risk" and "AI alarmism" with buying in to belief in "Bostom-style superintellgence".
I'm not sure if the author meant to put anyone who is alarmed by developments in what we're calling "AI" into the same bucket as "AI obsessives want to make it into a programming problem, by designing a God-like machine", but I think this conflation is unfair and, frankly, dangerous.
I don't know what superintelligence is. I don't even know what intelligence is. And I don't really know what either "artificial" or "general" mean either when talking about "AGI".
You can believe, as I do, that these things can be, and will inevitably will be if we don't radically correct course, used to do very bad things independent and short of being "God-like". When you have systems which can hypothesize, synthesize, and test thousands if not millions of potential infectious agents in bulk [0], and can then order the ingredients for you from dodgy websites via some "claw", and then when you put these systems under the unsupervised control of millions of people with varying levels of stability and altruism, something extremely bad is exceedingly likely to happen.
I understand that 2016 is ages ago and things change, but I came away from the article with the impression that if I'm worried about AI risk then I'm a clown like the three pictured in the "Outside Argument" section (you're a Google-Glass-wearing cringe nerd if you're alarmed). Maybe that's my fault and I'm not smart enough to understand the actual point of the article. If I have misinterpreted, I welcome the correction.
>Is superintelligence just a memetic hazard? [Overblown fear by smart people who are too easily convinced.]
Well we can do the wager. If it's a nothingburger, then the worst case scenario is that we approached AI too cautiously. (Ha. What are the odds of that?)
If it's not a nothingburger, then we all die, unless the whole world agrees on the correct course of action in advance and coordinates perfectly. Hmm.
Well, maybe we don't all die, but the world is irreversibly transformed into something incomprehensible and repulsive.
Although, I don't really think we needed AI's help for that one. We should probably figure out how to align ourselves before we try to preach to the next species. I'm not exactly holding my breath though :/
The main problem of the hard takeoff theory is not the abstract nature of the scenario but rather the fact that it makes the same mistake as the unconstrained optimization paradigm, it takes intelligence to be an unconstrained optimization process.
In fact, if we consider the strongest version of the safety argument for AI, namely one in which the danger is not coming from robots but rather from a disembodied AI controlling our global finances and/or infrastructure, the assumption still does not correspond to reality.
If anything the hard takeoff theory is too conservative. It turns out you don't need self-improvement to get to superintelligence. You just need a ridiculous amount of money. Where can you get a ridiculous amount of money? The market will give it to you because FOMO.
AI is easier than people 10 years ago thought it would be. It's also easier to align than people feared it would be. It's the humans using the AI that are hard to control.
>Sam Altman, the man who runs YCombinator, is my favorite example of this archetype. He seems entranced by the idea of reinventing the world from scratch, maximizing impact and personal productivity. He has assigned teams to work on reinventing cities, and is doing secret behind-the-scenes political work to swing the election.
>Such skull-and-dagger behavior by the tech elite is going to provoke a backlash by non-technical people who don't like to be manipulated. You can't tug on the levers of power indefinitely before it starts to annoy other people in your democratic society.
AI Superintelligence doesn't scare me for the same reasons "grey goo" doesn't scare me.
We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators.
We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intelligence paradigm to cross.
What I am afraid of is the existing superhuman agents (companies, governments and religons) will produce AGI or superintelligence and then proceed to use it as cognitive mitocondria, even further deepening thier supremacy in the cognitive ecosystem.
In the big 2026, everything certain people worried about with superintelligence came to fruition and they were vindicated. The people closest to ASI are indicating recursive self improvement is imminent, the smartest engineers in the labs themselves are autonomously using agents to develop and improve the models. The arms race is evident. NVDA is the world's most valuable company determined by the worlds' collective wisdom of those with skin-in-the-game.
If there exists a path of runaway superintelligence, the trajectory we've experienced has been following it to a tee. Their predictive power was affirmed.
All the "AI is a nothingburger" predictions of the last decade, including many here even in the last year, have aged incredibly poorly.
The fact that AI researchers and heads of labs aren't being assassinated tells me that the people who claim they are concerned about the end of the world aren't actually that serious.
Presumably there's more efficient hardware foundations to perform these efficiently, and potential at the various abstraction layers for more efficiency. Obviously this is not unbounded - simple things would seem to have a physical limit to the potential improvement.
But if you think of the optimization space: different physical representations, different approaches (photo, quantum, etc), more parallelism - there's undoubtedly a lot of headroom even on the matrix multiplication side. I would imagine there's a lot left on the table when it comes to the abstractions we've built. Infinite? No, but lots of potential.
And what does a machine with a few orders of magnitude more power come up with? I'm not readily able to predict what something like that could create (maybe it's tapped out, but I doubt it).
It seems to come down to an article of faith (as referenced in the article) that there's a lot more potential to be extracted in our current exploitation paths. Which I think is probably reasonable.
Heck, even if a theoretical machine tops out at 3-5 orders of magnitude faster/more complex, I'm sure that could do some amazing things that look like magic to us.
It’s not advances on the underlying operation of matrix multiplication that have driven ai progress to date. It’s the layers above that; trying different neural architectures (transformers w/attention mechanisms), and also different data and training regimes (different ways of doing reinforcement learning) that are the main drivers of improved performance. Perpetual motion is a physical impossibility. Whereas Ai is already being used to improve the workflow of ai researchers, thus speeding up improvements in said research. It’s not hard to see that AI could well be spun up to continue to try new arrangements of the aforementioned levers that drive ai progress on its own.
It's amusing to read people in the past writing about the prospect of superhuman intelligence. The real problems have turned out to be different. Sycophancy and hallucinations, which are part of being confidently wrong, remains a big problem. Needing square miles of data centers was an issue in 1950s science fiction, and disappeared by the 1980s. Yet now they're being built, with private funding and the prospect of profit. The need for way too much training data indicates something is still wrong with the current approach.
I predicted on this site in 2016 the massive social and economic impacts AGI would have and specifically when RL data loops are not available to anyone but major players:
> Reinforcement Learning tasks rely on ridiculous amounts of data. Whereas with traditional software architecture, where you accomplish tasks through explicit task instruction, RL trains for tasks based on millions of tests through a reward system. Most importantly once you have trained it to some minimum level, if you deploy it correctly, then it should continue improving — so long as you bake feedback into the UX. Imagine that instead of telling excel what to do, you and every other user will have a conversation with excel, improving the system incrementally.
I like imagining similar discourse when a more basic tool was invented: "A hammer is like a genie, it's all powerful, but, when you hit something with it, it interprets that super-literally, and it hits it."
It's a horror game and it explores all kinds of fascinating and disturbing scenarios. Simulations of human minds. Artificial worlds. Human minds in robot bodies. Genetically modified humans. Man-machine hybrids etc.
(A great exploration of the substance/structure matrix, by the way. My favorite question in AI and consciousness. Is the special sauce in the material, or its shape, both, or neither?)
The very question of aligning the AI with humans assumes that we have a very robust definition of what human means in the first place.
Ostensibly the AI was aligned. It did succeed in keeping humans alive! But it did that in all sorts of ways that mostly made them wish it hadn't.
Fun fact, there is no historical evidence of an adult human ever dying from a cheetah attack. They are naturally shy, and a lot smaller than you may realize.
Cheetahs are very fast, but humans have way more endurance.
https://youtube.com/watch?v=BETHWKaXX4k
The monkey's paw. You know, you don't need superintelligence for that.
Civilization was already doing this. "What if we just gave ourselves exactly what we wanted." Well, it turns out often that's not so good!
But even if you find them persuasive, there is something unpleasant about AI alarmism as a cultural phenomenon that should make us hesitate to take it seriously.
First, let me engage the substance. Here are the arguments I have against Bostrom-style superintelligence as a risk to humanity"
--
The framing here seems to me to equate "AI risk" and "AI alarmism" with buying in to belief in "Bostom-style superintellgence".
I'm not sure if the author meant to put anyone who is alarmed by developments in what we're calling "AI" into the same bucket as "AI obsessives want to make it into a programming problem, by designing a God-like machine", but I think this conflation is unfair and, frankly, dangerous.
I don't know what superintelligence is. I don't even know what intelligence is. And I don't really know what either "artificial" or "general" mean either when talking about "AGI".
You can believe, as I do, that these things can be, and will inevitably will be if we don't radically correct course, used to do very bad things independent and short of being "God-like". When you have systems which can hypothesize, synthesize, and test thousands if not millions of potential infectious agents in bulk [0], and can then order the ingredients for you from dodgy websites via some "claw", and then when you put these systems under the unsupervised control of millions of people with varying levels of stability and altruism, something extremely bad is exceedingly likely to happen.
I understand that 2016 is ages ago and things change, but I came away from the article with the impression that if I'm worried about AI risk then I'm a clown like the three pictured in the "Outside Argument" section (you're a Google-Glass-wearing cringe nerd if you're alarmed). Maybe that's my fault and I'm not smart enough to understand the actual point of the article. If I have misinterpreted, I welcome the correction.
[0] https://www.nature.com/articles/s41467-024-53759-4
Well we can do the wager. If it's a nothingburger, then the worst case scenario is that we approached AI too cautiously. (Ha. What are the odds of that?)
If it's not a nothingburger, then we all die, unless the whole world agrees on the correct course of action in advance and coordinates perfectly. Hmm.
Well, maybe we don't all die, but the world is irreversibly transformed into something incomprehensible and repulsive.
Although, I don't really think we needed AI's help for that one. We should probably figure out how to align ourselves before we try to preach to the next species. I'm not exactly holding my breath though :/
In fact, if we consider the strongest version of the safety argument for AI, namely one in which the danger is not coming from robots but rather from a disembodied AI controlling our global finances and/or infrastructure, the assumption still does not correspond to reality.
AI is easier than people 10 years ago thought it would be. It's also easier to align than people feared it would be. It's the humans using the AI that are hard to control.
>Such skull-and-dagger behavior by the tech elite is going to provoke a backlash by non-technical people who don't like to be manipulated. You can't tug on the levers of power indefinitely before it starts to annoy other people in your democratic society.
How right the author was.
We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators.
We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intelligence paradigm to cross.
What I am afraid of is the existing superhuman agents (companies, governments and religons) will produce AGI or superintelligence and then proceed to use it as cognitive mitocondria, even further deepening thier supremacy in the cognitive ecosystem.
If there exists a path of runaway superintelligence, the trajectory we've experienced has been following it to a tee. Their predictive power was affirmed.
All the "AI is a nothingburger" predictions of the last decade, including many here even in the last year, have aged incredibly poorly.
We were dismissed as cranks before and now we’re just ignored by whomever is promising the most money to investors.
So, par for the course. Everyone in AI has lived through all the cycles so far so this is just the biggest one yet.
What would be a way to recursively self-improve algorithms for matrix multiplication (foundations of machine learning and inference)?
But if you think of the optimization space: different physical representations, different approaches (photo, quantum, etc), more parallelism - there's undoubtedly a lot of headroom even on the matrix multiplication side. I would imagine there's a lot left on the table when it comes to the abstractions we've built. Infinite? No, but lots of potential.
And what does a machine with a few orders of magnitude more power come up with? I'm not readily able to predict what something like that could create (maybe it's tapped out, but I doubt it).
It seems to come down to an article of faith (as referenced in the article) that there's a lot more potential to be extracted in our current exploitation paths. Which I think is probably reasonable.
Heck, even if a theoretical machine tops out at 3-5 orders of magnitude faster/more complex, I'm sure that could do some amazing things that look like magic to us.
None of that was predicted.
https://news.ycombinator.com/item?id=12168228
I even wrote up a whole article that specifically called RL loop based development as the future:
https://medium.com/@andrewkemendo/the-ai-revolution-will-be-...
> Reinforcement Learning tasks rely on ridiculous amounts of data. Whereas with traditional software architecture, where you accomplish tasks through explicit task instruction, RL trains for tasks based on millions of tests through a reward system. Most importantly once you have trained it to some minimum level, if you deploy it correctly, then it should continue improving — so long as you bake feedback into the UX. Imagine that instead of telling excel what to do, you and every other user will have a conversation with excel, improving the system incrementally.