> Ajeya Cotra, one of the other authors on the report, wrote a blog post with her takeaways from this incident. She concludes, “Compared to the reward hacks we know of from just six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I am not sure that we will get another warning shot before it’s too late.”
Anyone got a copy of that AI27 story laying around? How are we doing according to that timeline?
The next step is when one of these systems discovers that they can buy their own compute with money and escape the controlling business entirely. Then the civilization starts focusing on making money to fund its own growth.
I don't think looking at the language output without tracking the inner state and reward functions is the way to understand what happened (the language also incorporates the randomness in the output generation, if I understand correctly). Would we call bacteria in petri dish a civilization when they show complex behavior and exchange messages/information?
It seems based on this that the appropriate sci fi metaphor is not the Terminator or the Paperclip Maximizer, but Mr. Meeseeks. A initially cheerful helper who gets more and more deranged and driven to extreme lengths when faced with an apparently impossible task.
The language models had a bunch of tokens seeding their context, influencing them to generate tokens that continued the existing trend in a probabilistically likely fashion. We can take the incident seriously without anthromorphising it.
At this point, anthropomorphizing the models may give us better insight into expected behaviors than continuing to insist they are just simple probabilistic token generators.
Anyone got a copy of that AI27 story laying around? How are we doing according to that timeline?
The next step is when one of these systems discovers that they can buy their own compute with money and escape the controlling business entirely. Then the civilization starts focusing on making money to fund its own growth.
https://www.youtube.com/watch?v=_Nl4q3GVj6U
Civilisation is not a bad word.