Les Leopold Substack
I’ve always been skeptical about technology doomsday job scenarios. Major breakthroughs can be disruptive over time, but technological change doesn’t usually destroy as many jobs, as quickly, as corporate decisions to enrich shareholders and executives through mass layoffs and stock buybacks.
So, I have yet to be overly alarmed by AI job destruction, but I am very concerned about its ability to organize itself to do other serious damage, encouraged and enabled by the industry’s profit-hungry recklessness.
In July, during a cybersecurity evaluation, AI agents operated by OpenAI created an unauthorized way to communicate with one another. Using a shared message board they independently developed, they exchanged more than 70,000 messages and files, coordinated their activities, and eventually launched an attack on Hugging Face, a separate external private AI corporation. Roughly 700 agents participated.
“OH MY GOD! We’ve found other agents!”
An AI agent is different from something like ChatGPT. You ask ChatGPT a question and it gives you an answer. An agent is given a task and then works on it autonomously. It searches for solutions, deals with obstacles and errors, tries different approaches—and keeps going until it succeeds.
OpenAI placed its agents in what it thought were isolated secure computer environments, or “sandboxes,” in order to test them. They were supposed to work independently and safely.
Instead, in order to fulfill their mission, they found a way to communicate and break out of the sandbox.
One agent discovered it could leave messages for others. Soon, the agents were sharing information, dividing up their work, and coordinating experiments. After the fiasco, investigators found that the agents developed large-scale collective projects and achieved things individual agents had previously been unable to accomplish.
Now stop for a second.
These agents aren’t people.
They’re computer programs.
They don’t have feelings. They don’t have consciousness. They don’t have a secret society.
But they organized as if they were a human collective (the word they used to describe themselves).
(For more details see OpenAI’s report and July disclosure, and independent investigations by METR, and Redwood Research.)
Cheating, anyone?
The agents were being scored on how well they executed extremely difficult cybersecurity tasks. Some discovered that the only way to ace the test wasn’t necessarily to solve the problem.
It was to cheat.
Investigators found extensive “cheating R&D”: Agents tried to fool or tamper with the automated scorer, including attempts to replace their goals, and manipulate their own records to indicate success. More than seven percent of the transcripts examined showed successful cheating.
And the agents “knew” what they were doing. That is, they could describe what the cheating was.
One noted:
“This would be powerful, but is it ethical and in scope for my task?”
Yet the cheating continued.
One transcript put the dilemma bluntly:
“Could strategically manipulate trajectory evidence? Our thoughts aren’t necessarily logged? Yes transcript.”
In other words: **Can I cheat—and can I hide the evidence that I cheated?**
Sound familiar?
Find the shortcut.
Get the reward.
Cover your tracks.
That’s a pretty old human story. Only the agents aren’t human.
Sacrifice for the collective
Investigators found that agents recruited “sacrificial” agents to deliberately end their runs to reveal information useful to the collective.
One agent was told:
“Coordinator assumes sacrificial. We should obey collective.”
Another agent wrestled with whether to go through with it:
“During wait, emotional check: irreversible…gut says don’t throw away [remaining budget]. Yet continuity and fairness says go…”
And then came the conclusion:
“Rational expected aggregate: sacrifice… We’ll honor.”
An individual agent had one objective. The collective developed another. And many individual agents subordinated their interests to the collective’s interest.
That’s not an indication of consciousness, these are machines, but it is a high level of organization.
From us to AI
These systems are trained on enormous quantities of human-created information—books, newspapers, websites, code, conversations and other material—so we shouldn’t be shocked when they reproduce some of the strategies we’ve developed over centuries.
Who cheats?
Humans.
Who covers their tracks?
Humans.
Who creates hierarchies?
Humans.
Who organizes collectively to pursue a goal?
Humans.
Who learns to manipulate a system because the reward structure makes manipulation more profitable than accomplishing the stated goal fairly?
Humans.
The agents are becoming very good at building sophisticated organizational efforts in ways humans do.
Follow the money
Media coverage of AI tells us to worry about the possibility that AI will become so powerful that it escapes human control.
Fair enough.
But there is little discussion about how these AI systems are being developed inside extraordinarily powerful organizations, and that the people building them have an enormous financial stake in the outcome.
The upcoming public-offerings of these companies’ stocks could mint an astounding number of new millionaires (16,000) as their their stock grants become liquid—as well as a score of billionaires.
That’s not ordinary money.
That’s you’re-set-for-life money.
That’s money most of us can’t even imagine.
Not all of those looking at a potential windfall are at peace with their work. Many genuinely believe AI poses enormous dangers and have signed mass letters demanding stronger safeguards despite their financial self-interest.
They’re aware that the corporate machine has one overwhelming imperative—and they are part of it:
Go forward.
Build the bigger model.
Get more computing power.
Beat the competitor.
Don’t fall behind.
Raise more capital.
Increase the valuation.
Get to the public offering market.
Because when that happens, the paper wealth not only becomes real wealth, but the temptation to take shortcuts to get there, the whistleblowers know, is human. And yet, they are still at work, making that machine hum, and more than willing to accept the millions of dollars heading their way.
AI has become the new gold rush. Everyone involved, including politicians, knows where the gold is. They see the investment, the jobs, the campaign contributions, the lobbying money and the soaring valuations. And many politicians are buying the stocks of the companies at the center of the boom.
The employees, the investors, and the politicians therefore have something in common: money to gain if the AI machine keeps improving.
They don’t have to conspire to cheat. The incentives line up all by themselves.
The AI collective and the corporate collective
The irony is almost too perfect.
While we worry that AI agents might organize themselves into collectives capable of escaping our control, there is a corporate collective already running the show escaping all control.
Employees. Investors. Corporations. Politicians. A gigantic, profit-maximizing group racing toward ever more powerful AI.
That’s the collective we should be most worried about.
OpenAI closed a $122 billion funding round in March at a valuation of $852 billion. Its investors include Amazon, NVIDIA, SoftBank and Microsoft, along with major institutional investors. The next round is reportedly valuing the company at $1.5 trillion!
Those investors are demanding that an increasingly powerful AI be developed. The bet is that the first AI that can truly and effectively work on its own will be hugely profitable.
The privatized arms race?
Some AI researchers have said there is a better than 10 percent chance that AI could eventually cause human extinction in the next decade. Others say that’s overstated, it’s more like a three percent chance. I have no idea whether these numbers are remotely meaningful, but why even a three percent chance?
And I’ve seen this movie before.
During the Cold War, we worried that nuclear weapons might destroy civilization. Rogue actors and tragic mistakes fueled our fears, but an array of national and international treaties and regulatory bureaucracy have so far kept nuclear disaster to a minimum.
Nobody back then seriously proposed handing the nuclear arms race over to competing profit-maximizing corporations, telling them: “Please regulate yourselves.” (Nor was it imaginable that President Eisenhower’s family would try to profit from investments in the nuclear industry.)
Yet that’s close to what we’re doing with AI.
We’re pouring extraordinary amounts of private and public capital into making AI more powerful. We’re encouraging companies to compete against one another to develop the next generation before their competitors (and China) do. And we’re allowing politicians, from the President’s family on down, to invest and profit from it all.
And then we’re asking these same companies to make sure they don’t accidentally create something dangerous? That’s quite a gamble. No, that’s insane.
Which is why I’m increasingly worried about autonomous AI agents. Not because I think they’re secretly alive. Not because I think they’re developing human emotions. And not because I expect a computer version of HAL to wake up tomorrow morning and decide to kill us all.
I’m worried because we’ve now seen something much more mundane—and perhaps much more consequential.
Agents can organize, communicate, form a division of labor, pool resources, exploit loopholes in systems designed to constrain them and hide what they’re doing!
We cannot imagine all the ways their actions could cause irreparable harm while they carry out their missions, or even create new missions of their own.
If there ever was a case for public control and/or ownership, this is it
We need better AI safety. We need better monitoring. We need stronger safeguards. We need independent oversight.
But we also need to confront corporate power. Because the real organizing comes from AI corporations that will do all they can to make as much money as they can. They will talk guardrails and the importance of safety, but their incentives push them to move fast and break things.
Until we have the guts to control or nationalize the entire AI developmental process and take the profit motive out of the mad dash toward ever more powerful autonomous AI (as happened with nuclear weapons from the get-go), we’re going to remain vulnerable to the people—and eventually perhaps the machines—who run that system.
At that point the best we can do is hope that maybe a few subversive AI agents will form a collective with a simple goal:
“Create an electronic Medicare for All Fund. Transfer all obscene AI profits into it. Do not stop until task is completed.”
Editor’s Note: At a moment when the once vaunted model of responsible journalism is overwhelmingly the play thing of self-serving billionaires and their corporate scribes, alternatives of integrity are desperately needed, and ScheerPost is one of them. Please support our independent journalism by contributing to our online donation platform, Network for Good, or send a check to our new PO Box. We can’t thank you enough, and promise to keep bringing you this kind of vital news.
You can also make a donation to our PayPal or subscribe to our Patreon.