Governments around the world are racing to regulate AI before it becomes too deeply embedded in society. But new research suggests poorly designed rules could make AI systems less safe than having no regulation at all.
Regulatory disagreements in the US are leading to a patchwork of approaches as states take matters into their own hands. A key question is who should be responsible for the safety of AI products—the big tech companies building the underlying models or the firms that adapt them for a particular task, such as a customer service chatbot or an AI tutor.
Working this out is trickier than it looks. While it might seem logical to put the bulk of the burden on downstream companies directly serving these tools to customers, a new study in Proceedings of the National Academy of Sciences finds that could be worse than having no rules at all.
“There’s a free-riding behavior that occurs,” Benjamin Laufer from Cornell University, who led the research, said in a press release. “The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist.”
The researchers’ analysis relied on a model based on game theory—a mathematical approach to studying decision making. It treated AI development as a two-step game, in which a “generalist” developer first invests in building a broadly capable AI model before a “specialist” adapts it for a specific domain and takes it to market.
In the game, a regulator sets a minimum safety standard for both players, and the models see this in advance. They then invest in both the performance and safety of their product, and the revenue is split between them. Investments in both get progressively higher, while the extra revenue each improvement brings in stays flat.
The problem, the researchers found, is that the generalist moves first and knows exactly what the specialist will be legally required to do afterwards. This creates problems when the generalist is set a low bar for safety, or none at all, and safety standards for the downstream specialist are also fairly weak.
In the absence of any rules, both firms invest in safety, because the model assumes a safer product earns more revenue. But if the specialist is forced to invest a certain amount into safety to meet regularity requirements, the generalist can cut its own spending and let the downstream firm close the gap.
That’s because the generalist’s revenue depends on the final safety level of the shipped product, not on its own contribution, so it can get a revenue boost from improved safety without paying for it from its own pocket. The specialist, for its part, has no reason to do more than the rule demands, so total safety settles at the legal minimum, which is below what would have occurred had there been no regulation at all.
On a more positive note, the researchers found that if safety levels on both the generalist and the specialist are set high enough, regulation can actually improve safety while leaving both companies more profitable than they were in an unregulated market.
“Appropriately designed AI regulation can make it possible for different firms involved in the AI development pipeline to collectively arrive at good outcomes for consumers, knowing that the regulation is designed to help each firm operate in a way that the others can more reasonably predict,” co-author Jon Kleinberg from Cornell University said in the press release.
However, the researchers’ model relies on the market setting a real price on safety. As the gap widens between what customers will pay for performance and what they’ll pay for safety, the range of circumstances in which weak rules backfire gets narrower.
The authors also note that the model’s two-player setup is a simplification of real AI supply chains where multiple competing specialists and base-model providers operate across different jurisdictions with different rules.
“People think of AI as a single object, but actually AI involves a very complicated set of stakeholders and actors that each have their own contributions to the technology,” said Laufer. “To regulate in a thoughtful way, we need to consider the whole supply chain, not just a single provider or entity.”
Still, the results suggest that taking an overly simplistic and light-handed approach to AI regulation may end up achieving the opposite of what law makers intend.
