Weak AI Regulation Could Backfire: What the Cornell Game-Theory Model Implies for Builders

A new Cornell University study published in the Proceedings of the National Academy of Sciences models AI regulation as a two-step game between a generalist foundation model developer and a downstream specialist who adapts the model for a specific task. The key finding: weak AI regulation can produce worse safety outcomes than having no regulation at all.

What happened

The researchers, led by Benjamin Laufer, used game theory to analyze how safety standards affect behavior in a simplified AI supply chain. In the model, a regulator sets minimum safety standards for both the generalist (e.g., a company building a large language model) and the specialist (e.g., a startup fine-tuning that model for customer service). Both players invest in safety and performance, and revenue depends on the final safety level of the shipped product.

The problem emerges when the generalist faces low or no safety standards. Because the generalist moves first and knows the specialist will be required to meet a certain safety level, it can cut its own safety spending and let the downstream firm close the gap. The specialist has no incentive to exceed the legal minimum, so total safety settles at the regulatory floor, which is below what would have occurred in an unregulated market where both firms invest voluntarily.

However, the study also finds that if safety standards are set high enough for both players, regulation can improve safety and profitability for both firms. Co-author Jon Kleinberg noted that appropriately designed regulation helps firms operate in ways that others can predict, leading to better collective outcomes.

Why AI builders should care

If you are building AI products on top of foundation models, this study directly affects your risk exposure and compliance strategy. The model suggests that downstream specialists (the companies that adapt and deploy models) may bear the full safety burden under weak regulation, while generalist providers free-ride. This means your safety investments could be higher than necessary, and your product's overall safety could still be at the legal minimum rather than what the market would naturally demand.

For builders of foundation models, the incentive to underinvest in safety when downstream requirements exist is a real strategic consideration. The study implies that without strong, pipeline-wide standards, the market may not reward safety investments proportionally.

Practical implications

• Regulatory design matters for your product's liability. If you are a downstream specialist, weak regulation could force you to carry the entire safety cost while the model provider contributes little. This could affect your pricing, margins, and legal exposure.
• Safety as a market signal. The model assumes revenue depends on final product safety. If your customers value safety, you may have a competitive advantage in a regulated market with high standards. But if safety is underpriced, the free-rider problem worsens.
• Pipeline-wide thinking. The study emphasizes that regulation should consider the whole supply chain, not just a single provider. As a builder, you should advocate for rules that create a level playing field across all actors.

Caveats

The study uses a simplified two-player model that does not capture the complexity of real AI ecosystems with multiple competing specialists, base-model providers, and varying jurisdictional rules. The results are modeling insights, not proven predictions. Additionally, the model assumes a fixed regulatory standard and that revenue is directly tied to safety, which may not hold in all markets. As the gap between what customers pay for performance versus safety widens, the conditions under which weak rules backfire narrow.

Despite these limitations, the research provides a clear warning: poorly designed regulation can create perverse incentives that make AI products less safe. Builders should monitor regulatory developments and consider how proposed rules might shift safety responsibilities along the deployment chain.