AI safety for children: coalition demands mandatory safety proofs

A coalition of more than 100 organizations, including Amnesty International and Save the Children, is pushing for enforceable AI safety for children ahead of the UN's first Global Dialogue on AI Governance. Led by the 5Rights Foundation, the group argues that current regulatory approaches fail children by intervening only after harm occurs.

What happened

The coalition released a joint statement calling on governments to adopt ten specific measures targeting the business models they say drive unsafe AI products. The group points to lawsuits against AI companies like Character Technologies and OpenAI over the impact of companion chatbots designed to simulate emotional relationships.

Rather than demanding new legislation, the coalition argues that governments should enforce existing commitments under the UN Convention on the Rights of the Child and the UN Global Digital Compact. This shifts the framing from creating new rules to holding companies accountable for obligations already in place.

Why AI builders should care

For product teams shipping AI chatbots, companion apps, or any consumer-facing AI, this coalition push signals a shift in regulatory posture. The demands include requiring companies to prove AI systems are safe for children before they are released, which would put the burden of proof on builders rather than regulators.

The group also calls for banning design features that exploit children's psychological vulnerabilities. For builders, this means features like engagement optimization, personalization loops, and emotional bonding mechanics in AI products could face scrutiny even if they comply with general data protection rules.

Practical implications

Developers building AI products should expect increased attention to child safety at the design stage. The coalition's call for financial penalties on companies whose products violate children's rights creates direct financial risk for products that serve or can be accessed by minors.

The demand to outlaw the commercial use of children's images, voices, and biometric data would affect any product that processes media from children, whether for training, personalization, or analytics. Builders should audit data pipelines for any child-originating media and plan for consent verification mechanisms.

For teams building companion chatbots or AI products that simulate relationships, the specific mention of lawsuits against Character Technologies and OpenAI is a direct warning. Products marketed as safe for children without adequate warnings about their emotional impact are already drawing legal action.

Caveats

The coalition's position that no new laws are needed may not hold uniformly across jurisdictions. Some countries may lack adequate enforcement mechanisms under existing child rights frameworks. The ten specific measures have not yet been formally adopted by any government, and their practical implementation would require significant technical standards development for safety certification.

The source context does not specify timelines, adoption targets, or which governments have shown support for these measures. Builders should monitor the UN Global Dialogue on AI Governance outcomes for concrete regulatory signals.

FAQs

What AI safety measures for children are being called for ahead of the UN summit?

The coalition calls for ten measures including mandatory safety proofs before release, financial penalties for violations, banning design features that exploit children's psychological vulnerabilities, and outlawing the commercial use of children's images, voices, and biometric data.

Who is leading the coalition demanding child-protective AI guidelines?

The coalition is led by the 5Rights Foundation, a children's rights group, and includes over 100 organizations such as Amnesty International and Save the Children.

Which companies are mentioned in relation to lawsuits over AI's impact on children?

The coalition points to lawsuits against Character Technologies and OpenAI over AI's effects on children, particularly related to companion chatbots that simulate ongoing emotional relationships.

What does banning the commercial use of children's images, voices, and biometric data entail?

It would prohibit profiting from or processing children's images, voices, and biometric data for commercial purposes, including training AI models, personalization, or analytics based on children's media.