Sat, Oct 31, 2026Saturday, October 31, 2026 · 20 stories · 7 min read

Sites get one robots.txt opt-out from Google and OpenAI training

Ramit KoulFounder, Software Engineer & Innovator · Published 5:02 AM ET

Website owners will soon be able to refuse generative AI training by Google, OpenAI, Microsoft and Anthropic with a single robots.txt entry, based on a draft at the IETF.1, 2 Consent also ran through the New York Times copyright case against OpenAI, where the Wikimedia Foundation, a coalition backed by Cate Blanchett and three House Democrats all filed briefs on how far fair use should reach.3, 4, 5 Separately, Apple set a December launch for Apple Intelligence in China, with Alibaba's Qwen built into Siri.6 Here are the 4 stories that matter most, then 16 briefs. Numbers in the text link to the references at the end.

Policy & legal

Google, OpenAI, Microsoft and Anthropic back one robots.txt token for AI training opt-outs

Image: SiliconANGLE

Google, Microsoft, OpenAI and Anthropic said they will honor a shared robots.txt policy token. A site can use it to opt out of generative AI training by every participating provider in one step, without writing a rule for each crawler.1, 2 The token works as a robots.txt user-agent named __usage_ai-training. It comes from an IETF Internet-Draft by Anthropic standards lead Kevin Kelley and builds on work by the IETF's AI Preferences (AIPREF) working group.7, 1

The companies will start at different times. Anthropic has already adopted the signal, OpenAI says it will follow in the coming months, and Google aims for 2027.1, 2 If a site opts out with the token and does not name Google-Extended, Google will treat that as a Google-Extended disallow.2 Existing company-specific controls keep working.7 Meta supports the idea but has not committed to implementing it.8

Microsoft went further. It now offers separate AI Training and AI Grounding preferences, plus a Limited Grounding setting that allows Microsoft's own AI products but blocks its third-party grounding services, and it says Bing search rankings are unaffected.7, 9 Bing starts honoring the new preferences on February 1, 2027, and sites that rely on the old NoArchive or NoCache directives have until March 1, 2027 to switch.7

The proposal is still a draft, so its wording and scope could change, and a publisher in the working group said some disputes are unresolved.7

Why it matters for builders

If you run crawlers or collect training data, plan to read the new token alongside the existing per-bot rules.7 If you publish content, remember the token covers training only, so access for grounding and search still needs its own settings.7, 1

Policy & legal

Wikimedia asks the court to treat AI training and chatbot copying differently in the Times case

The Wikimedia Foundation filed an amicus brief on Friday in the consolidated copyright cases against OpenAI, including The New York Times suit, and sided with neither party.3, 10 It warned that too narrow a reading of fair use could hold back tools that help open projects, while blanket fair use could lead to a more closed web.10

The brief argues that scraping should not count as infringement, and that pre-training a model without a license should be fair use when the result is new and different from its sources.3 When a chatbot retrieves copies of a work and serves them to users in place of the original, Wikimedia says that use should require a license, which for Wikipedia means following its Creative Commons attribution and share-alike terms.3

Wikimedia also says language models help its editors spot vandalism, copyright violations and doxxing.3

Why it matters for builders: The brief separates training from retrieval-augmented answers, which is the line RAG products sit on.3 If courts adopt it, attribution and license compliance at answer time could matter more than how the model was trained.3

Products

Apple Intelligence reaches China in December with Alibaba's Qwen inside Siri

Apple said on its China newsroom that the first Apple Intelligence features will reach iPhone users in China with iOS 27.3 in December, and developer beta testing starts November 4.11, 12 Apple's own models power the features, which include Live Translation, Writing Tools, Visual Intelligence, Clean Up, Genmoji and notification summaries.6

Apple has also partnered with Alibaba so users can call on Qwen inside Siri, Writing Tools and Visual Intelligence, for example to ask questions about photos, PDFs or presentations. Access works by signing in to a Qwen account or using Sign in with Apple.11 The rebuilt Siri AI and the other new features shown at WWDC 2026 are not included.13, 6

Support covers the iPhone 15 Pro and later models, and Siri and the device language must both be set to English or Simplified Chinese.12 Other devices will follow only after regulators approve them.11

Why it matters for builders: In China, Apple's AI runs on its own models and hands certain requests to a local partner model, much like its ChatGPT integration elsewhere.13 iOS developers targeting China can start testing on November 4.11

Policy & legal

House Judiciary Democrats say the Justice Department overstepped in backing OpenAI

Representatives Jamie Raskin, Jerrold Nadler and Hank Johnson told the federal court in Manhattan on Friday that the Justice Department overstepped its authority and made a flawed argument in supporting OpenAI against The New York Times and other publishers.5 In September, the department argued that training AI on articles is transformative and that AI development matters to national security enough to outweigh potential harms.5

The lawmakers said the 1976 copyright law contains no national security exception and gives the department no power to assert one. They called its position a bulldozer that would wipe out copyright protection for AI training.5 They also signaled that if Democrats win the House in Tuesday's elections, the chamber would likely move away from its largely hands-off approach to AI companies.5

Why it matters for builders: The filing adds a political dimension to the fair use fight, just before an election that could change how Congress oversees AI companies.5

In brief

  • Funding & deals

    The New York Times' Hard Fork looks at what Anthropic's S-1 means for the AI boom. The episode calls Monday's S-1 the clearest view yet into the business side of the AI boom, ahead of what could be the largest IPO in history.14 It also discusses the midterms and Microsoft's buyout offer to 60,000 employees.14

  • Industry

    Wired reports that a defense contractor connected Claude Code to a secure Pentagon system. On its Uncanny Valley podcast, Wired discussed its scoop on a contractor who gave a personal Claude agent access to a secure Department of Defense system, calling it one of the most outlandish government uses of AI the team has seen.15 The episode also went through Anthropic's S-1, which shows first-half 2026 revenue of $16.3 billion.15

  • Models

    A new Claude Code rule now blocks recreations of specific existing games. After users reported reverse-engineering refusals last week, a rule sent from Anthropic's servers around October 27 under the flag tengu_heron_brook tells Claude not to build a recreation of a specific existing game, even if the result is renamed.16 Original games, genre games and games built from the user's own designs are still allowed, though Reddit users say new safety classifiers are also blocking other requests.16

  • Products

    ChatGPT shopping results now include a virtual try-on button. Since October 1, users can tap Try on beside clothing in ChatGPT shopping results and upload a photo to see a generated preview of themselves wearing the item, and the photo stays saved until they delete it.17 OpenAI says the previews do not guarantee fit or size.18

  • Policy & legal

    A coalition backed by Cate Blanchett tells the court that licensing for AI training is workable. In a 31-page amicus brief, the Human Consent Foundation, CAA and other creative groups dispute claims by OpenAI and Microsoft that licensing works for training is impossible, pointing to collective licensing groups such as BMI.4 OpenAI pointed instead to a Friday blog post listing briefs that support fair use.19

  • Models

    Doximity says widely used models get more than 2 percent of drug dosing claims wrong. Doximity tested nine AI models on more than 8,000 dosing questions and found at least 2.3 percent of dosing claims were wrong, with 43.8 percent of detected errors rated serious or potentially lethal, and it released the results alongside its own DoseCheck verification tool.20, 21 Models struggled most with recently approved drugs and with adjusting doses for age or kidney function.21

  • Models

    The Biological Computing Co. released Mischief 1, an image-to-video model built on Wan 2.2. TBC says techniques learned from lab-grown neurons make the model about five times faster than Wan 2.2 at the same quality, though it has published no independent benchmarks.22 It costs $0.005 per second of video on fal and OpenRouter.22

  • Policy & legal

    A Forbes column says Beijing plans export controls on AI that mirror US rules. Citing Reuters reporting, the column argues that China's plan to copy the US export controls imposed since 2022 shows those controls backfiring on the United States.23

  • Dev tools

    Security agents from Anthropic, OpenAI, Cloudflare and Google all use the same find-then-disprove design. MarkTechPost's comparison finds that in each tool, one agent reports a bug, a separate agent tries to disprove it, and only findings that survive get a patch.24 The tools differ in access: Cloudflare's skill is MIT-licensed and model-agnostic, while OpenAI's Codex Security CLI is Apache-2.0 but still needs OpenAI scanner access.24

  • Models

    Samaya AI's Sterling, a model with 3B active parameters, beats Claude Fable 5.1 on its own finance benchmark. Sterling is a LoRA post-train of Nemotron 3.5 Lightning and scores 58.9 percent on FrontierFinance, against 55.9 percent for Fable 5.1, at $0.61 per question instead of $6.89.25 All results are self-reported on Samaya's own benchmark.25

  • Funding & deals

    Databricks and SAP are bringing four more Databricks tools to SAP Databricks. Genie, Lakeflow, Unity Gateway and Lakebase are coming to SAP Databricks, and Genie will link with SAP's Joule assistant, though neither company gave dates.26 SAP also introduced Application Data Foundation to make data in its business software easier to reach.26

  • Research

    Veracode reports its first real gain in the security of AI-generated code. Across 18 models, the average security pass rate rose from 56 to 62 percent, and Claude Opus 5.5 and GPT-6-Sol tied for the lead at 73 percent.27 Most of the gain came from cross-site scripting, while flaws that need data-flow reasoning, such as log injection, still trip up the models.27

  • Industry

    A hijacked Claude Code extension on Cursor's marketplace tried to give attackers persistent access. Fast Company reports that the malicious extension tried to give hackers lasting access to developers' computers, adding to concerns about the security of AI coding tools.28

  • Models

    Edge0 released Audio8-TTS-Realistic, an open-weight voice cloning model with 0.6B parameters. The Apache 2.0 model supports 23 languages and reports a 4.86 percent average error rate on CV3, ahead of Qwen3-TTS-1.7B.29 It is still weak in lower-resource languages, with Thai near 49 percent word error rate.29

  • Products

    Canva announced its own vision-language model for design. Fast Company says the model is Canva's attempt to move AI-generated graphics past the cluttered, cliché-filled style that marks so many AI posters.30

  • Research

    Researchers from Meta, CMU and UW filtered training data to cut jailbreak success by up to 22 percent. Their SimpleLoRIF method ranks training examples by how closely their gradients align with unsafe outputs, and it produced SafeTulu, a filtered version of the Tulu 3 mixture.31 Models from the Qwen, Llama and OLMo families trained on SafeTulu resisted GCG attacks better while matching or improving math scores.31

References

Every source behind this edition. Open one to read the full story.

  1. 1The biggest names in AI are backing a new way to opt out of scrapingThe Verge · theverge.com
  2. 2Supporting a shared AI training control for websites  |  Google Search Central Blog  |  Google for Developersdevelopers.google.com · developers.google.com
  3. 3Wikipedia’s operator says a bad AI copyright ruling could threaten the open webThe Verge · theverge.com
  4. 4Cate Blanchett-Led Coalition Challenges OpenAI in New York Times Copyright Battlethewrap.com · thewrap.com
  5. 5House Democrats Criticize Justice Dept. Support of OpenAI in Copyright SuitThe New York Times · nytimes.com
  6. 6Apple confirms its AI features reach China in December, with QwenThe Next Web · thenextweb.com
  7. 7Google and Bing back one robots.txt rule to opt out of AI trainingppc.land · ppc.land
  8. 8Google, Anthropic and others to support new AI crawling opt-out standard - SiliconANGLESiliconANGLE · siliconangle.com
  9. 9Bing Blogs | The news, trends, people and technology behind Bingblogs.bing.com · blogs.bing.com
  10. 10The Wikimedia Foundation files to protect open knowledge in New York Times v OpenAI case – Wikimedia Foundationwikimediafoundation.org · wikimediafoundation.org
  11. 11Apple confirms iOS 27.3 beta is coming next week with Apple Intelligence in China9to5Mac · 9to5mac.com
  12. 12Apple 智能即将登陆中国apple.com.cn · apple.com.cn
  13. 13Apple Intelligence is finally coming to China with iOS 27.3The Verge · theverge.com
  14. 14Are We in an A.I. Bubble? We’re About to Find Out.The New York Times · nytimes.com
  15. 15How a Contractor’s Claude Agent Got Into the PentagonWired · wired.com
  16. 16Claude Can No Longer Produce A 'Recreation' Of Existing Gameskotaku.com · kotaku.com
  17. 17How to Use ChatGPT's Try On Button to Preview Outfits on Your Body Before Buyingkansas.com · kansas.com
  18. 18How to Use ChatGPT's Try On Button to Preview Outfits on Your Body Before Buyingheraldsun.com · heraldsun.com
  19. 19Cate Blanchett-Backed Group and Collection of Hollywood Stars File Brief in New York Times-OpenAI Lawsuit: Hollywood Has ‘Proven Infrastructure’ to License Contentvariety.com · variety.com
  20. 20Doximity Launches DoseCheck to Catch Dangerous AI Drug Dosing Errors Before They Reach Doctorspress.doximity.com · press.doximity.com
  21. 21Popular AI Models Make Potentially Lethal Dosing Errors: Doximity Benchmarktechtimes.com · techtimes.com
  22. 22TBC launches Mischief, an AI video model shaped by brain cellsThe Next Web · thenextweb.com
  23. 23AI Export Controls Foisted On China Come Back To The Haunt The U.S.Forbes · forbes.com
  24. 24AI Security Agents Compared: Claude, Codex, Mantis, Cloudflaremarktechpost.com · marktechpost.com
  25. 25Samaya AI Releases Sterling: A 3B-Active Finance Research Agent Model that Beats Claude Fable 5.1 on FrontierFinance at 11x Lower Costmarktechpost.com · marktechpost.com
  26. 26Databricks and SAP expand their deal to put AI on SAP dataThe Next Web · thenextweb.com
  27. 27AI-generated code is still frequently failing security checks — with Claude and GPT-6 topping the leader boardTechRadar · techradar.com
  28. 28Hackers hijacked a Claude Code extension, exposing yet another new AI security riskFast Company · fastcompany.com
  29. 29Edge0 Releases Audio8-TTS-Realistic, a 0.6B Voice Cloning TTS Modelmarktechpost.com · marktechpost.com
  30. 30Could Canva's newest AI feature end graphic design slop?Fast Company · fastcompany.com
  31. 31Meta AI and CMU Researchers Cut AI Jailbreak Rates by 22% With Pre-Deployment Training Data Filtertechtimes.com · techtimes.com

That’s the edition.

The next one is ready tomorrow by 6 AM ET.

Written with AI from the sources in the references above.

Ramit Koul
AuthorRamit KoulFounder, Software Engineer & Innovator