September 15, 2026, (Inside AI) — Cloudflare today launched a new Disallow AI Training setting that separates search indexing from AI model training for mixed-use crawlers, ending a long-standing tradeoff for website owners. The feature, available to all Cloudflare customers on every plan, lets sites remain discoverable in search while refusing to let the same crawler train on their content. Apple, Google, and Microsoft have either implemented or committed to honor the setting within specified timeframes.
The move addresses a structural problem: major internet companies operate crawlers that serve both search and AI training. Refusing one meant refusing the other. Cloudflare reports that fewer than 1% of its sites block search bots, but 17% already use some mechanism to block training. This disparity drove the need for granular controls.
Cloudflare classifies bots into three behaviors: Search, Training, and Agent. A mixed-use crawler does both Search and Training. The new Disallow AI Training setting publishes a preference in robots.txt, but Cloudflare’s network enforces it by identifying, classifying, and blocking non-compliant crawlers, then reporting operator behavior on Radar.
To qualify for Cloudflare’s new Accountable designation, bot operators must meet or commit to meeting requirements for publisher choice and transparency. Apple, Google, and Microsoft all qualify. Amazon, Anthropic, Meta, and OpenAI also separate their Search and Training crawlers, allowing Cloudflare to block training without affecting search.
Read: Cloudflare Acquires Ensemble AI Team to Slash Inference Costs
For Applebot, site owners can already opt out of training via a Disallow rule for “Applebot-Extended.” Apple has stated that disallowing training does not impact search ranking. Googlebot offers a similar opt-out for “Google-Extended” and a toggle in its webmaster portal to exclude content from generative search results. Microsoft’s Bingbot currently supports AI training preferences through a meta tag, and Microsoft is building support for a robots.txt “no training” preference at the domain level, targeted for early 2027. Until then, selecting Disallow AI Training will not automatically convey a no-training preference to Bing through robots.txt.
Existing Cloudflare domains that never configured granular controls will be migrated based on their legacy Block AI Bots setting. Those that previously configured Search/Training/Agent controls will have their selections preserved under the new definitions. New domains will be offered presets depending on whether the site earns money from advertising.
“Block” and “Block on pages with ads” now apply to all training crawlers, including mixed-use crawlers. Previously, blocking them could affect search discoverability. Disallow AI Training works by publishing a preference in robots.txt. An ads-only preference cannot be expressed that way because the list of ad-serving pages is too large and changes too frequently to enumerate.
Agents do not create the same search-discoverability tradeoff, and no well-established directive exists for expressing Disallow preferences to agents. Cloudflare is not including a Disallow setting for Agents for now, but will revisit as standards such as ai-prefs mature.
The company also outlined its next focus: AI Summaries. An opt-out for AI summaries is already a requirement for Accountable mixed-use crawler operators. By early next year, Cloudflare aims to let site owners control how much of their content is included in summaries, set once on Cloudflare rather than with each operator separately.
Data illustrates the mixed impact of summaries. More than half of consumers read summaries in search, and those consumers are over 40% more likely to end their search after reading one. However, consumers referred by AI search convert at between three times and over five times the rate of those referred by traditional search. AI may produce fewer visits while sending customers with greater intent.
Read: Apple Sues OpenAI for Stealing Unreleased Hardware Secrets in California
Cloudflare’s role, the company says, is not to choose for publishers but to provide visibility and control. The new controls are available now in the domain Security Settings. Cloudflare will continue engaging with crawler operators and tracking their controls, transparency, and reporting on Radar.
