Choose which AI crawlers your business wants to admit
Separate training, search and user-requested visits before blocking AI traffic. A site that sells visibility needs a different policy from one selling content.
By 
Paul. Software business
A business website exists for a reason. If its job is to help people discover a service, a blanket block on AI traffic can work against that purpose. If its content is the product, restricting collection may be commercially important. My recommendation is to choose access by purpose and site section, then check the effect. A single label, “AI bot,” is too broad to make the commercial decision.
That recommendation does not promise traffic from AI answers or claim that blocking a crawler always removes a site from them. Access is one condition a publisher can control. Inclusion, attribution and visits depend on systems outside the publisher’s control, and their value needs measurement.
The 2024 switch and today’s distinctions
On 3 July 2024, Cloudflare announced one-click blocking of AI bots, scrapers and crawlers. It presented an enforcement option for site owners concerned about collection of their work. The announcement made the choice more accessible, but a convenient switch cannot determine what kind of access serves a particular business.
Looking back from November 2026 requires a second distinction: the original announcement is not a guide to the current dashboard. Cloudflare’s current documentation, updated 2 November 2026, describes separate Search, Agent and Training behaviors. Training includes some mixed-purpose crawlers. Do not read that later configuration back into the July 2024 launch.
For a current provider-specific example, OpenAI’s crawler documentation distinguishes GPTBot, which collects material that may be used for model training, from OAI-SearchBot, which serves search discovery. It also describes ChatGPT-User as handling some user-initiated visits, with different treatment of robots rules. These are current documented roles, not a claim that every name or behavior existed when Cloudflare made its announcement.
A request and a blocked connection do different jobs
A robots.txt policy asks cooperating crawlers to follow rules. It does not authenticate a person or prevent a non-cooperating program from requesting a public page. RFC 9309, which defines the protocol, explicitly separates its rules from access authorization. Private documents therefore need actual access controls rather than a disallow directive.
An edge rule can deny matching requests before they reach the site’s origin. Its effectiveness depends on which requests it recognizes and where traffic enters. It is not proof that every scraper is excluded, nor does a new block remove copies already collected elsewhere. Evaluate the intended boundary instead of describing the setting as complete protection.
There is also a practical interaction: a permissive robots rule cannot make a request succeed if another layer blocks it. When checking access, consider the complete path through bot rules, other firewall controls and the origin application. A text-file change alone is not a verified policy change.
Decide by the value of the page
For an illustrative local consultancy, public service pages help prospective customers understand the offer. Allowing a search crawler may support that aim, while permission for training is a separate decision. Keeping search access open does not oblige the owner to approve every form of collection. It also does not guarantee a recommendation or a citation.
For an illustrative shop, product discovery matters, but automated collection can impose costs or expose information the owner would rather limit. Public catalogue pages and account pages require different treatment. A crawler policy should not become an excuse to leave customer details publicly accessible. Protecting those details is an application-access requirement.
A paid research publication has a different bargain: the text itself is sold. Restricting collection can be rational even if it reduces some discovery opportunities. The publisher may choose to expose summaries while protecting full material through authentication. That is a commercial design choice, not a claim about what copyright law automatically permits or prohibits.
Documentation sits between these examples. A company may want answers to point users toward its public instructions, while keeping customer-specific implementation notes private. Choose the accessible set deliberately. All these scenarios are illustrative; they do not describe AL+ customers or measured outcomes.
The strongest argument for blocking broadly
A small publisher may lack the time to track changing crawler identities. It may judge uncertain attribution and infrastructure costs more important than speculative visibility. A broad block can be a reasonable default in that situation, especially when content is the business’s main asset. Selective access has a real maintenance cost.
The counterargument is strongest for a brochure site whose essential information is already public and whose owner wants it discovered. Blocking search-oriented access there can remove an opportunity without protecting an exclusive product. That is why selective policy is my starting recommendation for such sites, subject to evidence about their actual traffic and costs.
Do not infer business value from bot-request totals alone. Thousands of crawler requests may produce no useful enquiry. Conversely, a small number of attributable visits may matter to a specialist supplier. Without referral and enquiry evidence, the commercial effect remains unknown.
Make the policy reversible and reviewable
Write a short record of which content should be discoverable, which collection purposes are permitted and who maintains the rules. Check representative requests against the effective configuration and review relevant logs where available. Verify crawler identity using the provider’s documented methods rather than trusting a self-declared user-agent string alone.
Current controls have qualifications. Cloudflare’s policy documentation describes different behavior for mixed-purpose crawlers and for voluntary directives versus edge blocking. Read the applicable mode before assuming that “disallow” and “block” mean the same thing. No account setting was changed or tested for this article.
Include this policy in website maintenance if someone else runs the site. Ask for a clear account of intended access and evidence that the rules implement it. The business should choose the trade, then keep enough evidence to reconsider it.