The DeMicco Digest
Grab your headphones and enjoy a mini-podcast version of this blog. Sit back and listen while we walk you through the key points!
You spent the last two years making your website readable to AI. Now your server logs show that most of your visitors are machines, and nobody told you which ones to let in.
For most of the past two years, I have been telling industrial companies to make their websites legible to machines. Structured data, clear entities, specifications as text rather than images, the whole infrastructure I laid out in March. The argument was that buyers now ask answer engines, and answer engines cite the sources they can read. I stand by it. But there is a consequence I did not spend enough time on, and it is now showing up in server logs, including at industrial companies that never expected to be interesting to anyone.
Once a site is easy for machines to read, machines read it. All of them. The search crawlers you want, the AI crawlers you probably want, the AI crawlers you may not want, and a long tail of unidentified scrapers that pull your entire product catalog every night from data centers on three continents and send exactly nothing back. A site that was built to be cited is also a site that is easy to copy, and the same openness serves both.
The reaction I see most often is to treat all of it as an attack and block aggressively. That is the wrong instinct for an industrial supplier, and this post is about why, and what to do instead.
Table Of Contents
How much of your traffic is not human
More than half, if you are typical. Imperva’s annual bot report found that automated traffic accounted for a majority of all web traffic in 2024, the first time in a decade it exceeded human traffic, and the share attributed to AI crawlers and scrapers has been rising since. Your own logs will tell you the real number for your site, and I would ask your developer or your hosting provider for it before reading further, because the rest of this only matters if you know what is actually hitting your server.
The other number worth knowing is what those crawlers give back. Cloudflare, which sits in front of a large share of the web, published crawl-to-referral ratios in mid-2025: for every page an AI company’s crawler fetched, how many visits did that company’s product send back to the site. Google’s search crawler returned roughly one visit for every fourteen pages crawled. The AI crawlers ran into the thousands of pages per referral, and some into the tens of thousands. That is not a scandal. It is how answer engines work: they read a great deal and send traffic rarely, because the answer is delivered in place. But it means you should never evaluate AI crawling by the traffic it returns. It will look like theft by that measure, even when it is doing exactly what you built the site to invite.
Sorting the machines
The mistake is treating “bots” as one category. There are at least four, and they deserve different answers.
| Category | Examples | What they do with your content | Default answer for an industrial supplier |
|---|---|---|---|
| Search crawlers | Googlebot, Bingbot | Index for search results, including Google’s AI Overviews, which draw on the ordinary search index | Allow. Blocking these removes you from search. |
| Answer-engine and user-agent crawlers | OpenAI’s OAI-SearchBot and ChatGPT-User, PerplexityBot, and similar fetchers that retrieve pages to answer a live question | Read your page to answer a buyer’s question, often with a citation | Generally allow the declared ones. This is the visibility you built for. Enforce at the edge regardless, since only a declared identity can be governed by robots.txt. |
| Training crawlers | OpenAI’s GPTBot, Anthropic’s ClaudeBot, Apple’s Applebot-Extended, Common Crawl’s CCBot, Google’s Google-Extended token | Collect content to train or ground models | A business decision. For public specifications and guides, allowing is usually harmless and may help. For proprietary technical content, consider blocking. |
| Unidentified and aggressive scrapers | Anything without a declared identity, or a declared one that ignores your rules; high-volume fetches from data-center ranges | Copy catalogs, pricing, and documentation; sometimes probe for vulnerabilities | Rate-limit and challenge at the edge. Block persistent offenders. |
Each of the named crawlers is documented by its operator, with the user-agent string it uses and the robots.txt instruction that controls it. Your developer can look those up in an hour. The documentation changes, so this post will not list the directives; it will tell you what they can and cannot do.
Three things your developer needs to know, and most do not
robots.txt is a request, not a lock
The file that tells crawlers what to avoid is honored by the well-behaved and ignored by the rest. Google, Bing, and the major AI companies say they respect it. Compliance has been uneven and, in at least one publicized 2025 dispute between an infrastructure provider and an answer-engine company over undeclared crawling, openly contested. The scrapers that concern you never read it at all. If you need something actually protected, protection happens at the server or the edge, through authentication, rate limiting, and bot management, not in a text file.
Blocking Google-Extended does not remove you from AI Overviews
This one causes real damage when it is misunderstood. Google-Extended is a token you can add to robots.txt to keep Google from using your content to train and ground its Gemini models. It has no effect on Google Search, and Google’s AI Overviews and AI Mode draw on the ordinary search index built by Googlebot. So an owner who says “block the AI” and gets Google-Extended blocked has changed nothing about AI Overviews, and an owner who wants out of AI Overviews cannot get there through robots.txt without also leaving Search. Google says this plainly in its crawler documentation. Decide what you actually want before anyone edits the file.
llms.txt is a proposal, not a standard
You may have been told to add an llms.txt file to help AI systems understand your site. It is a reasonable idea from 2024 that has not been adopted. John Mueller of Google’s Search team has said publicly that Google does not use it, and no major AI company has confirmed that it does. It will not hurt you. It will not help you either, and a vendor who presents it as a ranking factor for AI is telling you something about their diligence. The same caution applies to newer robots.txt extensions that some infrastructure providers have introduced to signal permissions for AI use: they express a preference, and Mueller has said Google’s crawlers do not act on them.
What actually needs protecting
Here is where an industrial company should spend its energy, because the question is not “how do I stop the bots” but “what would it cost me if this were copied.”
- Pricing and quotes. If your prices live on public pages, competitors’ scrapers already have them. Anything with real commercial sensitivity belongs behind a login or a quote request, where a crawler cannot go.
- Customer and distributor portals. Authenticated, always. This should not need saying, but it is not unusual to find a portal that was merely unlinked, which a crawler treats as an invitation.
- Proprietary technical documentation. Engineering data you consider a competitive asset, as opposed to the specifications a buyer needs to select the product. The first goes behind a gate. The second stays public, because that is what gets you cited.
- Your server’s ability to serve humans. A scraper pulling forty thousand pages an hour can slow the site for the engineer trying to download a spec sheet. Rate limiting at the edge is a performance fix as much as a security one.
Everything else, the product pages, the application guides, the FAQs, the case studies, should be as open and as machine-readable as you can make it. That content exists to be found, and an AI answer engine quoting your torque specification to a buyer who then requests a quote is the outcome the whole investment was for.
A practical policy, in six steps
- Get the bot share from your logs or your CDN. Total, by declared identity, and by origin. If you are behind a content delivery network, the dashboard has this; ask for it.
- Sort every identified crawler into the four categories above. Write the decision down per crawler. This is a one-page policy, and it is the document your developer edits robots.txt from.
- Check your CDN’s default settings for AI crawlers. Cloudflare in particular began blocking AI crawlers by default for new customers in mid-2025 and has announced further default changes for later this year. A default that was set without you may be blocking the answer-engine crawlers you want. Look before you assume.
- Move anything commercially sensitive behind authentication. Not behind robots.txt. Behind a login.
- Rate-limit and challenge the unidentified. Your CDN or web application firewall can throttle high-volume fetches from data-center ranges and challenge clients that do not behave like browsers. Persistent offenders get blocked. Declared, well-behaved crawlers do not need this treatment.
- Measure citations, not referrals. Track where your company appears in AI answers for the questions your buyers ask. That is the return on machine-readable content. Referral traffic from AI will stay small, and judging the strategy by it will lead you to block the thing that is working.
The trade you are actually making
An industrial supplier’s public technical content is not the crown jewels. The crown jewels are the tolerances you hold, the lead times you keep, the engineers who answer the phone, and the relationships that took a decade to build. The specification sheet is an advertisement for all of that, and it does its job when it is read, by a person or by the machine that person asked. Letting it be read widely costs you very little. Locking it away costs you the buyer who never found you.
Sort the bots. Protect what is worth protecting. Then let the rest of them read. They are the audience you built the site for, even if they will never fill out a form.
Sources & Further Reading
The following resources informed this article. All claims have been verified against multiple sources where possible.
- Google Search Central, “Overview of Google crawlers and fetchers” (Google-Extended section) – https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
- Google Search Central, “Introduction to robots.txt” – https://developers.google.com/search/docs/crawling-indexing/robots/intro
- MIT Technology Review, “Cloudflare will now, by default, block AI bots from crawling its clients’ websites,” July 1, 2025 – https://www.technologyreview.com/2025/07/01/1119498/cloudflare-will-now-by-default-block-ai-bots-from-crawling-its-clients-websites/
- Cloudflare blog, crawl-to-referral ratio analysis (June/July 2025) – Link Cloudflare’s July 2025 blog post on AI crawler crawl-to-referral ratios (search the Cloudflare blog for “crawl-to-refer ratio”); cite Cloudflare Radar data
- Imperva, “Bad Bot Report” (2025 edition) – https://www.imperva.com/resources/resource-library/reports/bad-bot-report/ (verify the 2025 edition figure before publishing)
- Search Engine Roundtable, “Google: Cloudflare Content Signals Robots.txt Directive Has No Effects,” July 6, 2026 – https://www.seroundtable.com/google-cloudflare-content-signals-41631.html
- OpenAI, Anthropic, Perplexity, Apple, Common Crawl crawler documentation – Link each operator’s current crawler documentation page; do not hard-code directives in the article
Joe DeMicco is the Founder and President of Amplify Industrial Marketing + Guidance (AIMG), a full-service B2B industrial marketing agency with 30+ years serving manufacturers, fabricators, distributors, and trade associations. He is also the founder of Industrial Web Search (IWS), a keyword-driven B2B supplier directory platform. Joe teaches the marketing module for the Goldman Sachs 10,000 Small Businesses program and writes about AI, marketing, and B2B strategy at demicco.com.
