The DeMicco Digest
Grab your headphones and enjoy a mini-podcast version of this blog. Sit back and listen while we walk you through the key points!
An AI assistant on a website will, sooner or later, answer a visitor’s question by sending them somewhere you did not intend. Here is what actually breaks, how to keep it from embarrassing you, and why the right answer is governance rather than retreat.
Two decisions I see companies make, sometimes in the same month. In the first, a company approves an AI search tool for its website with almost no debate, because the existing site search is so bad that anything would be an improvement, and because the search logs will finally tell them what buyers are actually asking for. In the second, a company turns off the conversational half of its AI assistant after it answers a visitor’s question by linking to a competitor of the very company the visitor had been reading about. The summary feature stays on. The chat does not.
Both decisions are correct. That is the point of this post. The failure in the second case is not a reason to abandon AI on the website. It is a reason to understand what actually went wrong, which is the same handful of things that go wrong every time, none of them about the model.
Table Of Contents
Two products, one label
When a vendor says “AI search,” ask which of two things they mean, because the risk profiles are not close.
The first is retrieval. The tool indexes your content, understands that “pressure relief valve” and “safety valve” might be the same query, ranks results by relevance rather than keyword match, and may summarize the top hits with links. It replaces the box on your site that currently returns nothing for a part number typed with a hyphen. This is a real improvement for almost every industrial website I have seen, and the risk is low, because everything it shows is anchored to a page you wrote and the visitor can click through to check. A summary can still misstate a page, so summaries should carry their source links, but the failure is small and visible rather than confident and untraceable.
The second is generation. A conversational assistant that reads your content and composes an answer in its own words. This is what the demo shows, because it is impressive. It is also where every public failure comes from, because the assistant is now speaking for the company, and it will speak with equal confidence whether it is right, wrong, or reading the wrong document.
Most industrial companies should turn on the first immediately and treat the second as a governed launch with a kill switch. Most vendors bundle them and switch both on.
What actually fails
I have watched enough of these launches to say that the failures cluster. They are not exotic. They are the same five things.
The corpus was never defined
The assistant was pointed at “the website.” But the website contained a supplier directory, a partner listing, a news feed that quotes other companies, and a resource library with third-party content. The model treated every document as the company’s own voice. When a visitor asked for a supplier of a particular capability, the assistant found one in the directory and recommended it, because nobody had told it that directory entries were not endorsements. That is the competitor link, and it is entirely predictable once you see the corpus.
Ranking let the wrong content win
A page about another organization, rich in text, outranked the company’s own two-paragraph service page for the query the company most wanted to own. Retrieval is relevance-driven, and relevance is about text, not about who you are. If your own pages are thin, the assistant will find something thicker.
No citation requirement
An answer without a source cannot be checked by the visitor and cannot be traced by you. The Air Canada chatbot I will come back to gave a customer an answer that contradicted the airline’s own policy page, and the customer had no way to see the discrepancy until it cost him. Citation is not a nicety. It is how a wrong answer gets caught before it costs money.
Product queries treated like editorial queries
Someone asks for a part by number, or a product by specification, and the assistant writes three thoughtful sentences instead of returning the product. Product search and content search are different problems with different ranking logic, and a tool that is good at one is usually mediocre at the other until they are separated.
Nobody read the transcripts
Every one of the failures above was visible in the conversation logs within days of launch. Nobody was assigned to read them. The problem surfaced when a visitor complained, or worse, when the visitor was a customer.
The precedent every owner should know
In February 2024, the British Columbia Civil Resolution Tribunal decided Moffatt v. Air Canada. In late 2022 a customer had asked the airline’s website chatbot about bereavement fares, was told he could apply for the discount retroactively within ninety days, bought a full-fare ticket on that basis, and was then refused because the airline’s actual policy, published elsewhere on the same site, did not allow retroactive claims. Air Canada argued it could not be held liable for what its chatbot said. The tribunal member called that submission “remarkable,” found the airline liable for negligent misrepresentation, and ruled that the chatbot was simply part of the airline’s website, that the company was responsible for all the information on it, and that it made no difference whether the information came from a static page or a chatbot. The airline paid.
Two things make this ruling more useful than its modest dollar amount suggests. The chatbot in question dates from 2022, before the current generation of AI assistants, and the decision does not turn on what kind of software it was. It turns on the website. That means the principle already covers the assistant you are about to launch, however it is built. The second is the ordinary logic of the finding: a visitor reasonably relies on what a company’s site tells them, and the company owns the consequences. For a manufacturer, the equivalent of a wrong refund policy is a wrong pressure rating, a wrong lead time, a compatibility claim the engineer relied on. Plan for it before launch, not after.
Why it is still worth doing
If the previous sections read as a warning, here is the other side, and it is the side I come down on.
Site search on industrial websites is broken almost everywhere. Buyers arrive with part numbers, synonyms, misspellings, and questions phrased in their own plant’s vocabulary, and the keyword box returns nothing. Every zero-result search is a buyer you had and lost on your own property. Retrieval-based AI search fixes most of that, and it fixes it without composing a single sentence on your behalf.
The second reason is the one the company in my first conversation understood. The search logs are a record of what buyers want, in their own words, at the moment they wanted it. No survey gets you that. No agency brief gets you that. Six months of AI search logs will tell you which products people cannot find, which questions your site does not answer, and which vocabulary your buyers use that your engineers do not. That is the content plan, and it is also the list of questions you want answer engines to cite you for. I have written about that infrastructure before; the search log is where you learn what to build with it.
The third reason is that the failures are governable. Every one of the five above has a fix that a marketing manager and a developer can implement in a week. That is not true of most technology risks.
Governance before launch: the checklist
- Define the corpus, in writing. First-party content by default. Directories, partner listings, partner profiles, and third-party resources are excluded or, if included, labeled so the assistant knows they are not the company’s voice. This one decision prevents the competitor link.
- Pin the answers you cannot afford to get wrong. For the fifty queries that matter most, your core products and services, specify which page should rank first. Good tools allow curated results. If the vendor’s does not, that is a finding.
- Retrieval first, generation second. Launch with ranked results and links. Enable composed answers only when citations are mandatory on every response and the assistant declines to answer when it cannot cite.
- Set the off-site policy. The assistant links to your pages, and to nothing else unless you list it. No exceptions the tool invents on its own.
- Separate product search from content search. A part number or specification query goes to a product index with its own ranking. A how-to question goes to content. If the vendor cannot separate them, ask how they handle a part number.
- Build a test set before you switch anything on. Fifty real questions from sales and customer service, each with the answer and the page it should come from. Run it at launch and every month. A tool that cannot pass your own test set should not be talking to your customers.
- Assign the transcript reader. One person, weekly, for the first quarter, reading every conversation. Then monthly. This is where you catch the next failure mode before a customer does.
- Install the kill switch and name who can pull it. The conversational layer should be disable-able in minutes by someone in marketing, without a developer or a vendor ticket.
- Disclose. Tell the visitor they are talking to an automated assistant and where to reach a person. The tribunal’s reasoning turned on a visitor reasonably relying on what the site said; make the reliance informed.
- Baseline the old search first. Zero-result rate, search-to-inquiry rate, top hundred queries. You cannot show the new tool is better without knowing what the old one did.
One more item that belongs to the transcript reader and to whoever owns your privacy policy. Conversation logs contain what visitors typed, which will include names, email addresses, and sometimes questions a customer would not want repeated. Treat the logs as customer data: covered by the privacy policy, held under a retention rule, and never pasted into a third-party tool for analysis without the same review any customer data would get. I wrote about that discipline last winter, and an AI assistant’s logs are the newest place it applies.
The one test that exposes a weak vendor
Take a general-purpose AI model, the kind anyone can use, and instruct it to answer questions using only your website. Ask it the same twenty questions you asked the vendor’s tool. If the general model, with nothing but your domain and no special engineering, gives better answers than the product you are paying for, the vendor’s retrieval is broken, and no amount of tuning the conversational layer will fix it. I have seen this test end a vendor conversation, and it should. Retrieval is the whole value. A product that retrieves worse than a free tool is not an AI search product. It is a chat window.
What good looks like a year in
A search box that returns the right product for a hyphenated part number. A results page with a two-sentence summary and links, and, for the questions the company has chosen to answer conversationally, a cited paragraph that declines when it does not know. A monthly report showing zero-result rate falling, search-to-inquiry rate rising, and a list of the ten questions the site still cannot answer, which becomes next month’s content. A transcript log someone has read. A kill switch nobody has needed.
The failure in the second decision I described is real when it happens, and turning the chat off is the right call that week. It is also the beginning of the tool working, because it forces someone to write down what the corpus is, what the ranking should favor, and who is reading the logs. That is not a retreat from AI on the website. That is the part of the project that should have come first.
Sources & Further Reading
The following resources informed this article. All claims have been verified against multiple sources where possible.
- Oldroyd, McElheran, Elkington, “The Short Life of Online Sales Leads,” Harvard Business Review, March 2011 – https://hbr.org/2011/03/the-short-life-of-online-sales-leads
BYU ScholarsArchive record of the same article – https://scholarsarchive.byu.edu/facpub/9711
- Gartner, “The B2B Buying Journey” – https://www.gartner.com/en/sales/insights/b2b-buying-journey
- Lead Response Management Study (Oldroyd / InsideSales.com, 2007) – leadresponsemanagement.org (verify the domain still resolves before linking)
Joe DeMicco is the Founder and President of Amplify Industrial Marketing + Guidance (AIMG), a full-service B2B industrial marketing agency with 30+ years serving manufacturers, fabricators, distributors, and trade associations. He is also the founder of Industrial Web Search (IWS), a keyword-driven B2B supplier directory platform. Joe teaches the marketing module for the Goldman Sachs 10,000 Small Businesses program and writes about AI, marketing, and B2B strategy at demicco.com.
FAQ
What is the difference between AI site search and an AI chatbot?
AI site search uses retrieval to rank and summarize a site’s own pages, returning links to existing content. An AI chatbot or conversational assistant composes new answers in its own words. Retrieval carries little risk because everything it shows is a page the company wrote; the conversational layer is where errors such as recommending competitors or inventing policies occur, and it should be governed before launch.
Why would an AI site assistant recommend a competitor?
Is a company liable for what its website chatbot says?
In Moffatt v. Air Canada (2024 BCCRT 149), the British Columbia Civil Resolution Tribunal found the airline liable for negligent misrepresentation by its website chatbot, holding that a chatbot is part of a company’s website, that the company is responsible for the information on it, and that it makes no difference whether the information comes from a static page or a chatbot. The chatbot predated current AI assistants, and the reasoning applies regardless of the technology. Companies should treat assistant output as their own published statements and govern it accordingly.
What should be checked before launching AI search on a website?
How do I know if an AI search vendor's product is any good?
Compare it against a general-purpose AI model instructed to answer using only your website. If the general model answers your test questions better than the vendor’s product, the vendor’s retrieval is weak. Retrieval quality, not the conversational interface, determines whether the product is worth buying.
