Before you buy AI, fix the data it would have to read
Most disappointing AI pilots are not model failures. The assistant was pointed at an environment where the answer genuinely was not available, and it did the only thing it could.
The pilot goes in, people are impressed for a fortnight, and then usage falls off a cliff. The conclusion drawn is that the technology is overhyped.
Sometimes it is. More often the assistant was asked questions its inputs could not answer, and produced confident approximations instead, which is exactly what happens when you ask anyone to summarise a document set that contradicts itself.
What the tool inherits
An assistant over your own content inherits everything about that content.
If there are four versions of the pricing sheet in three locations, it will find the wrong one, and it will not know it is wrong. If access control is loose, it becomes very good at surfacing things people were never supposed to see, quickly and in a nicely formatted summary. If nobody has retired the 2019 policies, they carry the same weight as the current ones.
That last point deserves emphasis, because it is the failure that generates complaints rather than shrugs. Search tools were bad enough at finding sensitive material that permissions problems stayed hidden. An assistant is good at it. A tenancy-wide rollout is, among other things, an audit of your permissions that you did not ask for and will not enjoy.
The unglamorous work that makes it worth having
Sort out permissions first. Not as a phase two. Before rollout, know what “everyone” has access to, and reduce it. Oversharing that was theoretical becomes practical the moment the tool can read.
Retire the duplicates and mark what is current. One authoritative version of the documents that matter, and the old ones out of scope. This is librarian work and it produces most of the value.
Decide where the data may go. Whether content leaves your tenancy, whether it trains anything, and which classes of information are simply not allowed near a tool, are decisions to make in advance and write down. For a firm with confidentiality obligations this is not optional.
Pick a task, not a capability. “AI for the business” produces a pilot nobody owns. “Draft first-pass responses to inbound RFI questions” or “summarise site reports into a weekly exception list” produces something you can judge in a month.
Where AI is genuinely good in an operational business
Setting aside the assistant-over-documents case, the wins we would point at are narrower and more boring than the marketing.
Extraction from unstructured input, such as supplier invoices, dockets, or emailed orders arriving in twelve different formats, into something a system can accept. Classification and triage, so the queue arrives pre-sorted. Drafting the repetitive document that a person then edits, rather than the finished article. Summarising long records into the exception a human should look at.
The pattern in all of those is that AI does the shapeless part and a system or a person does the decision. The pilots that disappoint tend to have it the other way around.
And a governance point
Whatever you deploy, staff are already using something. The realistic choice is not whether AI enters the business but whether it does so with a sanctioned tool and a written position, or through a personal account with client material pasted into it.
A short, readable AI use policy is worth more than a long one nobody reads: what is approved, what is never to go near it, and who to ask. That takes an afternoon and it prevents the incident that is otherwise waiting to happen.
The technology is real. The preparation is where the value is decided.