Getting access to a generative AI model is the easy part. A subscription or an API key takes an afternoon, and the demo almost always impresses everyone in the room.
The hard part starts after the demo. The model has to work on your data, inside your systems, under your legal obligations, and with your staff actually using it. That is where most projects slow down, and seven challenges come up again and again.
None of them is a reason to avoid the technology. They are, however, the difference between a pilot that impresses for a month and a system the business can rely on for years. Teams that name these challenges at the start can budget for them, staff for them and design around them. Teams that discover them one by one tend to lose momentum, and sometimes lose the project.
Your data is not ready for it
Generative AI is only as good as the information it can reach. In many organizations that information is scattered across shared drives, old databases, email archives and scanned documents, with duplicates, contradictions and outdated versions sitting side by side.
Access is the other half of the problem. A model connected to company data will happily surface anything it is allowed to read, including files that were never meant for a wide audience. Before any rollout, someone has to decide which sources the system may use and which stay off limits.
The practical fix is unglamorous data work. Audit the sources you plan to connect, clean the worst duplicates and errors, and set permissions before you connect anything. We covered the data side in depth in our guide to the data challenges generative AI faces, and it is the challenge teams most often underestimate.
The compute bill grows faster than expected
Large models are expensive to run. Every question a user asks costs processing time on powerful hardware, and a tool that seems cheap in a pilot with fifty users can become a serious line item when five thousand people use it all day.
Training or heavily fine-tuning your own model costs far more again, which is why most organizations adapt an existing model instead. Costs also hide in unexpected places, such as storing conversation logs, running evaluations and keeping a test environment alive.
The fix is to match the tool to the task. As IBM’s overview of generative AI challenges points out, smaller and cheaper models handle routine work like classification and extraction perfectly well, and retrieval based approaches often beat expensive fine-tuning. Reserve the biggest models for the jobs that genuinely need them, and track spending per use case from day one.
Usage based pricing makes discipline harder, because enthusiastic users can run up a bill simply by using the tool as intended. Set budgets per team, alert the owners when a use case passes its expected cost, and review the most expensive workflows monthly. A use case that only makes sense at pilot scale is a finding worth having early, while you can still redesign it.
Your existing systems were not built for this
A generative AI tool rarely lives on its own. It has to pull records from your customer database, hand results to your help desk software and respect the login rules your IT team set up years ago. Many of those older systems have no modern way to connect, and some cannot respond fast enough for a live conversation.
Integration work is slow because it touches everything. Each connection needs testing, security review and a plan for what happens when the AI service goes down in the middle of a working day.
A phased approach works better than a big launch. Connect one system at a time, starting with read only access, and use standard connectors and APIs where they exist. A small central platform team that owns the shared connections stops every department from building its own fragile version.
The people who can do this work are scarce
Running generative AI well needs a mix of skills that few teams already have. You need engineers who understand models and data pipelines, plus people who can evaluate output quality, design prompts and spot when the system is confidently wrong.
Hiring all of that in a tight market is slow and expensive. Meanwhile, your existing staff know the business but may never have worked with probabilistic systems that give a different answer each time.
Most organizations close the gap from inside. Train the people you have, starting with the teams closest to the first use cases, and borrow specialist help for the early design decisions. A small central team supporting many departments spreads scarce expertise further than scattering it.
Governance and compliance cannot wait
Generative AI creates records, decisions and customer facing text, and all of it falls under rules you already live with. Privacy law governs the personal data in your documents, and newer rules such as the EU AI Act add duties around transparency and high risk uses.
Someone also has to answer for the output. When a generated answer misleads a customer or leaks information it should not have, “the model did it” is not an acceptable explanation. Clear ownership, approval gates for sensitive uses and logs you can audit are part of the implementation, not paperwork for later.
This is where fairness belongs too. A system that treats some customers worse than others is a governance failure long before it becomes a headline, and one challenge in ensuring fairness in generative AI shows how early that risk enters a project. Our article on the responsibility of developers using generative AI sets out who should carry which part of that duty.
The model will sometimes make things up
Generative models predict plausible text. They do not check facts before answering, so they can produce a confident, well written response that contains invented figures, wrong dates or policies that do not exist. This is known as hallucination, and no current model is free of it.
The risk depends on the job. An invented detail in a brainstorming draft is harmless, while the same habit in a medical summary, a contract clause or a customer refund answer is a real problem.
You manage this rather than eliminate it. Connect the model to verified sources so answers are grounded in your documents, test it against a fixed set of questions before release, and keep a human reviewer on anything high stakes. Routing low confidence answers to a person instead of the customer is a simple control that prevents most of the damage.
Your staff have to want to use it
The last challenge has nothing to do with technology. People who did not ask for a new tool, do not trust it, or quietly fear it is there to replace them will find ways not to use it. A technically sound rollout can still fail at this human step.
Change management is therefore part of implementation. Involve the eventual users while you are choosing use cases, be honest about what the tool will and will not change in their jobs, and train people on real tasks from their own work rather than generic demos.
Early wins matter more than launch speeches. One team saving a visible amount of time each week convinces neighbouring teams far better than a company wide announcement.
Tackle them in the right order
The challenges interact, so the order you address them matters. Start with one narrow use case that has an owner, a measurable benefit and forgiving stakes if the output is imperfect. Prove the pattern there before you widen it.
Fix the data for that use case only, rather than launching a company wide data cleanup that never finishes. Put the governance gates, cost tracking and human review in place while the audience is small, because retrofitting them onto a popular tool is politically much harder.
Finally, treat evaluation as a permanent habit rather than a launch task. Models get updated, data changes and users find new prompts, so the testing you did in month one says less and less about the system by month six. The organizations that keep testing are the ones still happy with their implementation a year later.
Frequently Asked Questions
Why do so many generative AI pilots never reach full use?
Pilots usually run on clean sample data with a small, friendly group of users. Full use exposes the data quality, integration, cost and governance problems described above, all at the same time. Projects that plan for those problems during the pilot are the ones that survive contact with the whole organization.
What is the biggest challenge for a small company?
Cost discipline and skills, in that order. A small company cannot afford to experiment with the largest models everywhere, and it rarely has spare specialists. The workable route is a hosted model, one or two well chosen use cases and training for existing staff instead of new hires.
Can hallucinations be stopped completely?
No. Current models generate plausible text rather than verified facts, so the risk can be reduced but not removed. Grounding answers in your own documents, testing before release and keeping human review on important outputs are the accepted controls.
How long does implementation usually take?
A narrow first use case can be live in weeks. Connecting company data properly, clearing governance review and training staff usually stretches a serious rollout across several months. The technology is rarely the slowest part; the data and the approvals are.

Leave a Reply