The guide to buying AI for care and support providers was written by our Director of Data and AI, Sudha Regmi. Sudha has 15 years of experience in AI. Prior to Nourish, she spent 7 years at an international consulting company, helping large and midsize companies across the world design and implement their Data & AI strategy.
Everyone and their brother wants to sell you AI at the moment. Which means buying AI is fast becoming a mandatory skill for decision makers in care and support services.
That was the conversation behind my session at the Nourish Partnership Conference. There is plenty to be excited about: helping teams review care information, identify concerns and spend less time on administration.
Before joining Nourish, I spent years as a Data & AI Strategist, advising clients on technology decisions, including whether to build or buy. Those conversations always started with understanding the requirement. Working in social care has made the importance of that particularly tangible.
For many providers, buying an established solution will make more sense than building and maintaining their own. But advances in generative AI have made it much easier to build a convincing prototype. Developing a robust solution that people can rely on in a care service remains considerably harder. A polished demonstration tells you very little about the validation and engineering underneath it. So, beware the shiny demo!
I work closely with my colleagues at Nourish, such as our Director of Partnership, Denise Tack, and Head of Legal & Compliance, Jess Osmond, in evaluating AI vendors in the market, either for our use, or while advising our customers on choosing the right AI vendors for them. Here are the six questions we typically ask while evaluating.
The Six Questions
1) What are we actually asking AI to do?
Many care providers want to use AI in care planning. That can mean quite different use cases, potentially involving different AI techniques.
You might want help turning a carer’s observations into a clear draft. You might want to identify inconsistencies in a plan, or bringing together evidence that someone’s needs are changing. Each requires different information, evaluation and professional involvement.
Being clear about the task makes it easier to choose the right product. If you understand your objective, it is easier to understand what AI applications you need. Who will use the output? What will they do with it? What would improve for the service or the person receiving care?
The intended use also matters when assessing regulatory requirements. The AI and Digital Regulations Service’s social care guide directs providers towards Medicines and Healthcare products Regulatory Agency (MHRA) guidance on intended purpose and whether a product qualifies as a medical device. Ask suppliers to explain which requirements apply to their product and your proposed use.
2) What information does AI need, and where should it live?
For AI to be useful in care, it needs a strong foundation of data underneath it. When buying AI, even a capable model can produce poor results if the information it receives is incomplete, outdated or missing the relevant context.
Consider understanding a change in someone’s needs. The evidence might sit across daily observations, their preferences and previous reviews. The system needs to connect those records correctly and distinguish a recent development from an old concern. That context must stay current as care continues.
This is where data gravity matters. As information accumulates in a system, moving it elsewhere, as well as preserving the context that makes it useful, becomes more complex. There is a practical reason to bring AI closer to where care information already lives and is maintained.
An integration can connect systems, but someone still has to keep records current, preserve permissions and handle changes. In practice that costs more than just you initial setup fee when buying AI. It creates ongoing responsibility, which in turn creates ongoing cost.
AI within an established care platform can build on existing data foundations and workflows. Specialist partners can add valuable capabilities too. In either case, ask how the supplier maintains the quality of that context and brings findings back into the work of the care team.
3) What happens when AI gets something wrong?
A care-plan summary can read beautifully while leaving out an important detail. An audit review can produce sensible findings while missing the vulnerability you most needed to know about.
That is why a promise of “zero hallucinations” should make you pause. Finding no errors in a test does not inherently establish that a generative AI product is infallible; it could still produce unsupported information.
There are ways to reduce this risk, including selecting appropriate source material, limiting what the system is asked to do and checking claims against the record. Those measures require deliberate engineering and evaluation.
When buying AI ask the supplier what it tested, who assessed the results and whether the cases reflect your service. What did the system miss? How did it handle contradictory information or gaps in the record?
Users should be able to inspect the evidence behind a finding. An explanation that sounds reasonable is not enough!
4. What has the AI supplier made dependable, and what is left with us?
Responsible AI principles should influence how the product is built and what happens after it reaches your service.
Take bias. Could differences in the quantity or style of care notes influence who gets flagged for attention? Has the supplier tested performance across the people and settings the product is intended to support? How does it address uneven results?
Care expertise should inform what a good result looks like. There should also be ongoing monitoring, clear responsibility for investigating failures and evaluation when models or rules change, including a way to withdraw an update that causes problems.
Then ask what “human in the loop” means for the person doing the work. Can they understand the finding, inspect its evidence and challenge or escalate it? Or do they have to repeat the whole analysis to decide whether it is reliable?
Good governance should make innovation useful. It should not become a way of transferring the supplier’s quality assurance work onto carers.
5. What happens to our data, and what control do we keep?
A UK-based supplier or a “UK-hosted” product does not necessarily mean all your information stays in the UK or EU. An AI product may involve several services, each with different processing and storage arrangements.
The care record is only part of the picture. Where does the information sent to the model and its generated answers go? What about logs and metadata recording how information was processed and how an answer was produced? Those can contain sensitive information too.
At Nourish, these questions have required deliberate technology and supplier choices. Our work on EU hosting has included examining processing logs and metadata, alongside the underlying care information, across the services involved.
Ask your supplier to explain that journey, including access, retention, training use and deletion. The ICO’s international transfers guidance explains why access by a separate organisation overseas can matter even when the database itself has not moved.
EU hosting alone does not establish GDPR compliance. Involve your data protection lead to assess the lawful basis, conditions for processing health information and contractual responsibilities. Establish whether a data protection impact assessment is required, and understand what happens to your information if you leave.
6. What will it actually take to be worth buying AI?
A product needs to deliver value in everyday use, once the work of implementing and operating it is included.
If AI saves time generating a draft but adds substantial checking and correction, measure both. Include integration, staff training and ongoing support alongside licence or usage charges.
A focused trial helps establish whether the benefit is there. Agree how the task works today, what improvement would justify adoption and when you would stop or adjust. Involve the people who will use the product, and assess the quality of the outcome alongside the time involved.
I remain excited about what AI can contribute to care. At Nourish, that means considering the information already available, the care context and how intelligence can support professionals within their existing workflows.
Providers should expect suppliers to explain how they make their products dependable and provide evidence for the benefits they claim. Ask what has been tested, understand the responsibilities you retain and assess the benefit in your own service.
