The guide to buying AI for care and support providers was written by our Director of Data and AI, Sudha Regmi. Sudha has 15 years of experience in AI. Prior to Nourish, she spent 7 years at an international consulting company, helping large and midsize companies across the world design and implement their Data & AI strategy.

Everyone and their brother wants to sell you AI at the moment. Which means buying AI is fast becoming a mandatory skill for decision makers in care and support services.

That was the conversation behind my session at the Nourish Partnership Conference. There is plenty to be excited about: helping teams review care information, identify concerns and spend less time on administration.

Before joining Nourish, I spent years as a Data & AI Strategist, advising clients on technology decisions, including whether to build or buy. Those conversations always started with understanding the requirement. Working in social care has made the importance of that particularly tangible.

For many providers, buying an established solution will make more sense than building and maintaining their own. But advances in generative AI have made it much easier to build a convincing prototype. Developing a robust solution that people can rely on in a care service remains considerably harder. A polished demonstration tells you very little about the validation and engineering underneath it. So, beware the shiny demo!

I work closely with my colleagues at Nourish, such as our Director of Partnership, Denise Tack, and Head of Legal & Compliance, Jess Osmond, in evaluating AI vendors in the market, either for our use, or while advising our customers on choosing the right AI vendors for them. Here are the six questions we typically ask while evaluating.

What are we actually asking AI to do? 🔗

What information does AI need, and where should it live? 🔗

What happens when AI gets something wrong? 🔗

What has the AI supplier made dependable, and what is left with us? 🔗

What happens to our data, and what control do we keep? 🔗

What will it actually take to be worth buying AI? 🔗

Many care providers want to use AI in care planning. That can mean quite different use cases, potentially involving different AI techniques.

You might want help turning a carer’s observations into a clear draft. You might want to identify inconsistencies in a plan, or bringing together evidence that someone’s needs are changing. Each requires different information, evaluation and professional involvement.

Being clear about the task makes it easier to choose the right product. If you understand your objective, it is easier to understand what AI applications you need. Who will use the output? What will they do with it? What would improve for the service or the person receiving care?

The intended use also matters when assessing regulatory requirements. The AI and Digital Regulations Service’s social care guide directs providers towards Medicines and Healthcare products Regulatory Agency (MHRA) guidance on intended purpose and whether a product qualifies as a medical device. Ask suppliers to explain which requirements apply to their product and your proposed use.

For AI to be useful in care, it needs a strong foundation of data underneath it. When buying AI, even a capable model can produce poor results if the information it receives is incomplete, outdated or missing the relevant context.

Consider understanding a change in someone’s needs. The evidence might sit across daily observations, their preferences and previous reviews. The system needs to connect those records correctly and distinguish a recent development from an old concern. That context must stay current as care continues.

This is where data gravity matters. As information accumulates in a system, moving it elsewhere, as well as preserving the context that makes it useful, becomes more complex. There is a practical reason to bring AI closer to where care information already lives and is maintained.

An integration can connect systems, but someone still has to keep records current, preserve permissions and handle changes. In practice that costs more than just you initial setup fee when buying AI. It creates ongoing responsibility, which in turn creates ongoing cost.

AI within an established care platform can build on existing data foundations and workflows. Specialist partners can add valuable capabilities too. In either case, ask how the supplier maintains the quality of that context and brings findings back into the work of the care team.

A care-plan summary can read beautifully while leaving out an important detail. An audit review can produce sensible findings while missing the vulnerability you most needed to know about.

That is why a promise of “zero hallucinations” should make you pause. Finding no errors in a test does not inherently establish that a generative AI product is infallible; it could still produce unsupported information.

There are ways to reduce this risk, including selecting appropriate source material, limiting what the system is asked to do and checking claims against the record. Those measures require deliberate engineering and evaluation.

When buying AI ask the supplier what it tested, who assessed the results and whether the cases reflect your service. What did the system miss? How did it handle contradictory information or gaps in the record?

Users should be able to inspect the evidence behind a finding. An explanation that sounds reasonable is not enough!

Responsible AI principles should influence how the product is built and what happens after it reaches your service.

Take bias. Could differences in the quantity or style of care notes influence who gets flagged for attention? Has the supplier tested performance across the people and settings the product is intended to support? How does it address uneven results?

Care expertise should inform what a good result looks like. There should also be ongoing monitoring, clear responsibility for investigating failures and evaluation when models or rules change, including a way to withdraw an update that causes problems.

Then ask what “human in the loop” means for the person doing the work. Can they understand the finding, inspect its evidence and challenge or escalate it? Or do they have to repeat the whole analysis to decide whether it is reliable?

Good governance should make innovation useful. It should not become a way of transferring the supplier’s quality assurance work onto carers.

A UK-based supplier or a “UK-hosted” product does not necessarily mean all your information stays in the UK or EU. An AI product may involve several services, each with different processing and storage arrangements.

The care record is only part of the picture. Where does the information sent to the model and its generated answers go? What about logs and metadata recording how information was processed and how an answer was produced? Those can contain sensitive information too.

At Nourish, these questions have required deliberate technology and supplier choices. Our work on EU hosting has included examining processing logs and metadata, alongside the underlying care information, across the services involved.

Ask your supplier to explain that journey, including access, retention, training use and deletion. The ICO’s international transfers guidance explains why access by a separate organisation overseas can matter even when the database itself has not moved.

EU hosting alone does not establish GDPR compliance. Involve your data protection lead to assess the lawful basis, conditions for processing health information and contractual responsibilities. Establish whether a data protection impact assessment is required, and understand what happens to your information if you leave.

A product needs to deliver value in everyday use, once the work of implementing and operating it is included.

If AI saves time generating a draft but adds substantial checking and correction, measure both. Include integration, staff training and ongoing support alongside licence or usage charges.

A focused trial helps establish whether the benefit is there. Agree how the task works today, what improvement would justify adoption and when you would stop or adjust. Involve the people who will use the product, and assess the quality of the outcome alongside the time involved.

I remain excited about what AI can contribute to care. At Nourish, that means considering the information already available, the care context and how intelligence can support professionals within their existing workflows.

Providers should expect suppliers to explain how they make their products dependable and provide evidence for the benefits they claim. Ask what has been tested, understand the responsibilities you retain and assess the benefit in your own service.

Those questions apply to Nourish too.

We spoke to Angie Kemp and Nicola Hughes from Autism Together about the impact Nourish Transparency has had on their service while at the Nourish Partnership Conference.

As early adopters of the platform they have been instrumental in the development and application of Nourish Transparency. Helping to ensure the platform addresses real challenges and offers effective solutions for care providers.

In our Nourish Transparency case study video they detail how the solution helps them to demonstrate a clear picture of the level of care the people they support need, and the true cost of their care.

With reports and dynamic sampling functionality Nourish Transparency enables them to specifically and accurately evidence the level of funding required to deliver quality care in their community.

The Nourish Partnership Conference 2026 was held on September 8th, at the Waldorf Astoria in London.

The event highlighted and celebrated how connected digital ecosystems work best when the people behind them work together.

Throughout the day our partner expo gave providers the opportunity to meet the people behind the technology, discover new solutions and discuss how different systems can work together around their unique needs.

While in person and on stage our partners, guests and fine selves will address some of the biggest questions shaping care technology from workforce and privacy to AI and data. All underpinned by a commitment to digital connections and, crucially, human ones.

We also celebrated the inaugural DEC awards, Born from the co-produced Digital Excellence in Care Framework, the awards celebrate organisations leading the way in digital care and support. The awards recognise care providers using technology and data with purpose to improve quality, safety and outcomes.

AI can help care teams work more consistently and spot risk earlier. However, social care is a deeply personal service that is both context-heavy and filled with risk. Which means we need strong governance and clear guardrails for AI. The English social care regulator, the Care Quality Commission, recently released their guideline for the safe and effective application of AI in care settings. In this blog we explore how to design AI for social care that exemplifies these standards so it can best elevate the quality of care and support available.

In practice, safe AI is more than just a ‘model’, it’s a product. When we view it as a product, it is easier to make deliberate product decisions for the application of the model. Decisions like, what the AI is for, how it’s presented, and where humans need to stay in control on the output. We must put sensible processes around our reviews. As well as adding automated checks and record‑keeping so we can learn and improve over time. 

Think of responsible AI as an end to end workflow.

Inputs > AI suggestion > safety checks > human review > learning loop

The details vary by product, but the principles don’t.

Data input: Define boundaries and responsibilities 

Start by being clear about boundaries. What information the AI can use, what it must ignore, how up-to-date the inputs are, and who owns data quality. This prevents false confidence and makes issues easier to investigate. 

Expert context: Encode clinical intent before inference 

Care quality depends on domain knowledge. You should build and maintain clear guidance on ‘what good looks like. This is in terms of terminology, standards and policies. Crucially, with named owners and a review cadence. So the AI stays aligned with your service as practice and policy evolve. 

AI analysis: Require evidence, not opinions 

Whatever the AI outputs, it should be available to review. Which means showing the relevant source text and including a brief explanation. You want to avoid the logic of ‘because the model says so’. It is vital that the evidence the model utilises is identifiable and interrogatable.

Automated guardrails for AI: Validate before humans see it 

AI can be trained to add lightweight automated checks. These are useful for catching the obvious problems before a human sees them. For example, unsupported claims, inconsistencies or repeated items. These checks don’t need to be fancy or overly detailed, however, they do need to be reliable. 

Human-in-the-loop: Keep clinical judgement in the driver’s seat 

We must keep humans in control. You should design the experience so reviewers can quickly understand the suggestion, accept or reject it, and escalate edge cases. Then use real-world feedback to improve the system on a regular cadence. 

The goal is simple, make it easy to do the right thing, and hard to do the wrong thing. The reality is simpler than you’d suspect to achieve.

There are a number of risks in AI that are well documented. It is important to note some of the AI risks in social care don’t come from ‘the model’ alone. They come from product decisions like wording, defaults, missing context, and unclear accountability. 

It is important that you address these issues with your product teams. Once these basics are set, governance becomes repeatable.

Governance is how you earn trust over time, and sets the standard for your guardrails for AI. It is vital to test, monitor, update safely, and keep an audit trail of your actions.

Evaluation: Agree what “good” looks like, then keep testing it!

Before you ship any new features it is crucial you test against a reference set of real cases reviewed by experts. After you ship, keep sampling and checking performance so any drift that occurs doesn’t surprise you. A good way to do this is to define a small set of ‘go/no‑go’ checks for your releases. Be explicit about data handling and access. If you can’t explain those basics, you can’t govern the system. 

Change control: Make updates safe (and easy to roll back) 

Treat updates like a high-stakes product release. Ensure you are doing everything thoroughly, document changes, getting the right sign-off, and make rollback a straightforward process. 

Metadata and auditability: Make every finding explainable 

Audit trails are essential aspects of guardrails for AI. You need to keep an audit trail so you can answer important questions and evidence your working. Questions like, ‘What did the AI see?’ ‘What did it suggest?’ and ‘What did the human decide?’. 

The effectiveness of any AI system depends on the quality, relevance, and governance of the data it uses. Organisations should be able to clearly explain what data is being used, where it comes from, and how it is curated to ensure it remains accurate, current, and appropriate for the task at hand. This curation process is critical, as it provides the AI with the right context, rules, and safeguards to support decision-making in complex care environments. Rather than relying on vast amounts of unstructured information, responsible AI carefully selects data sources that reflect best practice, regulatory requirements, and the realities of delivering care.

Equally important is the security of that data. Care providers, practitioners, and the people they support place significant trust in the organisations that handle their information. AI vendors should be transparent about where their products are developed and hosted, how data is stored and processed, and the measures in place to prevent unauthorised access, misuse, or data leakage. This includes robust technical safeguards, strong governance frameworks, and ongoing monitoring to ensure sensitive information remains protected at every stage.

Safe AI in social care comes from disciplined choices. Teams that design for review, keep humans in control, add sensible checks, and maintain an audit trail. With these guardrails for AI it can quickly become a practical support for care teams. One that they can trust because they can understand it without asking them to accept the findings of a ‘black box’ at face value. Below is a basic checklist for your team to utilise. This is designed as a starting point for your team and is not definitive. After all, AI models, like care communities, are at their best when they champion their unique perspectives!

If you’d like to learn more about how we’re designing AI to help care teams work more consistently and spot risk earlier contact us directly.