Language models can now read a hospital letter and produce a fluent summary in seconds. That is useful, and it is also the source of the main risk: fluent output is easy to trust. In clinical software, the question is not whether AI can help, but how to make sure the help is checked, bounded and accountable.

Start from the person who signs

In general practice, the person who acts on information is responsible for it. A GP who changes a dose, a pharmacist who authorises a repeat, an administrator who files a letter — each is accountable for that decision. Software that assists them must make the decision easier to check, not easier to skip.

That leads to a few design rules we apply in Reedley:

  • Every statement links to its source. If the software says the letter recommends 10 mg, the reviewer should see the sentence that says so, verbatim, beside the record.
  • Confidence is a reason, not a number. "Dose stated explicitly on page 2" can be checked. "93% confident" cannot, and invites misplaced trust.
  • Inference is labelled as inference. What the record states, what a rule derived and what a model inferred are different kinds of statement and should look different.
  • No personas. Software that speaks as "I" invites a relationship of trust that a tool has not earned. It also blurs who is accountable.
  • Decisions belong to named people. Proposals are accepted, edited or rejected by a person, and that decision is recorded.

Some things AI should not do in this setting

There are tasks we think AI assistance should not perform in general practice software, at least not without a separately governed and regulated route: diagnosing, interpreting what a result means for a patient, prescribing, or acting without a person. Conditional wording — "consider", "if", "may" — should never be turned into an instruction.

Regulators make a related point. The MHRA's guidance says that whether software is a medical device depends on its intended purpose, judged partly from how it is described and marketed, not simply from the technology it uses. Its 2026 guidance on ambient voice technology draws a line between tools that transcribe, summarise for clinician review or suggest codes, and tools that suggest diagnoses or treatments or act autonomously. That distinction is useful well beyond scribes.

Evaluation that tells the truth

Responsible use also depends on knowing how well the software performs, and saying so honestly:

  • Measure false negatives and false positives separately. A missed dose change is not the same as an unnecessary task.
  • Keep a holdout set that development never touches.
  • Report by document type and specialty, not only an average.
  • Never present results on synthetic data as real-world performance.

For Reedley, that last point matters today: the product has been tested on synthetic documents only, and its extraction accuracy with a model in the loop has not yet been measured. We say so on our website because it is the most important thing a clinician should know about us right now.

Accountability is a system property

Human review is necessary but not sufficient. People can over-trust a confident interface, especially under time pressure. Good design reduces that risk — by showing the source, slowing down where it matters and making uncertainty visible — but residual risk remains, and it belongs in a clinical safety case, owned by a Clinical Safety Officer, alongside the evidence that controls work.

This article is general information about UK primary care. It is not clinical, legal or regulatory advice. Figures are taken from the cited sources and are not Reedley results.