Clinical AI Governance Handbook

01

Scope and first principle

This handbook covers every use of artificial intelligence that touches a patient's care in this hospital: the deterministic clinical rules (early warning scores, red flags, medication checks), the generative drafting (notes, summaries, letters), and the agents that watch for events and prepare work. It does not cover back-office software with no clinical effect.

Two consequences follow, and they are not negotiable locally. First, any feature that would act without a human decision is a level-4 action and stays disabled. Second, every consequential action is recorded with the model, the version, the sources it used and the person who approved it — so that "who decided this?" always has an answer.

02

Who decides what

Governance fails when it is one committee with no owner. Four roles carry named responsibility; one group brings them together.

03

The five action levels

Every AI action in the platform has a level. The level decides who may request it, who must approve it, and whether it can take effect at all.

Overrides and acknowledgements are not the same thing

When a safety finding is shown, a clinician may either acknowledge it (they have read it and it does not change their decision) or override it (they are proceeding against it). An override requires a typed reason. A contraindicated finding — a documented allergy, a drug contraindicated in pregnancy — cannot be cleared at the pharmacy counter at all; it goes back to the prescriber.

04

Before a model goes live

Nothing is switched on because it works in a demonstration. The governance group signs a short validation record for each model, and the platform's registry holds the same facts so that anyone can see them later.

05

What is watched afterwards

Validation is not a certificate; it is the beginning of monitoring. Four numbers are reviewed monthly, all of them reported by the platform about itself.

A rising override rate is a healthy finding, not an embarrassment. It means clinicians are reading what the system produced. The failure mode to fear is an override rate of zero.

06

Incidents and the kill switch

An AI safety incident is any occasion where the platform contributed to, or failed to prevent, a risk to a patient — including a missed escalation, an unsafe draft that reached a signature, or a critical result that was never communicated.

On the day

1. Make the patient safe first. Clinical management comes before any investigation.
2. Suspend the model from the Guardian console if it is implicated. Features using it stop immediately; the rest of the hospital keeps working.
3. Preserve the trail. Do not attempt to correct history — the chain is append-only and editing it is detectable, which is the point.
4. Tell the clinical owner and the DPO the same day. If personal data was disclosed wrongly, the DPO decides on notification.
5. Record it on the hospital's existing incident system, cross-referenced to the audit entry id.

07

The audit trail as evidence

Each audit entry carries the hash of the entry before it. Changing or removing history breaks the chain, and verification says exactly where. That property is what turns a log into evidence.

• What is recorded: who acted (person or agent), their role, the action, its level, the patient, the model and version, the sources used, the decision, and any reason given for an override.
• What is not recorded: the full generated text of every draft. Summaries and identifiers are kept; the clinical record holds what was actually signed.
• Who may read it: roles with an audit permission. Reading the trail is itself an action within the hospital's access controls.
• Verification: run chain verification before any external audit, and after any database restore. A restore from backup is the common cause of a genuine break.

The compliance pack renders the current state of all of this on one page, intended to be printed on the day an assessor asks.

08

Consent, rights and retention

Under the Digital Personal Data Protection Act, the hospital is the data fiduciary. HealthTeky holds the mechanics; the duties stay with the hospital.

• Notice before consent. The patient is told the purposes in plain language before consent is taken. Consent without the notice is not consent.
• Purpose, scope and expiry on every artefact. Break-the-glass access expires automatically after 24 hours and every instance is reviewed by name.
• Withdrawal is as easy as giving. It takes effect on the next request; the artefact is retained as history, with the withdrawal recorded.
• Rights have deadlines. Access, correction, erasure, nomination and grievance are tracked against the statutory date, and an overdue request is visible before it becomes a complaint.
• Erasure is answered honestly. Clinical and financial records that law requires the hospital to keep are named, with the reason given to the patient, alongside what genuinely can be stopped: marketing, research use, outreach contact details.

09

Security of the AI layer

The AI layer has two risks a conventional application does not: text that tries to instruct the model, and data leaving the hospital inside a prompt.

• Patient data is never an instruction. Record and transcript content is enclosed and the model is told to ignore instructions inside it. Uploaded protocols are screened for injected text before indexing.
• Grounding is checked. Generated answers are checked against the source record; ungrounded statements are flagged rather than displayed as fact.
• Where the text goes. If a language model provider is configured, the prompt leaves the hospital. The governance group decides which provider, and the platform can run entirely offline — every deterministic rule still works with no model at all.
• Machine access. Other systems authenticate with a hospital API token, never a session cookie, with an optional IP allowlist. Only a hash of each token is stored.
• Keys. Provider keys live in the environment, never in source. Rotate on staff change and on any suspected exposure.

10

Human factors

The two ways clinical AI fails in practice are not technical.

Alert fatigue

An alert that fires too often is an alert nobody reads. This is why sepsis screening requires a clinician to suspect infection rather than firing on physiology alone, and why moderate findings are shown without blocking. Count alerts per shift at each review; if a category is being dismissed almost every time, it is the alert that needs changing, not the staff.

Automation bias

People trust a confident screen more than their own judgement, especially when tired. Three countermeasures are built in: every claim carries its source so it can be checked in one click, missing data is declared as missing rather than scored as normal, and a draft is visibly a draft until it is signed. Training should show staff a case where the system was wrong — and it should be a real one from this hospital's review log.

11

Go-live checklist

12

Templates

Copy these into the hospital's own document set. They are deliberately short; a template nobody completes governs nothing.

Model id:            ht-______________          Version: ____
Clinical owner:      Dr ____________________   Date: ________
Purpose (1 line):    _______________________________________
Intended population: _______________________________________
Excluded:            _______________________________________
Known limitations:   _______________________________________
Local validation:    Sample size ____  Period ____________
                     Method: __________________________________
                     Findings: ________________________________
Rollback:            Manual process if suspended: _____________
Decision:            [ ] Approved for use   [ ] Approved with conditions
                     [ ] Not approved
Conditions:          _______________________________________
Signed (Chair):      ____________________  Review due: _______

1. Incidents since last meeting            (owner: DPO / clinical owner)
2. Override rate by model, with commentary (owner: platform owner)
3. Ungrounded-answer flags                 (owner: clinical owner)
4. Break-glass accesses, reviewed by name  (owner: DPO)
5. Critical results not acknowledged in time
6. Consent: withdrawals, refused disclosures
7. Patient rights requests: open, overdue
8. Models proposed for approval or suspension
9. Actions, owners, dates

Date/time noticed:  ______________  Reported by: ______________
Patient (UHID):     ______________  Audit entry id: ____________
Model(s) involved:  ______________________________________
What happened:      ______________________________________
Patient harm:       [ ] None  [ ] Near miss  [ ] Harm — describe
Immediate action:   [ ] Model suspended at ______  [ ] Clinical action
Contributing factors (workflow, training, data, model):
                    ______________________________________
Decision:           [ ] Resume  [ ] Resume with change  [ ] Retire model
Change required:    ______________________________________
Closed by:          ____________________  Date: ______________