How would you know?
An academia–industry working session on evidencing AI reliability in regulated workflows
Zurich · one afternoon, 13:00 to 16:00 · date to be confirmed By invitation · Chatham House rule · approximately 30 people
Innovation project supported by Innosuisse
The question
A compliance officer is looking at a screen. It shows one field extracted from a client file, a beneficial owner's name, and a confidence score of 0.94. There is a button marked accept and a button marked send for review.
Do you sign?
Almost every vendor selling into this market publishes an accuracy figure. Very few can say what it was measured on, across which document population, or whether the confidence score behind it means anything at all. Which leaves the institution making a procurement decision it cannot defend to its supervisor on the evidence it was given, and leaves the supervisor asking a question nobody in the chain has the evidence to answer.
That gap is not going to be closed by any single institution, because what is missing is not a measurement. It is an agreement about what to measure.
This session is an attempt to write one down.
What makes this different from a conference
It is a working session, not a programme of talks. Three short position statements set up the problem and then stop. The rest of the afternoon is the room doing the work, at mixed tables where every table has at least one practitioner and at least one statistician.
Everyone leaves holding a one-page evaluation standard they helped write. For the institutions in the room, that is a set of questions to put to a vendor that the vendor cannot deflect. For the researchers, it is a contribution shaped by the people who have to live with it. For everybody, it is a document that exists at four o'clock and did not exist at one.
Programme
| 13:00 | Arrival and coffee. Deliberately not a lunch: people leave after lunch |
| 13:15 | The provocation. One screen, one field, one confidence score. Do you sign? Hands up, then: what would you need to know? The answers go on the board and stay there all afternoon |
| 13:30 | Three positions, eight minutes each, hard-stopped. A vendor, a practitioner and a researcher answer the same single question. Not talks |
| 13:55 | Working session I. Mixed tables. What would you have to see before you signed? Every table produces a list |
| 14:35 | Break |
| 14:50 | Working session II. Tables swap lists and attack each other's. Which items are unmeasurable? Which are measurable but nobody measures them? Which are theatre? |
| 15:20 | Drafting, in plenary. One page, on screen, edited live. Everyone watches their wording go in or get cut |
| 15:50 | Close. What each participant takes away, and what happens to the document |
| 16:00 | Apéro |
What we expect to argue about
These are the fault lines the material already suggests. If the afternoon does not reach them, it has failed.
- Accuracy is the wrong headline number, and everybody keeps quoting it anyway. It averages over error types whose consequences differ by orders of magnitude. What replaces it?
- A confidence score is not a probability. It usually comes from token probabilities, which are not the probability that the answer is correct. So what is a threshold actually thresholding?
- The stratum nobody can see. Cases the system escalates get a second pair of eyes; cases it accepts do not. A dashboard built from reviewer corrections therefore shows only the errors the system already suspected. How do you measure what you never look at?
- The human in the loop is not automatically a safeguard. A reviewer is a safeguard exactly where the reviewer is better than the machine, and almost nobody has measured where that is.
- What does a record establish when the model is gone? Retention runs ten years past the end of the relationship. Vendor model versions do not last ten years. If the decision cannot be reproduced, what has been retained?
Who is in the room
Around thirty people, by invitation, under the Chatham House rule: anything may be repeated, nothing may be attributed, nothing is recorded.
| Practitioners | Compliance, financial crime and operational risk, from several institutions |
| Researchers | The project's academic advisory board, and the wider COST Action network on Fintech and AI |
| Industry | WeCanGroup SA, whose platform the underlying research studies |
| Hosts | Fachhochschule Graubünden |
Speakers to be confirmed. The academic side will be drawn from the project's advisory board and from COST Action CA19130 on Fintech and Artificial Intelligence in Finance, a network of more than two hundred researchers across forty-nine countries, chaired from FHGR. The board's strength is unusually well matched to this subject: it is largely statisticians and econometricians, and the subject is measurement.
No supervisor will be present, and that is a deliberate limitation rather than an oversight. It means the room can be candid about what institutions actually do. It also means the document the room produces cannot claim to state what would satisfy a regulator, and it will say so where that question arises.
Why WeCanGroup is convening this rather than presenting
The firm has a product in this market and a research partnership examining whether it can be trusted. The honest position, and the useful one, is that they do not yet have the evidence either: the validation study runs into 2027 and nothing has been measured. Convening the people who will decide what counts as evidence is worth more than a demonstration, and it only works if the room believes the standard was not written to fit one product. WeCanGroup speaks early and chairs nothing.
Afterwards
The one-page output is published on this site, with participants credited if they wish and not if they do not. Three months later everyone who attended gets one email asking a single question: did you use it, and what happened?
That answer is the actual result. A good discussion is not.
Requesting an invitation
The room is small because a working session stops working above about thirty people. If you work in compliance, financial crime or operational risk at a financial institution, or you research evaluation and uncertainty and want to be in the room, write to the project team through the project page and say which of the two you are.
Convened by Fachhochschule Graubünden with WeCanGroup SA, as part of the innovation project "A specialized and secure AI orchestrator", 2026–2028.
Innovation project supported by Innosuisse