Teaching note

Companion to Wecan_CaseStudy.md. Step C4 of planning-logs/2026-07-30-wecan-artifact-production.md.

Innovation project supported by Innosuisse

For the instructor, not for distribution to students. It contains the discussion answers and the point of each question.

Synopsis

A Swiss RegTech firm has deployed an LLM-based compliance platform into private-banking onboarding. It works well enough to be in production and to have produced efficiency claims. Whether it is reliable enough is an open research question, currently the subject of a funded project, and the validation study has not been done.

The case is deliberately unfinished. Most technology cases are written after the outcome is known, which lets students reason backwards from a verdict. This one has no verdict, and that is the pedagogical asset.

Where this fits


Level	Master's or executive education. Works in an MBA elective as written, and at advanced bachelor level with more scaffolding on KYC and AML basics before question 1
Disciplines	Applied AI, fintech and regulatory technology, data governance, information systems, research methods
Prerequisites	Students should know what a language model is. No machine-learning mathematics is required. Familiarity with hypothesis testing helps for question 5 but is not essential
Session length	90 minutes standard. A 60-minute version and a two-session version are given below
Target course and length	Decided 03.08.2026: MSc or executive education, one 90-minute session. Owner FHGR, step C5 of `../planning-logs/2026-07-30-wecan-artifact-production.md`. The alternatives considered were a two-session bachelor format and a half-day workshop; both remain available through the variants below, but the 90-minute session is the one the material is tuned for

Learning objectives

By the end of the session students should be able to:

Distinguish accuracy from calibration, and explain why a threshold-routed workflow depends on the second in a way it does not depend on the first. Note the case body deliberately stops short of asserting that calibrated-but-less-accurate systems dominate: that is the empirical question the research project exists to answer, and the note should not teach it as settled.
Explain why aggregate accuracy is a poor deployment criterion when error costs are asymmetric.
Distinguish a hallucination from an extraction error and say why the ordinary plausibility check fails on the former.
Locate a human-in-the-loop threshold and name what they would measure to set it and to know they had set it wrong.
State what auditability requires of a system beyond producing the right answer.
Recognise when a number is not evidence, and articulate what would have to be true for it to become evidence.

Objective 6 is the one that transfers outside the domain, and it is the one worth protecting if time runs short.

Session plan, 90 minutes

Time	Segment	Mode
0 to 6	Warm-up, before anything else. "The system says this beneficial owner is Jean Dupont, with 94 percent confidence. Do you sign?" Show of hands. Then: what would you need to know? Write the answers up; you will come back to them at minute 86	Whole class
6 to 12	Mercier's decision. She has to name a threshold on Tuesday. Put the deputy's position and the COO's position on the board as two columns. Do not resolve it	Whole class
12 to 20	Frame the setting and the asymmetry, sections 1 and 2. Keep it short: the students have read it	Lecture
20 to 42	Question 1, evaluation metrics, working from Exhibits 2 and 3. Have groups compute the gap column themselves before you show it	Small groups, then plenary
42 to 60	Question 2, the threshold, working from Exhibit 4. This is the arithmetic segment and the heart of the class	Small groups, then plenary
60 to 72	Question 3, auditability, or question 4, the language disparity in Exhibit 2. Choose by cohort; question 4 lands better with executives	Plenary
72 to 86	Question 5, the research-integrity question, with Exhibit 5 on the board. Do not skip this	Whole class
86 to 90	Close on the warm-up. Ask the Jean Dupont question again and see whether the hands moved, and whether the list from minute 6 has changed	Whole class

60-minute version, and the timings do add up: warm-up and Mercier's decision (0 to 12), framing (12 to 18), question 2 with Exhibit 4 (18 to 38), question 5 (38 to 55), close (55 to 60). Question 2 rather than question 1 is the one to keep in a short class, because it is the only segment where the students have to produce a number and defend it.

Two-session version:

Session	Time	Content
One	0 to 12	Warm-up and Mercier's decision
One	12 to 25	Framing, sections 1 to 4, with Exhibit 1
One	25 to 55	Question 1, Exhibits 2 and 3
One	55 to 90	Question 2, Exhibit 4. Set the technical note as reading for session two
Two	0 to 10	Recap: what did we conclude about the threshold, and what did we assume
Two	10 to 35	Question 3, auditability
Two	35 to 55	Question 4, the language disparity
Two	55 to 85	Question 5, with Exhibit 5. Thirty minutes is right for this
Two	85 to 90	Close

Discussion guide

Question 1: how should reliability be evaluated?

What students say first: accuracy. Let it run, then push.

Where to take it. Draw out the distinction between how often is it right and does it know when it is wrong. The second question is calibration, and it is the one the platform's architecture actually depends on, because the whole design routes low-confidence cases to a human. If confidence scores are not calibrated, the routing is arbitrary and the human review is theatre.

The move that usually unlocks the room, and it works because the exhibits carry the numbers.

Ask them to compare two systems, both invented for the purpose:

	System A	System B
Overall accuracy	0.86	0.79
Confidence score	Reports 0.86 on every single field	Varies field by field, as in Exhibit 6
Calibrated?	Yes, perfectly. It is right 86 percent of the time and says 0.86	Roughly, and provably so in the three upper bands
Useful for routing?	No. No threshold divides its output at all	Yes

Watch the axis carefully, because this is where the usual version of this exercise goes wrong. System A is not uncalibrated. It is perfectly calibrated and useless, which is a different defect and the more instructive one. What A lacks is resolution: its score does not discriminate between easy cases and hard ones, so there is no threshold at which anything sensible happens.

Most students pick A on instinct, because 0.86 beats 0.79. Then ask: at what confidence would you auto-accept an A extraction? There is no answer. Every field looks identical to the router. So A forces you to review everything or nothing, and with B you can at least buy something with your reviewer hours.

Do not overclaim the reversal. B is not simply better. Under review-everything, A puts fewer errors on the record at the same cost, and a student who says so is right. The claim is narrower and worth stating precisely: once you are choosing a threshold, calibration and resolution are what make the choice meaningful, and accuracy alone cannot.

Push further if the group is strong: precision and recall are not symmetric here either. For sanctions screening a false negative is a regulatory finding, a false positive is a minute of somebody's time. Ask what that does to the threshold, and whether one threshold can serve both extraction and screening.

Question 2: where does the human sit?

The trap to expose: "human-in-the-loop" used as a reassurance rather than a design. Ask what the human actually does, on which cases, with what information in front of them, and how long they have.

Draw out the cost. Review capacity is finite. A threshold that sends 40 percent of cases to review has not automated anything. A threshold that sends 2 percent has automated a lot and is only safe if the confidence score is trustworthy, which returns to question 1.

The measurement question, which is the real content: how would you know your threshold is wrong? The answer students rarely reach on their own is that you have to audit a sample of what the system did not escalate. Errors in the escalated pile are visible by construction; errors in the auto-approved pile are the ones that hurt, and nothing surfaces them unless somebody deliberately looks.

Question 3: auditability

Key idea: the requirement lands on the record, not on the model. In three years the model will have been replaced, possibly twice.

Ask what must be retained: the input document, the extraction, the confidence score, the threshold in force at the time, the model version, whether a human reviewed it and what they changed. Then ask who pays for that storage and who tests that it can still be read.

Good provocation: if the vendor's model is an API from a third party that has since retired that version, can you reproduce the 2026 decision at all? If not, what does your audit record actually establish?

Question 4: data quality and bias

Concrete route in: extraction quality on a clean digital PDF versus a photographed scan; on German versus Italian; on a standard corporate structure versus an unusual one.

The propagation point: an extraction error does not stay put. A misread nationality flows into screening and into risk scoring, so a data-quality problem surfaces as a compliance outcome several steps downstream, where nobody attributes it to extraction.

The equity point, if the course covers it: if the pipeline is weaker on documents from certain jurisdictions, the practical effect is that clients from those jurisdictions get more friction. Nobody designed that.

Question 5: the research-integrity question

Do not skip this, and do not rush it. It is the question the case exists to make possible, and it is the one students will meet in professional life regardless of whether they work in compliance.

Set it up honestly. The efficiency figures arrived before the reliability study. The firm has a commercial interest in the figures being good. The research partner has a reputational interest in the figures being right. Neither party is behaving badly. The tension is structural.

What to draw out:

The difference between an estimate and a measurement is not pedantry; it is the whole difference between a marketing claim and a finding.
Asking "what was measured, on what sample, when" is a neutral, professional question, not an accusation. Students often need permission to see that.
A number can be anonymised and still be unverified. Removing the client's identity solves a confidentiality problem and does nothing whatsoever to the truth problem. These are two gates and passing one is often mistaken for passing both.
If the partner wants to publish sooner, the honest options are: publish as a position paper with no empirical claim, publish the method without the results, or wait. Fabricating a provenance is not on the list, and neither is publishing the number with a vague hedge.

Optional extension for research-methods courses. Give students section 10 of the case and ../data/figure-provenance.json, and ask them to classify each figure as measured, projected, unconfirmed or illustrative. The exercise is quick and the result is uncomfortable in a productive way: none of the project's performance figures currently qualifies as measured, and several precise-looking numbers come from documents whose own version history is dated in the future.

Board plan

Three columns, built up across the session.

WHAT WE MEASURE      | WHAT IT TELLS US        | WHAT IT HIDES
---------------------+-------------------------+--------------------------
accuracy             | how often right         | which errors, and how bad
calibration          | does it know it's unsure| nothing much, this is the good one
precision / recall   | error direction         | the cost asymmetry
throughput / cost    | efficiency              | reliability entirely

Leave the bottom-right cell for last and write it in a different colour. "Reliability entirely" is the sentence the case is built around, and it lands harder written than said.

Where the discussion goes wrong, and what to do

Distinct from the misconceptions table below, which covers wrong answers. These are failed discussions, and each one has a move.

It goes here	Do this
The room relitigates whether AI belongs in compliance at all and never reaches metrics	Concede the question and postpone it explicitly: "assume the bank has decided to deploy something; Mercier's Tuesday is about which threshold, not whether." If it recurs, promise the last ten minutes to it and hold them to that
One practising compliance officer takes the room hostage with war stories. Likely with executive cohorts	Use them rather than shutting them down: "you have set a threshold like this in practice, what did you actually watch afterwards to know it was right?" That converts anecdote into question 2
Students demand the withheld figures. Near-certain with the public edition, which tells them the numbers exist and does not show them	Do not defend the decision twice. Put it to the room instead: "if I gave you 80 percent, what would you then be able to conclude that you cannot now?" The honest answer is very little, and they get there faster than you can tell them. Exhibit 5 is the board move
The threshold discussion collapses into "it depends"	Force a number. Each group writes one threshold on a card with one sentence of justification, then read them out. Disagreement between groups is the discussion
The warm-up dies because nobody wants to guess in front of the room	Ask for hands rather than reasons, then call on somebody who raised one and ask only what would change their mind. Never open by asking a named student to justify a position
Somebody spots that the case's own numbers are invented and concludes the whole case is unserious	This is a good student. Agree at once, then draw the distinction: a stipulation asserts nothing about the world, whereas an unsourced figure asserts something it cannot support. That distinction is the case

Common misconceptions to pre-empt

Misconception	Correction
"More accurate is always better"	Not when errors are asymmetric and confidence is uncorrelated with correctness
"Human-in-the-loop makes it safe"	Only if the loop sees the right cases. Otherwise it adds cost and the appearance of safety
"Hallucination is just an error"	It is the error class that passes the plausibility check, which is the check humans actually apply
"The figures prove it works"	The figures are efficiency estimates supplied by an interested party. They are not reliability evidence and were never claimed to be
"Regulators will accept it if it is accurate"	Accuracy without an auditable record is not a defensible position

Assessment ideas

Short written (1,000 words). Design an evaluation protocol for the extraction component. Specify the metrics, the sample, and the acceptance thresholds, and justify each against the cost asymmetry.
Group presentation. You are the research partner. Present to the firm the conditions under which you would co-author a paper using their figures.
Analytical exercise. Using Exhibits 2 to 4, compute the calibration gap for each confidence band, the errors reaching the record at thresholds of 0.70, 0.80 and 0.90, and the reviewer hours each implies. Recommend a threshold to Mercier and defend it against both her deputy's position and her COO's.

Instructor's note on the case's own status

This case is currently classroom material and not an academic output, because the provenance of the firm's figures is unresolved (open question Q1). If it is later used as a research artifact, the figures must first be confirmed or removed.

That constraint is worth stating to students. A teaching case that is honest about its own evidential limits is a better demonstration of the point it is teaching than any amount of exhortation.

The project's two calendars, for the instructor

The case says the project is in month 5 of 24 and runs April 2026 to March 2028. If a student checks that against the funding application they will find January 2026, so here is the reason.

The signed funding agreement fixes no calendar start. It makes the start signature-relative: the project may begin only once the agreement is signed, and must begin within three months. It was signed at the end of January 2026, so the official Innosuisse start is 01.04.2026 and the filed status report confirms it, mapping June 2026 to project month 3.

The project also keeps an internal planning calendar whose first month is January 2026, three months ahead of the official clock. Internal working documents place the annotated dataset at month 12 and the validation report at month 15 of that internal calendar. On the official clock the same milestones fall about three months later.

Use the official dates in anything a funder, a student or the public will read. If a student spots the discrepancy, they have found something real, and it is a reasonable prompt for a discussion about why a project would run two calendars at once.

Instructor Material: AI-Native Compliance in Swiss Private Banking