Working paper

When the AI Hallucinates, Who Pays?

Mapping LLM hallucination types to legal liability in EU and Swiss financial regulation.

Joerg Osterriedera,*, Lennart Baalsa, Illia Kosterina, Vincent Pignonb, Nathan Douetb, Didier Baertschigerb

a University of Applied Sciences of the Grisons, Chur, Switzerland b Wecan Group, Geneva, Switzerland

* Corresponding author: joerg.osterrieder@fhgr.ch

Download PDF Read abstract Project overview

Abstract

Financial institutions are deploying large language models in compliance-critical functions: client onboarding, sanctions screening, beneficial ownership identification and suspicious activity reporting. These models hallucinate, and in regulated financial services a hallucination is not merely an inconvenience but a potential anti-money laundering violation, sanctions breach or data protection infringement.

Current frameworks, among them the EU AI Act (Regulation 2024/1689), the Swiss Federal Act on Data Protection, FINMA Guidance 08/2024 and the Swiss Anti-Money Laundering Act, address AI-generated inaccuracy in general terms, without treating hallucination as a failure mode with its own risk profile. This paper argues that the legal consequence of a hallucination depends on its type, and that regulation which does not distinguish types cannot allocate liability coherently.

We develop a compliance-specific taxonomy of four hallucination types (factual, numerical, fabricated and unsupported), defined by their evidentiary character rather than by their textual relationship to a source. We then map each type to the obligations it activates, the party that bears liability, and the enforcement mechanism that follows, across EU and Swiss law. Three constructed scenarios trace the resulting pathways.

Two findings are central. Detectability falls monotonically across the four types, and the AI Act's human oversight provision (Art. 14) is calibrated for the detectable end: for unsupported hallucinations, where nothing in the source is contradicted, oversight cannot function as the Act assumes. And only fabricated hallucinations routinely reach the provider, because fabrication is a defect in the system rather than an error in an extraction. That conclusion now rests on the revised Product Liability Directive alone, the AI Liability Directive having been withdrawn in 2025.

Contributions

1

A liability-oriented taxonomy

Four types classified by evidentiary character, what a reviewer would need in order to detect the error, rather than by the output's relationship to a source. This splits the extrinsic category that existing taxonomies treat as one thing.

2

A mapping framework

Each type mapped to the provisions it engages across the EU AI Act, the Swiss FADP, FINMA's supervisory framework and Swiss AML law, with the liable party and the enforcement route for each.

3

Traced consequences

Three constructed scenarios showing that types produce materially different legal outcomes, including where criminal exposure attaches and where it does not.

The four types

Existing taxonomies classify hallucination by how the output relates to its source. That is the right distinction for measuring model behaviour and the wrong one for allocating liability, because two hallucinations with identical source relationships can sit in different legal regimes.

F Factual

Wrong value, right answer available

The source carries the correct fact and the model produced an incorrect one. Verifiable against a ground-truth document, so a document comparison detects it mechanically.

N Numerical

Quantity or threshold distorted

Formally a subset of factual error, separated for a legal reason: financial regulation operates through thresholds, so numerical error interacts with binary legal consequences.

B Fabricated

The referent does not exist

An invented entity, document or regulation. There is no source to check against; detection requires the authoritative legal corpus and the expertise to use it.

U Unsupported

Plausible, but undocumented

An inference presented as a documentary finding. It may be true. Nothing contradicts it, so detection means noticing an absence, which no document comparison can do.

The mapping framework

Each type carries a distinct evidentiary character, which determines what oversight can detect, which obligation is engaged, and where liability lands.

F  Factual Wrong value where the source carries the right one
Verifiable against a ground-truth document. Cross-referencing detects it mechanically.
AMLA identification duty; AI Act Art. 15 accuracy; FADP Art. 6 data accuracy
Deployer. Provider exposure contractual only
N  Numerical Quantity, threshold or date distorted
Verifiable, and additionally checkable by arithmetic and range constraints.
AMLA reporting duty to MROS; AI Act Art. 15; AMLO-FINMA documentation
Deployer. Provider exposure contractual only
B  Fabricated Referent does not exist at all
No source to check against. Requires the authoritative legal corpus and legal skill.
Product Liability Directive defectiveness; AI Act Art. 15 and Art. 53 GPAI duties
Provider and deployer. The only type reaching statutory provider liability
U  Unsupported Plausible inference the source does not license
Not detectable by cross-reference. Nothing is contradicted; support is simply absent.
AI Act Art. 14 human oversight and Art. 26 deployer duties; FADP Art. 21
Deployer, by system design rather than by review

Detectability falls from F to U. Controls that work on F and N (document cross-referencing) do nothing for U, where no claim is contradicted. Only B routinely reaches the provider, because fabrication is a defect in the system rather than an error in an extraction.

Case scenarios

These are constructed hypotheticals

They are not case studies or incident reports. No institution, transaction, individual or enforcement action described is real, and the regulatory instrument invented in the second scenario is invented deliberately, as the object of analysis.

Types F + U

The phantom beneficial owner

A one-character surname substitution defeats screening, and an unsupported “no political exposure” statement papers over the gap. The combination produces layered liability: plainly negligent as to the detectable error, materially stronger as to the undetectable one.

Type B

The invented exemption

An assistant cites a circular that does not exist, following the real numbering convention and answering a genuine regulatory question. This is the one scenario that opens the provider pathway, and the one made harder by the AI Liability Directive's withdrawal.

Type N

The decimal point

A tenfold extraction error drops a transfer below an escalation threshold, and no report is filed. The clearest causal chain of the three, and the only one carrying criminal exposure under the reporting duty.

Regulatory coverage

EU AI Act
Art. 9 risk management, Art. 14 human oversight, Art. 15 accuracy, Art. 26 deployer obligations, and the Chapter V duties on general-purpose models. Annex III obligations apply from 2 August 2026.
Product Liability Directive
Directive (EU) 2024/2853 brings software within scope and applies from 9 December 2026. It now carries the provider pathway alone.
AI Liability Directive
Withdrawn. Announced in the Commission's 2025 work programme in February 2025, with formal notice published in October 2025. Proving causation against a provider is correspondingly harder.
Swiss FADP
Art. 6(5) data accuracy and Art. 21 automated individual decisions, which raise the question of when LLM-assisted processing becomes de facto automated.
FINMA Guidance 08/2024
Governance, risk management, data quality and documentation, on a principle-based footing. Fitness for purpose is required; type-specific evaluation is not yet specified.
Swiss AML framework
AMLA Art. 3, 4, 6 and 9, with AMLO-FINMA Art. 13. Art. 37 makes violation of the reporting duty criminal: up to CHF 500,000 intentional, CHF 150,000 negligent. It does not reach failures of due diligence, which run through supervisory channels.
Council of Europe
The Framework Convention on Artificial Intelligence (CETS 225), which will shape the Swiss regime in a more explicit direction than the current principle-based approach.

How the references were verified

A paper about fabricated and misattributed citations should be able to account for its own. Two separate checks were applied, because they catch different failures.

Does the work exist?

Every reference was resolved against an authoritative registry (Crossref for DOIs, the arXiv API for preprints, direct resolution for legal instruments). Identifiers were discovered by registry search, never transcribed, because a recalled identifier is itself an unverified claim. Anything that did not resolve was removed, along with the claim resting on it.

Does it say what is claimed?

Each in-text pinpoint (article, paragraph, threshold, penalty, instrument name) was checked against the primary source. This is not redundant: one reference had entirely correct metadata yet covered a different subject than the claim attached to it, which a metadata check passes.

What that found

Five pinpoint errors had survived an earlier identifier-level audit, among them two instruments that do not exist and one paragraph reference with no corresponding paragraph. A further reference could not be traced to any publication and was removed together with the economic figure it supported. The verification record is maintained alongside the source.

Limitations

  • No empirical validation. The claim that detectability falls monotonically from F to U is analytical, derived from the evidentiary structure of each type. It is testable, and it may be wrong.
  • Constructed scenarios. Because we constructed both the framework and the scenarios, they test internal coherence rather than external validity.
  • No enforcement record. Neither the AI Act's high-risk obligations nor the revised Product Liability Directive has produced decided cases, so claims about their application are predictions.
  • Jurisdictional scope. EU and Swiss law. Transfer to common law jurisdictions, where liability rests on negligence rather than statutory due diligence, is untested.
  • Taxonomy boundaries. The line between an unsupported inference and a fabricated fact is a matter of degree. The partition is useful because it tracks distinct control requirements, not because the categories are natural kinds.
  • Moving target. The AI Liability Directive was withdrawn while this paper was being prepared.

References

Generated from the paper's verified bibliography.

Hallucination and LLM reliability

  1. Bender, E. M., Gebru, T., McMillan-Major, A. & Shmitchell, S. (2021) On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency pp. 610–623 doi:10.1145/3442188.3445922
  2. Farquhar, S., Kossen, J., Kuhn, L. & Gal, Y. (2024) Detecting hallucinations in large language models using semantic entropy. Nature 630(8017), pp. 625–630 doi:10.1038/s41586-024-07421-0
  3. Huang, L. et al. (2025) A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Transactions on Information Systems 43(2), pp. 1–55 doi:10.1145/3703155
  4. Ji, Z. et al. (2023) Survey of Hallucination in Natural Language Generation. ACM Computing Surveys 55(12), pp. 1–38 doi:10.1145/3571730
  5. Kalai, A. T., Nachum, O., Vempala, S. S. & Zhang, E. (2025) Why Language Models Hallucinate. arXiv:2509.04664 doi:10.48550/arXiv.2509.04664
  6. Lewis, P. et al. (2020) Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401 doi:10.48550/arXiv.2005.11401
  7. Manakul, P., Liusie, A. & Gales, M. (2023) SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing pp. 9004–9017 doi:10.18653/v1/2023.emnlp-main.557
  8. Maynez, J., Narayan, S., Bohnet, B. & McDonald, R. (2020) On Faithfulness and Factuality in Abstractive Summarization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics pp. 1906–1919 doi:10.18653/v1/2020.acl-main.173
  9. Min, S. et al. (2023) FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing pp. 12076–12100 doi:10.18653/v1/2023.emnlp-main.741
  10. Rawte, V., Sheth, A. & Das, A. (2023) A Survey of Hallucination in Large Foundation Models. arXiv:2309.05922 doi:10.48550/arXiv.2309.05922
  11. Rawte, V. et al. (2023) The Troubling Emergence of Hallucination in Large Language Models – An Extensive Definition, Quantification, and Prescriptive Remediations. arXiv:2310.04988 doi:10.48550/arXiv.2310.04988
  12. Tonmoy, S. M. T. I. et al. (2024) A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models. arXiv:2401.01313 doi:10.48550/arXiv.2401.01313
  13. Weidinger, L. et al. (2022) Taxonomy of Risks posed by Language Models. 2022 ACM Conference on Fairness Accountability and Transparency pp. 214–229 doi:10.1145/3531146.3533088
  14. Xu, Z., Jain, S. & Kankanhalli, M. (2024) Hallucination is Inevitable: An Innate Limitation of Large Language Models. arXiv:2401.11817 doi:10.48550/arXiv.2401.11817
  15. Zhang, Y. et al. (2023) Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv:2309.01219 doi:10.48550/arXiv.2309.01219

Legal hallucination, empirical

  1. Dahl, M., Magesh, V., Suzgun, M. & Ho, D. E. (2024) Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models. Journal of Legal Analysis 16(1), pp. 64–93 doi:10.1093/jla/laae003
  2. Magesh, V. et al. (2025) Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Journal of Empirical Legal Studies 22(2), pp. 216–242 doi:10.1111/jels.12413

EU law

  1. European Commission (2022) Proposal for a Directive of the European Parliament and of the Council on adapting non-contractual civil liability rules to artificial intelligence (AI Liability Directive). COM(2022) 496 final Source
  2. European Union (2024) Regulation (EU) 2024/1624 of the European Parliament and of the Council of 31 May 2024 on the prevention of the use of the financial system for the purposes of money laundering or terrorist financing. OJ L, 2024/1624, 19 June 2024 Source
  3. European Union (2022) Regulation (EU) 2022/2554 of the European Parliament and of the Council of 14 December 2022 on digital operational resilience for the financial sector (Digital Operational Resilience Act). OJ L 333, 27 December 2022, pp. 1–79 Source
  4. European Union (2024) Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). OJ L, 2024/1689, 12 July 2024 Source
  5. European Union (2016) Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). OJ L 119, 4 May 2016, pp. 1–88 Source
  6. European Union (2024) Directive (EU) 2024/2853 of the European Parliament and of the Council of 23 October 2024 on liability for defective products and repealing Council Directive 85/374/EEC. OJ L, 2024/2853, 18 November 2024 Source

Swiss law

  1. Swiss Confederation (1997) Federal Act of 10 October 1997 on Combating Money Laundering and Terrorist Financing (Anti-Money Laundering Act, AMLA). SR 955.0 Source
  2. Swiss Financial Market Supervisory Authority (2015) Ordinance of 3 June 2015 of the Swiss Financial Market Supervisory Authority on Combating Money Laundering and Terrorist Financing in the Financial Sector (FINMA Anti-Money Laundering Ordinance, AMLO-FINMA). SR 955.033.0 Source
  3. Swiss Confederation (2023) Federal Act of 25 September 2020 on Data Protection (FADP). SR 235.1, in force since 1 September 2023 Source
  4. Swiss Financial Market Supervisory Authority (2024) FINMA Guidance 08/2024: Governance and risk management when using artificial intelligence. FINMA Guidance 08/2024, 18 December 2024 Source
  5. Swiss Financial Market Supervisory Authority (2025) FINMA Survey: Artificial Intelligence Gaining Traction at Swiss Financial Institutions. FINMA press release, 24 April 2025 Source

International standards

  1. Council of Europe (2024) Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law. CETS No. 225, opened for signature 5 September 2024 Source
  2. Financial Action Task Force (2021) Opportunities and Challenges of New Technologies for AML/CFT. FATF, Paris, July 2021 Source
  3. Financial Action Task Force (2023) International Standards on Combating Money Laundering and the Financing of Terrorism and Proliferation: The FATF Recommendations. FATF, Paris, adopted 2012, as amended November 2023 Source

Comparative jurisdictions

  1. Information Commissioner's Office (2023) Guidance on AI and Data Protection. ICO, Wilmslow Source
  2. Monetary Authority of Singapore (2018) Principles to Promote Fairness, Ethics, Accountability and Transparency (FEAT) in the Use of Artificial Intelligence and Data Analytics in Singapore's Financial Sector. MAS, Singapore Source
  3. Prudential Regulation Authority (2023) Model Risk Management Principles for Banks. Supervisory Statement SS1/23, Bank of England Source

AI liability and regulation scholarship

  1. Buiten, M., de Streel, A. & Peitz, M. (2023) The law and economics of AI liability. Computer Law & Security Review 48, pp. 105794 doi:10.1016/j.clsr.2023.105794
  2. Hacker, P. (2023) The European AI liability directives – Critique of a half-hearted approach and lessons for the future. Computer Law & Security Review 51, pp. 105871 doi:10.1016/j.clsr.2023.105871
  3. Hacker, P., Engel, A. & Mauer, M. (2023) Regulating ChatGPT and other Large Generative AI Models. 2023 ACM Conference on Fairness Accountability and Transparency pp. 1112–1123 doi:10.1145/3593013.3594067
  4. Novelli, C. et al. (2024) AI Risk Assessment: A Scenario-Based, Proportional Methodology for the AI Act. Digital Society 3(1) doi:10.1007/s44206-024-00095-1
  5. Veale, M. & Zuiderveen Borgesius, F. (2021) Demystifying the Draft EU Artificial Intelligence Act –- Analysing the good, the bad, and the unclear elements of the proposed approach. Computer Law Review International 22(4), pp. 97–112 doi:10.9785/cri-2021-220402

RegTech and compliance

  1. Arner, D. W., Barberis, J. & Buckley, R. P. (2017) FinTech, RegTech, and the Reconceptualization of Financial Regulation. Northwestern Journal of International Law & Business 37(3), pp. 371 Source
  2. Canhoto, A. I. (2021) Leveraging machine learning in the global fight against money laundering and terrorism financing: An affordances perspective. Journal of Business Research 131, pp. 441–452 doi:10.1016/j.jbusres.2020.10.012
  3. European Banking Authority (2023) Follow-up Report on the Use of Machine Learning for Internal Ratings-Based Models. EBA/REP/2023/28, 4 August 2023 Source
  4. Kurum, E. (2020) RegTech solutions and AML compliance: what future for financial crime? Journal of Financial Crime 30(3), pp. 776–794 doi:10.1108/jfc-04-2020-0051
  5. Zetzsche, D. A., Buckley, R. P., Arner, D. W. & Barberis, J. N. (2017) From FinTech to TechFin: The Regulatory Challenges of Data-Driven Finance. SSRN Electronic Journal doi:10.2139/ssrn.2959925

Download and cite

Download the PDF Project overview

Citation

Osterrieder, J., Baals, L., Kosterin, I., Pignon, V., Douet, N., & Baertschiger, D. (2026). When the AI hallucinates, who pays? Mapping LLM hallucination types to legal liability in EU and Swiss financial regulation. Working paper, University of Applied Sciences of the Grisons.

BibTeX

@techreport{osterrieder2026hallucination, author = {Osterrieder, Joerg and Baals, Lennart and Kosterin, Illia and Pignon, Vincent and Douet, Nathan and Baertschiger, Didier}, title = {{When the AI Hallucinates, Who Pays? Mapping LLM Hallucination Types to Legal Liability in EU and Swiss Financial Regulation}}, institution = {University of Applied Sciences of the Grisons}, year = {2026}, type = {Working paper} }