ExplAIn - Explainable AI for High-Stakes Decisions
Making AI transparent and verifiable - tools to look inside the black box across medicine, cybersecurity and justice
Overview
Making AI transparent and verifiable - tools to look inside the black box across medicine, cybersecurity and justice
Would you trust a decision you can't understand?
AI systems are increasingly used to make decisions in high-impact contexts: medicine, cybersecurity, justice. The problem is not that these decisions are being made - it is that in most cases we cannot reconstruct the reasoning behind them. This is not a perceptual issue: it is a concrete technical limitation.
From black box to glass box
Most current models operate as opaque pipelines: given an input, they produce an output, and the intermediate process is invisible. The practical consequence is that we cannot verify what a model is relying on, identify systematic errors, or evaluate reliability in critical contexts.
Explainability does not require changing the architecture of the model. It is about adding analysis tools that let us observe and interpret behaviour from the outside, and in some cases from the inside. The goal is to move from an opaque system to an analysable one.
What ExplAIn delivers
- Methods to understand which features drive a prediction
- Detection of anomalous behaviour and bias in the data
- Outputs that are readable for non-technical stakeholders
- A path from performant to verifiable models
Why now
In 2021 the European Commission proposed the AI Act to regulate and classify AI applications by the risk of harm they can cause, placing in the high-risk category every system used in sensitive contexts that affect people's lives: education, healthcare, finance, privacy. With the AI Act about to enter into force, adopting AI systems that are reliable, robust, non-discriminating and safe will not just be the right thing to do - it will be the law.
Technical Specs
Application areas
Three concrete examples of what explainable AI acts on:
Computer Vision - Ribeiro et al.
Many classifiers rely not on semantic information but on secondary cues - for instance, a husky classified as a wolf because of the snowy background. Explanation surfaces show exactly which pixels drove the decision.
Malware Detection - Perasso et al.
Models that classify malware often pick up superficial patterns rather than truly discriminating behaviour. Integrated-gradient attributions expose where the model is actually looking.
Text Translation - Google Translate
Words are translated based on context, sometimes encoding gendered or stereotyped assumptions that are invisible in the output. Explainability tools make these dependencies visible.
What ExplAIn is and isn't
ExplAIn does not propose an alternative to AI. It develops tools to analyse and correct it - because a model that works but cannot be examined is, in practice, a tool we cannot control.
Project Partners
Università di Genova
Legacoop LiguriaTimeline
EU Commission proposes the AI Act, defining 'high-risk' AI systems TBD
Methods consolidated across vision, cybersecurity and NLP TBD
Reusable explainability toolkit aligned with AI Act compliance TBD
Gallery
No media available.