Debiasing CLIP with Neural Interventions
Gómez-Grabowska, Amelia (Universitat Autònoma de Barcelona)
Gonzàlez, Jordi (Centre de Visió per Computador)
Gomez Bigorda, Lluis (Centre de Visió per Computador)

Imprint:	Cham, Switzerland : Springer, 2026
Description:	15 pàg.
Abstract:	This paper presents an inference-time method to mitigate demographic bias in CLIP-like cross-modal retrieval models through targeted neural interventions in their internal attention mechanisms. We first identify "expert" attention heads that encode demographic information by systematically analyzing CLIP's internal representations in response to labeled inputs. At inference, we intervene these heads - replacing their activations with demographic prototypes or by neutralizing them (zero ablation). We chose to intervene specifically at the CLS token, as it aggregates information globally across image patches and is directly responsible for the final image embedding. Across fairness benchmarks such as SISPI and So-B-IT, our interventions achieve bias reduction comparable to or exceeding state-of-the-art methods, while being substantially lighter and requiring no retraining.
Rights:	Aquest material està protegit per drets d'autor i/o drets afins. Podeu utilitzar aquest material en funció del que permet la legislació de drets d'autor i drets afins d'aplicació al vostre cas. Per a d'altres usos heu d'obtenir permís del(s) titular(s) de drets.
Language:	Anglès
Series:	Lecture notes in computer science ; 16485
Document:	Capítol de llibre ; recerca ; Versió acceptada per publicar
Subject:	Cross-modal Retrieval ; Fairness ; Neural Interventions
Published in:	European Conference on Information Retrieval. Delft, (48th : 2026 : Delft) , p. 413-427, ISBN 978-3-032-21324-2

DOI: 10.1007/978-3-032-21324-2_32

Available from: 2027-03-30 Postprint 15 p, 1.5 MB

The record appears in these collections:
Books and collections > Book chapters