Research

Multimodal systems are usually evaluated in conditions they will never meet: every modality present, every input clean, every speaker in the same language. My work asks what these systems actually learn, and what happens to them when those assumptions break.

Multimodal & cross-modal representation learning

Conventional multimodal architectures dedicate a separate network to each modality. We have shown repeatedly that a single branch can carry several modalities at once — for multimodal training (ICASSP 2023), for recommendation under cold-start and missing modalities (ACM RecSys 2024, ACM TORS 2025), and as a parameter-efficient collaborative branch (ACM RecSys 2025). The recurring finding is that the modality gap is an architectural artefact more than an inherent property of the data.

On the geometry side, Face-Voice Association with Inductive Bias for Maximum Class Separation (ICASSP 2026) replaces learned class prototypes with a fixed simplex, which both removes parameters and improves separation in the joint space.

Robustness under missing, noisy and corrupted inputs

A model that needs all of its inputs is not deployable. Robust Harmful Meme Detection under Missing Modalities via Shared Representation Learning (ACM Web Conference 2026) formulates a shared representation that degrades gracefully when a modality is absent at inference rather than collapsing. SB-BEVFusion (IEEE ICIP 2026) carries the same idea into autonomous-driving perception, under sensor malfunction and corruption.

Vision–language and foundation-model evaluation

Large vision–language models are typically benchmarked on English, Western-centric data. From Native Memes to Global Moderation (ACM Web Conference 2026) evaluates them cross-culturally on hateful-meme detection, where the gap between reported and real-world performance is largest. The FAME and FLAG challenge series extends this to multilingual face–voice association, and the 2027 edition adds gender-controlled evaluation to test whether models learn identity-specific structure or demographic shortcuts.

Efficient learning

Distillation-based layer dropping (ICASSP 2026) and light-weight ASR training frameworks (Interspeech 2025) target dynamic speech networks that must trade accuracy against compute at inference, not at design time.

Machine learning for science

At DESY I led interpretable-ML methodology for X-ray detector data: Explainable Machine Learning for Diffraction Patterns (J. Appl. Cryst. 2023) and the data-reduction line that followed (J. Appl. Cryst. 2023, 2024), now used for reduction on large-scale serial-crystallography experiments.