PaperfootJoin free
arXiv

Speech emotion models lose accuracy on unfamiliar speakers

Across 21 tests, accuracy dropped by 6.9 to 63.1 percentage points when models faced new speakers compared to control tests using standard random splits.

Paper by Prince Baah-Mensah, Dennis Owusu Asamoah
Preprint: empirical study

Edited by Boris Djordjevic · Paperfoot

Paper

In a preprint, researchers introduced a two-condition audit protocol to evaluate seven speech emotion recognition model configurations from five published papers across three datasets, with a fourth corpus as an external check. The configurations spanned convolutional, recurrent, transformer, vision-transformer, and self-supervised wav2vec2 architectures. The researchers evaluated each model twice under identical recipes: a control condition reproducing common random splits, and a strict condition with disjoint speakers.\n\nPublished headline numbers reproduced close to the control condition. Across 21 model-dataset pairs, however, accuracy dropped by 6.9 to 63.1 percentage points under strict speaker independence. The results quantify speaker memorization as a dominant, previously unmeasured component of published performance, indicating that models may be learning speaker identity and tone rather than generalized emotional cues.

Why it matters

Across 21 model-dataset pairs on speech benchmarks, the study quantifies speaker memorization as a dominant, previously unmeasured component of published performance. The evidence supports the authors' recommendation of two-condition reporting as a community standard to test whether systems learn generalized cues rather than speaker identity.

Caveats

The evaluation is limited to four speech datasets, and the abstract does not report whether the scores reflect multiple training runs or how individual architectures performed across the range of drops. The study is a preprint that has not yet been peer-reviewed.

In this paper

speech emotion recognition

Methods
wav2vec2 with a dense head (evaluated)EmoFormer CNN-transformer (evaluated)1D-CNN with fully connected layers (Model A) (evaluated)LSTM (Model B) (evaluated)GRU (Model C) (evaluated)BLSTM-DSA (evaluated)mel-spectrogram vision transformer (evaluated)
Datasets
RAVDESS (evaluated)EMO-DB (evaluated)SAVEE (evaluated)EARS (evaluated)

What it shows

Published headline SER accuracies reproduce under leaked conditions but drop by 6.9 to 63.1 percentage points under strict speaker independence across 21 model-dataset pairs.

Across 21 model-dataset pairs, published headline numbers reproduce close to the control but drop by 6.9 to 63.1 percentage points under strict speaker independence.

Abstract

The accuracy collapse tracks learning under leakage, where from-scratch models with high control accuracy suffer severe drops under strict SI while low-gap from-scratch models underfit both conditions.

Third, the collapse tracks how much the model learned under leakage: from-scratch models with high control accuracy suffer severe drops under strict SI, whereas low-gap from-scratch models simply underfit both conditions.

I Introduction

Speaker memorization is driven by evaluation protocols and data augmentation rather than specific recurrent heads.

All three exhibit nearly identical control accuracies and nearly identical SI collapses on every dataset, proving that the memorization effect is driven by the evaluation protocol and data augmentation, not by a specific recurrent head.

IV-D Robustness Notes

When restricted to original four-emotion subsets, models achieve higher control accuracies but their strict speaker-independent accuracies remain strictly lower.

When the Kumar ViT and Li models are restricted to the four-emotion subsets used in their original papers, their control accuracies rise, but their SI accuracies remain strictly lower, preserving the gap.

IV-D Robustness Notes

Quoted word for word from the full text on arXiv.

The paper

Beyond Accuracy: What Are Speech Emotion Recognition Models Learning?

Prince Baah-Mensah, Dennis Owusu Asamoah

arXiv, Oct 24, 2026, Open access, Preprint, not peer-reviewed

doi.org/10.48550/arxiv.2610.33952

arXiv:2610.33952v1 in eess.AS

AbstractPDFHTML

Comments: 6 pages, 4 tables

Cite as BibTeX
@misc{baahmensah2026beyond,
  title = {Beyond Accuracy: What Are Speech Emotion Recognition Models Learning?},
  author = {Baah-Mensah, Prince and Asamoah, Dennis Owusu},
  year = {2026},
  eprint = {2610.33952},
  archivePrefix = {arXiv},
  primaryClass = {eess.AS},
  doi = {10.48550/arxiv.2610.33952},
  url = {https://arxiv.org/abs/2610.33952}
}

Related