1 University of Illinois Urbana-Champaign, Urbana, IL, USA 2 National Center for Supercomputing Applications, Urbana, IL, USA
Accepted as an Oral at Learning to Listen: ICML 2026 Workshop on Machine Learning for Audio
Synthetic accented speech is a promising way to improve automatic speech recognition (ASR) when real accented recordings are scarce. We ask what makes such data useful for ASR fine-tuning: target-accent phoneme edits that expose the recognizer to accent-specific pronunciations, or random phoneme perturbations that act as augmentation in phoneme space. In a few-shot TTS pipeline, we compare LLM-generated accent edits with matched-rate random substitutions and oracle controls using ground-truth accented phonemes and prosody. Random substitutions recover much of the ASR gain: LLM target-accent edits improve over random by only a small margin, ground-truth phonemes stay close to the random baseline and nearly converge with it as the synthetic ASR fine-tuning set grows larger, and adding ground-truth prosody yields only a modest further gain. Mixing synthetic with real accented speech also stabilizes low-resource fine-tuning, but a fixed synthetic budget can later dilute the information in real data, showing that the real–synthetic ratio matters.
The audio samples below correspond to the conditions reported in Table 1 of the paper.
Backbone TTS with a Standard American speaker — no adaptation, no phoneme editing.
TTS decoder adapted to the target-accent speaker from fewer than ten reference utterances; phoneme sequence unchanged.
Adapted decoder with phonemes edited by an LLM toward the target accent.
Adapted decoder with random, matched-rate phoneme perturbations — baseline isolating the augmentation effect.
Adapted decoder with ground-truth accented phonemes from the L2-ARCTIC perceived phoneme labels (oracle upper bound).
Original recordings from the target-accent speaker (TNI for Indian English, HKK for Korean English).
All conditions use the same six L2-ARCTIC prompts. Synthetic audio is produced by the same pipeline used to generate the training corpus reported in the paper.
| Transcript | American TTS | Adapt-only | Adapt + LLM | Adapt + Random | Adapt + GT | Real accent |
|---|
| Transcript | American TTS | Adapt-only | Adapt + LLM | Adapt + Random | Adapt + GT | Real accent |
|---|
Below is the full prompt used for the Indian English condition, including the in-context American → Indian English example pairs that are rendered into the prompt at inference time. The Korean condition uses the same template with the accent name swapped and a Korean-specific set of demonstrations.
Loading prompt…