This document outlines the architecture of an end-to-end, full-duplex speech-to-speech model: what it looks like, how it is built and how it is trained. It keeps the jargon minimal, but explains clearly what the model is and how it works.
What does it mean for an end-to-end speech model?
we want a model that can take in a stream of voice (current LLMs don’t input stream), choose when to speak and reply. This model will be tuned specifically for spoken/conversational language (think difference between spoken vs written english). This also means no VAD - we suspect even gpt-4o uses VAD but cannot confirm this.
There are a few of subtle things here that need to be called out
what is VAD - why is it used? why is it bad?
VAD or voice activity detection is an audio technique that detects silences. It is used in a bunch of speech processing tasks and highly used in the cascaded systems currently used.
The problem with VAD is two fold
- to minimise latency it is generally tuned to be very jumpy and fires at the smallest silences, this can be even pauses - this is why you get the multiple fires you tend to get
- it forces the model to do an inference pass whenever it fires, so the model can’t choose when to speak
- the model can’t choose not to speak - wait for the pause to be over
- the model has no ability to interrupt the human or cut them off
input streaming inference
Most LLMs do one pass with a set of inputs then another pass, our model has the ability to have inputs streamed into the model, but why is this important? Well this solves for the problem with VAD, the model with enough data can learn to
- interrupt the user and cut them off whenever it thinks the thought is completed
- wait on pauses naturally based on the context of what the human is talking about
These are the new abilities not possible with VAD but we also expect the model to be better at handling human interruption (human interrupting the model)
Okay so to summarise the architecture - a full speech to speech model with input streaming that can do the following:
- interrupt the human New
- wait on pauses based on context not some arbitrary heuristic, if you say “gimme 2 mins, it will give you 2 mins” New
- have better human interrupting ai performance than VAD
- tuned specifically for spoken and conversational language
- have instruct capabilities on the style, tone and emotion of voice output
How is this model built?
Model architecture
At a high level, what we are doing is expanding the vocabulary of a language model to handle multi-modal tokens to enable voice in, voice out. The reason to do this is to preserve the core corpus of knowledge a LLM has about the world - the associates, the world model, its core reasoning capabilities. What we are doing is extending those associates to let it associate multimodal speech tokens with its existing latent space.
This means training this model is far cheaper than training a full LLM.
This approach of multi-modal expansion has been proven to work before with works like SpeechGPT which is a demonstration of speech in speech out model. This is a fairly bad model with a many shortcomings but is a good starting point. You can see a demo of the model here
Tangent/Context: How does Speech tokenisation work?
This is a bit of a tangent and technical dive but important context for the architecture
Text tokenisation itself is quite jank (this line of regex controls a lot of how gpt-4 works) but speech tokenisation is much worse.
Speech is a continuous waveform, so we first discretise this in chunks of fixed duration N ms. The speech signal is segmented into short, overlapping frames, each representing a small portion of the audio.
Once the speech signal is discretised, the next step is to create codebooks (vocabulary/tokens) that capture different attributes of speech. These codebooks represent various aspects of the speech signal, such as prosody (the rhythm, stress, and intonation of speech), phonemes (the basic units of sound that distinguish one word from another), and other acoustic details.
A pioneering work in this field is the HuBERT (Hidden-Unit BERT) paper, which introduced a novel approach to create discrete units from speech waveforms. HuBERT utilises self-supervised learning to discover hidden units in the speech signal, which can then be used as input to downstream tasks. One key aspect of HuBERT is that it learns a single hidden unit per time step, effectively capturing the temporal dynamics of speech.
Newer approaches of speech tokenisation tend to create many more dimensions for each time step allowing the model to represent prosody, phonemes and acoustic features independently of each other which allows for model speech with intonation and emotion much more effectively. All of the details don't need to be lost into one discrete representation but have more space to exist.
The codec this design uses is called FaCodec which is a 6-dim codec with really good sounding audio
How does SpeechGPT work?
SpeechGPT used HuBERT to create discrete speech single unit speech representations and adds these tokens to LLama vocabulary. This is done in multi-modal expansion training which allows LLama to learn the associations between language and these discrete speech representations. The model then uses HIFI-GAN to reconstruct output waveforms.
You can see the output quality here, it is impressive it works but the output is pretty poor.
What are its shortcomings?
- The voice output is quite robotic and lacks expressiveness
- This comes from how the speech is tokenized but the fact is has only one dimension to represent everything from
phonemestoprosodytoemotionsis the problem - it makes it very hard for the model to produce this expressive speech - The
HIFI-GANproduces much lower quality speech synthesis than newer diffusion models and can generalise to less types of speech/speakers
- This comes from how the speech is tokenized but the fact is has only one dimension to represent everything from
- Related to previous point, but the model can’t generalise beyond the trained voice accent and tone
- This is because the timbre and speaker embeddings aren’t neatly separated in the reconstructor the
HIFI-GAN.
- This is because the timbre and speaker embeddings aren’t neatly separated in the reconstructor the
Referencing this image again you can see how the speaker information is separated from the tokenisation, it allows the model to parse and understand speech unrelated to the speaker specific information
- This has two key implications
- INPUT
- In the input side this means it is hard for
HuBERTto understand what a speaker is saying if their accent or style of speaking is different than from what it was trained on. This is whySpeechGPTperformance on understanding how I speak is quite bad
- In the input side this means it is hard for
- OUTPUT
- there isn't sufficient capacity or granularity in a single token to independently represent and control the multiple dimensions of speech, such as prosody, semantic content, acoustic details, and speaker-specific characteristics. The entanglement of these attributes in the output representation makes it challenging to precisely manipulate individual speech qualities
- INPUT
- This model still needs a
VADsystem to operate it - we talked about why this is undesirable above - As the model is quite old now, it uses
LLama-1which is a very old and not so good model - The model was largely trained on monologues of speech rather than truly conversational data, it loses the stylistic component of spoken english
How does this design fix them?
The easy fix out of the way is train on LLama-3 but beyond that the real fixes are in updating the codec from HuBERT to something multi-dimensional. For this we choose FaCodec for speech synthesis, we found between its number of dim (6), its speaker embedding, timbre separation and diffusion based reconstructor it had the best performance on what we were looking for.
To just repeat myself a bit, having 6 dimensions allows the model to learn to generate prosody and acoustic details in full resolution along with the content of the speech unlike something like HuBERT
Here are some audio examples of how it sounds better
- This is a demo of taking content from one audio and tone, intonation from another audio clip
- This is a demo of taking content and tone from one audio, with speaker embedding from another
This codec change, along with better data solves for all the other defects too from not needing VAD to conversational style.
Now there are a bunch of challenges in multi-dimensional output and want to break those down in detail
Challenges in fixing them
Output Challenges
First, let’s talk about the challenges in generation output. A LLM generate one token out per time-step, but if we have N dim which are correlated but not sequential in time how do you produce this output? Also what about the speed of generating 6 times the tokens to generate one time step.
The solution for this is multiple LM head and embedding layers one for each feature. Facodec has three features prosody, content and acoustic details. This way the model can learn the output associates from each layer independently and output them at the same time. This technique is seen in here and in Parler-TTS
Expanding on technical a bit
Specifically here, the design builds a delay pattern mask for features with multiple dimensions like
contentandacoustic detailsthis technique can be seen inMusicGEN
Input Challenges
For inputs into the model, the 6 dimensions per time-step poses a challenge too. It is dramatically reducing the context window by multiplying the input essentially by 6, it also has this correlated but not sequential problem.
There are two potential solutions for this (a lot of this is empirical)
- The understanding hit from 6 codecs won’t be that high and it should be fine - we compensate for this with good inference optimisation to ensure high throughput and speed
- Use a audio feature embedding tool like
Wav2Vecthat generate feature embeddings and adds them directly to the input embeddings, skipping tokenisation all together. This is what something likeGazelleuses
The advantages for Wav2Vec is higher performance and this correlated but not sequential problem doesn't come up, the key disadvantage is every dimension of the input speech is not explicitly represented like this but rather in the embedding’s latent space.
The first version of the model uses a Wav2Vec + Adapter as we had already trained similar models and knew it would work, with direct tokenisation as an experiment in other runs
TLDR on Model Architecture
SpeechGPT like multi-modal expanded model using LLama-3 and FaCodec that has 3 LM heads with 3 output embedding layers. Which uses a Wav2Vec + Adapter for input embeddings.
Data
The most important thing isn’t architecture but data, and here is, at a high level, how we think about data to create the type of model described above.
The model trains on mostly conversational data → think podcasts, youtube, movies, call recordings etc. We had about 1.5 million hours of this data and were collecting a lot more.
There are two classes of data here
Multi-Modal expansion data
This is singularly focused on teaching the model representations of speech token in its existing latent space. This can be large unstructured corpuses of data like
VoxPopuli(European parliament recordings)Conversational data
This is focused on teaching the model conversational style, fluidity and when to speak, when to interrupt. Here quality matters much more than quantity, we had tens of thousands of hours of this data too mainly from podcasts, and youtube. Tho a high quality subset is what gets used
There are a few key things to highlight about the data representation
- the model predicts
<interupt>tokens of when it should speak, these are added to the data for the model while it is trained - temporal tokens represent pauses and waiting for time - this allows the model to learn to pause and wait based on context
- the data has rapid back and forth with overlapping speech with “um’s” “ahhs”, teaching the model to naturally backchannel
What it costs to train
For reference, here is the cost of training SpeechGPT, a model with a single dimension of speech tokens and a much smaller dataset. That training run cost $15k
| Setting | Stage 1 | Stage 2 | Stage 3 |
|---|---|---|---|
| Batch size | 768 | 1536 | 128 |
| Peak learning rate | 2e-4 | 2e-4 | 2e-4 |
| Max length | 1024 | 512 | 1024 |
| Training steps | 900 | 4000 | 4200 |
| LoRA rank | - | - | 8 |
| LoRA alpha | - | - | 16 |
| Trainable parameters | 13B | 13B | 6M |
| Training device | 96 × A100 | 96 × A100 | 8 × A100 |
Runs of this model are expected to cost between $50k and $100k. In a series of models there are different sizes, from 3B to 70B, and the cost varies with them - the first model in the series is an 8B model based on LLama-3 8B
Training
Training uses accelerate for multi-node training on https://sfcompute.com/ (or similar)
The reason for choosing accelerate over other options is because we have been directly collaborating with the speech-research team at huggingface, plus both LLama3 code and SpeechGPT are already in torch making it much quicker to train.
Summary
This document explains in detail what the model is, and how it is built and trained.
It describes the base model; the action-tuning and trajectory training is post-training on these models.
Appendix
Evaluating Other Approaches
Adapters all the way
Could we now have an output adapter the same way we have an input adapter? One that takes the output text embedding matrix and converts to input tts embedding matrix?
There is a key problem here, one that is maybe solvable but will require a lot of data and compute for something that may or may not be learnable.
The constraint we are keeping here is to have a consistent forward pass, no buffer no cascading, matrix multiplications all the way down.
Let’s walk through the problem
At each time-step the LLM outputs n logit for n inputs, we want the nth logit which is the output.
Any given TTS system is trained to take m-tokens and output m-logits. A single token can represent anything → here 71 → h and 15339 → hello
h hello
hello
Show whitespace
h and hello. Three tokens: 71 for h, 198 for the newline, 15339 for hello.Now for the TTS system to work you have to develop 1→ 1 mapping from logits to discrete speech tokens spacing. One text token say h might represent no speech or hello might represent many phonemes of speech.
Now there are some ideas we have to solve this problem, but the reason we were considering it in the first place was this could result in mvp models for an order of magnitude less compute. To solve this mapping, will likely take a lot of compute and data resulting in a model that performs worse than our multi-modal expansion model.
Cascaded Systems
The above is a cascaded system which produces voice output
There are two key flaws in cascaded systems
- latency, there is a lot of hopping between services and that always come with cold-starts, http latency and more
- very lossy - at each step there is a huge loss of information, while transcribing there is a lot of emotive or expressiveness, while generating speech that information isn’t there per token but at best as a high level description
The advantage
- lot of interop and choice in which tools and systems to use
- can use closed-source SOTA
Glossary
thank you claude
prosody- The patterns of stress, intonation, and rhythm in speech. Prosodic features include pitch, loudness, and duration, which convey linguistic and emotional information beyond the basic phonemic content.
phonemes- The smallest units of sound that distinguish one word from another in a language. For example, the words "bat" and "pat" differ by a single phoneme - /b/ vs /p/. Acoustic models often aim to recognise or generate speech at the phoneme level.
timbre- The quality of a sound that distinguishes different types of sound production, such as voices or musical instruments. It's the property that allows you to differentiate between two sounds with the same pitch and loudness. In audio synthesis, capturing and controlling timbre is important for generating realistic and expressive sounds.
codec- Short for coder-decoder, a codec is an algorithm or device that compresses and decompresses audio or video data for efficient storage or transmission. In machine learning, neural audio codecs can be used to compress and reconstruct audio in a way that's optimised for the target application, such as speech or music.
backchannel- Listener responses during a conversation, such as "mhm", "uh-huh", or nods, which signal attention, understanding, or agreement without interrupting the speaker. Generating appropriate backchannels is a challenge in dialogue systems and conversational AI.
logit- The raw, unnormalised output of a neural network before applying an activation function like softmax. Logits represent the model's predicted log-odds or likelihood for each possible class or outcome. In audio models, logits are often used as intermediate representations before sampling or decoding to generate waveforms or spectrograms.