Skip to the memo The original Scalar editor is at scalar.video
scalar

Memo June 2024

Full-duplex speech, end to end

The architecture of an end-to-end, full-duplex speech model: one model that hears a continuous stream of voice, decides for itself when to speak, wait or interrupt, and replies in speech.

Contents

    This document outlines the architecture of an end-to-end, full-duplex speech-to-speech model: what it looks like, how it is built and how it is trained. It keeps the jargon minimal, but explains clearly what the model is and how it works.

    What does it mean for an end-to-end speech model?

    we want a model that can take in a stream of voice (current LLMs don’t input stream), choose when to speak and reply. This model will be tuned specifically for spoken/conversational language (think difference between spoken vs written english). This also means no VAD - we suspect even gpt-4o uses VAD but cannot confirm this.

    Fig. 1Current SystemsDiagram
    Current systems: the user sends a webrtc stream of audio to a buffer, then to VAD, which forces a fire every time into ASR (whisper), then the LLM, then TTS, which produces speech streamed back to the user. user stream of audio webrtc stream Buffer VAD Forces a Fire every time ASR (whisper) LLM TTS speech stream
    Current systems. Audio is buffered, and every time VAD fires it forces a pass through ASR, the LLM and TTS before speech streams back.
    Fig. 2Our SystemDiagram
    Our system: the user sends a webrtc stream of audio to our model, which runs continuous inference and outputs either silence or speech; speech goes back to the user as a webrtc stream. user webrtc stream stream of audio our model continuous inference silence speech webrtc stream
    Our system. One model takes the stream of audio, runs continuous inference, and at each step puts out either silence or speech.

    There are a few of subtle things here that need to be called out

    • what is VAD - why is it used? why is it bad?

      VAD or voice activity detection is an audio technique that detects silences. It is used in a bunch of speech processing tasks and highly used in the cascaded systems currently used.

      The problem with VAD is two fold

      • to minimise latency it is generally tuned to be very jumpy and fires at the smallest silences, this can be even pauses - this is why you get the multiple fires you tend to get
      • it forces the model to do an inference pass whenever it fires, so the model can’t choose when to speak
        • the model can’t choose not to speak - wait for the pause to be over
        • the model has no ability to interrupt the human or cut them off
    • input streaming inference

      Most LLMs do one pass with a set of inputs then another pass, our model has the ability to have inputs streamed into the model, but why is this important? Well this solves for the problem with VAD, the model with enough data can learn to

      • interrupt the user and cut them off whenever it thinks the thought is completed
      • wait on pauses naturally based on the context of what the human is talking about

      These are the new abilities not possible with VAD but we also expect the model to be better at handling human interruption (human interrupting the model)

    Okay so to summarise the architecture - a full speech to speech model with input streaming that can do the following:

    • interrupt the human New
    • wait on pauses based on context not some arbitrary heuristic, if you say “gimme 2 mins, it will give you 2 mins” New
    • have better human interrupting ai performance than VAD
    • tuned specifically for spoken and conversational language
    • have instruct capabilities on the style, tone and emotion of voice output

    How is this model built?

    Model architecture

    At a high level, what we are doing is expanding the vocabulary of a language model to handle multi-modal tokens to enable voice in, voice out. The reason to do this is to preserve the core corpus of knowledge a LLM has about the world - the associates, the world model, its core reasoning capabilities. What we are doing is extending those associates to let it associate multimodal speech tokens with its existing latent space.

    This means training this model is far cheaper than training a full LLM.

    This approach of multi-modal expansion has been proven to work before with works like SpeechGPT which is a demonstration of speech in speech out model. This is a fairly bad model with a many shortcomings but is a good starting point. You can see a demo of the model here

    Tangent/Context: How does Speech tokenisation work?

    This is a bit of a tangent and technical dive but important context for the architecture

    Fig. 3TokenizationScreenshot
    A post on X by Andrej Karpathy, 20 February 2024: “We will see that a lot of weird behaviors and problems of LLMs actually trace back to tokenization. We'll go through a number of these issues, discuss why tokenization is at fault, and why someone out there ideally finds a way to delete this stage entirely.” It includes a slide titled “Tokenization is at the heart of much weirdness of LLMs. Do not brush it off.” listing questions, each answered “Tokenization”: why can't LLMs spell words, do simple string processing like reversing a string, why they are worse at non-English languages and simple arithmetic, why GPT-2 had trouble coding in Python, why an LLM halts at the string endoftext, the trailing whitespace warning, SolidGoldMagikarp, why to prefer YAML over JSON, why LLMs are not actually end-to-end language modeling, and what is the real root of suffering.
    Andrej Karpathy on tokenization, posted on X on 20 February 2024.

    Text tokenisation itself is quite jank (this line of regex controls a lot of how gpt-4 works) but speech tokenisation is much worse.

    Speech is a continuous waveform, so we first discretise this in chunks of fixed duration N ms. The speech signal is segmented into short, overlapping frames, each representing a small portion of the audio.

    Once the speech signal is discretised, the next step is to create codebooks (vocabulary/tokens) that capture different attributes of speech. These codebooks represent various aspects of the speech signal, such as prosody (the rhythm, stress, and intonation of speech), phonemes (the basic units of sound that distinguish one word from another), and other acoustic details.

    A pioneering work in this field is the HuBERT (Hidden-Unit BERT) paper, which introduced a novel approach to create discrete units from speech waveforms. HuBERT utilises self-supervised learning to discover hidden units in the speech signal, which can then be used as input to downstream tasks. One key aspect of HuBERT is that it learns a single hidden unit per time step, effectively capturing the temporal dynamics of speech.

    Newer approaches of speech tokenisation tend to create many more dimensions for each time step allowing the model to represent prosody, phonemes and acoustic features independently of each other which allows for model speech with intonation and emotion much more effectively. All of the details don't need to be lost into one discrete representation but have more space to exist.

    The codec this design uses is called FaCodec which is a 6-dim codec with really good sounding audio

    Fig. 4FaCodec and HuBERTDiagram
    One 20 ms time step, two ways. FaCodec gives six tokens for prosody, content and acoustic detail, and keeps the speaker in separate embeddings. HuBERT gives a single unit for all of it.

    How does SpeechGPT work?

    SpeechGPT used HuBERT to create discrete speech single unit speech representations and adds these tokens to LLama vocabulary. This is done in multi-modal expansion training which allows LLama to learn the associations between language and these discrete speech representations. The model then uses HIFI-GAN to reconstruct output waveforms.

    You can see the output quality here, it is impressive it works but the output is pretty poor.

    What are its shortcomings?

    • The voice output is quite robotic and lacks expressiveness
      • This comes from how the speech is tokenized but the fact is has only one dimension to represent everything from phonemes to prosody to emotions is the problem - it makes it very hard for the model to produce this expressive speech
      • The HIFI-GAN produces much lower quality speech synthesis than newer diffusion models and can generalise to less types of speech/speakers
    • Related to previous point, but the model can’t generalise beyond the trained voice accent and tone
      • This is because the timbre and speaker embeddings aren’t neatly separated in the reconstructor the HIFI-GAN.

    Referencing this image again you can see how the speaker information is separated from the tokenisation, it allows the model to parse and understand speech unrelated to the speaker specific information

    Fig. 4, againThe speaker pathDiagram
    The same figure, speaker path picked out. FaCodec’s timbre extractor moves the speaker out of the tokens; HuBERT’s one unit carries the speaker along with everything else.
    • This has two key implications
      • INPUT
        • In the input side this means it is hard for HuBERT to understand what a speaker is saying if their accent or style of speaking is different than from what it was trained on. This is why SpeechGPT performance on understanding how I speak is quite bad
      • OUTPUT
        • there isn't sufficient capacity or granularity in a single token to independently represent and control the multiple dimensions of speech, such as prosody, semantic content, acoustic details, and speaker-specific characteristics. The entanglement of these attributes in the output representation makes it challenging to precisely manipulate individual speech qualities
    • This model still needs a VAD system to operate it - we talked about why this is undesirable above
    • As the model is quite old now, it uses LLama-1 which is a very old and not so good model
    • The model was largely trained on monologues of speech rather than truly conversational data, it loses the stylistic component of spoken english

    How does this design fix them?

    The easy fix out of the way is train on LLama-3 but beyond that the real fixes are in updating the codec from HuBERT to something multi-dimensional. For this we choose FaCodec for speech synthesis, we found between its number of dim (6), its speaker embedding, timbre separation and diffusion based reconstructor it had the best performance on what we were looking for.

    To just repeat myself a bit, having 6 dimensions allows the model to learn to generate prosody and acoustic details in full resolution along with the content of the speech unlike something like HuBERT

    Here are some audio examples of how it sounds better

    • This is a demo of taking content from one audio and tone, intonation from another audio clip
    • This is a demo of taking content and tone from one audio, with speaker embedding from another

    This codec change, along with better data solves for all the other defects too from not needing VAD to conversational style.

    Now there are a bunch of challenges in multi-dimensional output and want to break those down in detail

    Challenges in fixing them

    Output Challenges

    First, let’s talk about the challenges in generation output. A LLM generate one token out per time-step, but if we have N dim which are correlated but not sequential in time how do you produce this output? Also what about the speed of generating 6 times the tokens to generate one time step.

    The solution for this is multiple LM head and embedding layers one for each feature. Facodec has three features prosody, content and acoustic details. This way the model can learn the output associates from each layer independently and output them at the same time. This technique is seen in here and in Parler-TTS

    • Expanding on technical a bit

      Specifically here, the design builds a delay pattern mask for features with multiple dimensions like content and acoustic details this technique can be seen in MusicGEN

    Input Challenges

    For inputs into the model, the 6 dimensions per time-step poses a challenge too. It is dramatically reducing the context window by multiplying the input essentially by 6, it also has this correlated but not sequential problem.

    There are two potential solutions for this (a lot of this is empirical)

    • The understanding hit from 6 codecs won’t be that high and it should be fine - we compensate for this with good inference optimisation to ensure high throughput and speed
    • Use a audio feature embedding tool like Wav2Vec that generate feature embeddings and adds them directly to the input embeddings, skipping tokenisation all together. This is what something like Gazelle uses
    Fig. 5Two ways inDiagram
    Two ways to get a single 20 ms time step of speech into the model. Top: FaCodec turns it into a tokenized vector of 6 tokens per time step, which goes through the embedding matrix to the input embeddings. Bottom: Wav2Vec turns it into an audio feature vector, an adapter turns that into adapted embeddings, and these are combined with the text input embeddings to form the input embeddings. tokenized vector 6 x tokens / time step embedding matrix input embeddings {1015325 40692916} FaCodec Single time step (20ms) Wav2Vec Audio feature vector Adapter Adapted Embeddings text input embeddings input embeddings
    High level illustration of both approaches. Top, in violet: FaCodec tokens through an embedding matrix. Bottom, in gold: Wav2Vec features through an adapter, joined with the text input embeddings.

    The advantages for Wav2Vec is higher performance and this correlated but not sequential problem doesn't come up, the key disadvantage is every dimension of the input speech is not explicitly represented like this but rather in the embedding’s latent space.

    The first version of the model uses a Wav2Vec + Adapter as we had already trained similar models and knew it would work, with direct tokenisation as an experiment in other runs

    TLDR on Model Architecture

    SpeechGPT like multi-modal expanded model using LLama-3 and FaCodec that has 3 LM heads with 3 output embedding layers. Which uses a Wav2Vec + Adapter for input embeddings.

    Data

    The most important thing isn’t architecture but data, and here is, at a high level, how we think about data to create the type of model described above.

    The model trains on mostly conversational data → think podcasts, youtube, movies, call recordings etc. We had about 1.5 million hours of this data and were collecting a lot more.

    There are two classes of data here

    1. Multi-Modal expansion data

      This is singularly focused on teaching the model representations of speech token in its existing latent space. This can be large unstructured corpuses of data like VoxPopuli (European parliament recordings)

    2. Conversational data

      This is focused on teaching the model conversational style, fluidity and when to speak, when to interrupt. Here quality matters much more than quantity, we had tens of thousands of hours of this data too mainly from podcasts, and youtube. Tho a high quality subset is what gets used

    There are a few key things to highlight about the data representation

    • the model predicts <interupt> tokens of when it should speak, these are added to the data for the model while it is trained
    • temporal tokens represent pauses and waiting for time - this allows the model to learn to pause and wait based on context
    • the data has rapid back and forth with overlapping speech with “um’s” “ahhs”, teaching the model to naturally backchannel

    What it costs to train

    For reference, here is the cost of training SpeechGPT, a model with a single dimension of speech tokens and a much smaller dataset. That training run cost $15k

    Fig. 6Training SpeechGPTTable
    SettingStage 1Stage 2Stage 3
    Batch size7681536128
    Peak learning rate2e-42e-42e-4
    Max length10245121024
    Training steps90040004200
    LoRA rank--8
    LoRA alpha--16
    Trainable parameters13B13B6M
    Training device96 × A10096 × A1008 × A100
    Training SpeechGPT, stage by stage. Batch size, schedule, LoRA settings and hardware for each of its three stages.

    Runs of this model are expected to cost between $50k and $100k. In a series of models there are different sizes, from 3B to 70B, and the cost varies with them - the first model in the series is an 8B model based on LLama-3 8B

    Training

    Training uses accelerate for multi-node training on https://sfcompute.com/ (or similar)

    The reason for choosing accelerate over other options is because we have been directly collaborating with the speech-research team at huggingface, plus both LLama3 code and SpeechGPT are already in torch making it much quicker to train.

    Summary

    This document explains in detail what the model is, and how it is built and trained.

    It describes the base model; the action-tuning and trajectory training is post-training on these models.

    Appendix

    Evaluating Other Approaches

    Adapters all the way

    Could we now have an output adapter the same way we have an input adapter? One that takes the output text embedding matrix and converts to input tts embedding matrix?

    There is a key problem here, one that is maybe solvable but will require a lot of data and compute for something that may or may not be learnable.

    The constraint we are keeping here is to have a consistent forward pass, no buffer no cascading, matrix multiplications all the way down.

    Let’s walk through the problem

    Fig. 7An output adapterDiagram
    input_ids and audio_values go into an ASR plus LLM model, which outputs n logits for n inputs; an adapter passes them to Parler, which takes m tokens and outputs m logits. input_ids audio_values ASR+ LLM n logits (for n inputs) adapter Parler takes m-tokens -> m-logits
    Adapters all the way. An adapter would have to turn the LLM’s n logits into the m tokens a TTS model like Parler expects.

    At each time-step the LLM outputs n logit for n inputs, we want the nth logit which is the output.

    Any given TTS system is trained to take m-tokens and output m-logits. A single token can represent anything → here 71 → h and 15339 → hello

    Fig. 8TiktokenizerExample
    Tiktokenizer meta-llama/Meta-Llama-3-8B
    h
    hello
    Token count3
    h\n
    hello
    71, 198, 15339

    Show whitespace

    Llama 3’s tokenizer on h and hello. Three tokens: 71 for h, 198 for the newline, 15339 for hello.

    Now for the TTS system to work you have to develop 1→ 1 mapping from logits to discrete speech tokens spacing. One text token say h might represent no speech or hello might represent many phonemes of speech.

    Now there are some ideas we have to solve this problem, but the reason we were considering it in the first place was this could result in mvp models for an order of magnitude less compute. To solve this mapping, will likely take a lot of compute and data resulting in a model that performs worse than our multi-modal expansion model.

    Cascaded Systems

    Fig. 9Cascaded SystemDiagram
    A cascaded system: the human sends an audio stream or chunks to ASR (Deepgram or AssemblyAI); the transcript goes to the LLM (OpenAI); its output text goes to ElevenLabs; the audio output goes back to the human. Cascaded System HUMAN audio stream/chunks ASR Deepgram AssemblyAI transcript LLM OpenAI output text ElevenLabs audio output
    A cascaded system. Audio goes to an ASR service, the transcript to an LLM, its output text to ElevenLabs, and audio comes back to the human.

    The above is a cascaded system which produces voice output

    There are two key flaws in cascaded systems

    • latency, there is a lot of hopping between services and that always come with cold-starts, http latency and more
    • very lossy - at each step there is a huge loss of information, while transcribing there is a lot of emotive or expressiveness, while generating speech that information isn’t there per token but at best as a high level description

    The advantage

    • lot of interop and choice in which tools and systems to use
    • can use closed-source SOTA

    Glossary

    thank you claude

    prosody
    The patterns of stress, intonation, and rhythm in speech. Prosodic features include pitch, loudness, and duration, which convey linguistic and emotional information beyond the basic phonemic content.
    phonemes
    The smallest units of sound that distinguish one word from another in a language. For example, the words "bat" and "pat" differ by a single phoneme - /b/ vs /p/. Acoustic models often aim to recognise or generate speech at the phoneme level.
    timbre
    The quality of a sound that distinguishes different types of sound production, such as voices or musical instruments. It's the property that allows you to differentiate between two sounds with the same pitch and loudness. In audio synthesis, capturing and controlling timbre is important for generating realistic and expressive sounds.
    codec
    Short for coder-decoder, a codec is an algorithm or device that compresses and decompresses audio or video data for efficient storage or transmission. In machine learning, neural audio codecs can be used to compress and reconstruct audio in a way that's optimised for the target application, such as speech or music.
    backchannel
    Listener responses during a conversation, such as "mhm", "uh-huh", or nods, which signal attention, understanding, or agreement without interrupting the speaker. Generating appropriate backchannels is a challenge in dialogue systems and conversational AI.
    logit
    The raw, unnormalised output of a neural network before applying an activation function like softmax. Logits represent the model's predicted log-odds or likelihood for each possible class or outcome. In audio models, logits are often used as intermediate representations before sampling or decoding to generate waveforms or spectrograms.

    After the memo

    We later built bench.audio to put voice agents head to head.

    Try it Write to us about voice