Zonos-v0.1 beta by Zyphra, featuring two expressive and real-time text-to-speech (TTS) models with high-fidelity voice cloning. 1.6B transformer and 1.6B hybrid under an Apache 2.0 license.

Hey, appreciate the detailed feedback! > The first is if you feed more than one short paragraph of text, it immediately becomes completely unstable We trained on snippets of up to 30 seconds and our current architecture doesn't generalize well to longer sequences, so if you feed it too much text at once it will break yeah. We have some checkpoints trained on longer sequences that we might release at some point, but for now I'd recommend generating chunks <30s. > The second is that voice cloning frankly does not sound very similar to the original voice, and is pitched down. It's honestly not nearly as good as other solutions, which is a pity. Yeah we've found that some voices don't work well. Particularly ones recorded in poor acoustic environments or where there is significant background noise. We'll try to release a tiny complementary model for cleaning up speaker embeddings in the coming days. > The third is that even if you voice clone, no matter how much you mess with emotion sliders, it is unable to reproduce the intonation and manner of speech of the original, having little dynamic range and sounding downright depressed or monotone. This is very unfortunate, as it makes voice cloning even further from the original. Did you try playing with the "Pitch Std" slider? The current default of 20 is quite low and won't be expressive. To get more accurate voice cloning you might want to tick some of the unconditional toggles. We've found that setting Emotion, Pitch Std, and Speaking Rate to unconditional can help a lot here, though at the cost of control and perhaps some stability. Cloning from prefix audio tends to yield the best results, but requires you to manually transcribe the clip and put it before your text prompt. > I tried both models, but found there to be little difference in these properties, with the hybrid model sounding a tad more coherent. This is definitely a groundbreaking work, and with some refinement could easily become the OS SOTA. I'm just disappointed I'm gonna have to wait a while before this is usable in my applications I think at least some of these issues stem from the inference code being a bit weird and having bad defaults. We'll be working on this over the next few days to get it on par with the API (which currently hosts the Transformer model).