I’ve been working on a voice agent project recently and quickly realized that building the pipeline (STT → LLM → TTS) is the easy part. The real challenge is evaluation, making sure the system performs reliably across accents, contexts, and multi-turn conversations.

I went down the rabbit hole of voice eval tools and here are the ones I found most useful:

1. Deepgram Eval
2. Strong for transcription accuracy testing.
3. Provides detailed WER (word error rate) metrics and error breakdowns.
4. Speechmatics
5. I used this mainly for multilingual evaluation.
6. Handles accents/dialects better than most engines I tested.
7. Voiceflow Testing
8. Focused on evaluating conversation flows end-to-end.
9. Helpful when testing dialogue design beyond just turn-level accuracy.
10. Play.h.t Voice QA
11. More on the TTS side, quality and naturalness of synthetic voices.
12. Useful if you care about voice fidelity as much as the NLP part.
13. Maxim AI
14. This stood out because it let me run structured evals on the whole voice pipeline.
15. Latency checks, persona-based stress tests, and pre/post-release evaluation of agents.
16. Felt much closer to “real user” testing than just measuring WER.

I’d love to hear if anyone here has explored other approaches to systematic evaluation of voice agents, especially for multi-turn robustness or human-likeness metrics.