Benchmark: pgvector vs Pinecone vs Qdrant vs Weaviate
Posted by K3NCHO@reddit | programming | View on Reddit | 12 comments
Benchmarked 5 vector databases on 1M vectors at 1536 dims. Latency, recall, and cost
Latency (p50):
- Qdrant: \~3ms
- Milvus: \~4ms
- pgvector HNSW: \~5ms
- Weaviate: \~8ms
- Pinecone Serverless: \~12ms
pgvector is way faster than people think. The "Postgres is slow for vectors" thing is from the IVFFlat era. HNSW changed that
Recall:
- Qdrant: 99.2%
- Milvus: 99.0%
- pgvector: 98.5%
- Pinecone: \~95% (can't tune HNSW params)
Monthly cost (1M vectors):
- pgvector on Neon: $30-150
- Pinecone Serverless: $50-80
- Qdrant Cloud: $65-102
- pgvector on RDS: $260
We use pgvector on Neon in production. Replaced Pinecone and RDS, latency dropped from 200ms to 80ms
programming-ModTeam@reddit
This post was removed for being off topic for r/programming.
dontquestionmyaction@reddit
Afaik pgvector has been essentially unmaintained for a while now.
https://github.com/tensorchord/VectorChord/ has mostly replaced it in common use. Any reason it wasn't included in your testing?
Somepotato@reddit
Pgvector was just updated a couple months ago and doesn't have an abysmal license so no that isn't really true.
dontquestionmyaction@reddit
Vectorchord uses the Elastic License or AGPL.
Where's the issue?
K3NCHO@reddit (OP)
vectorchord is superior to pgvector in almost every way, but sadly neon doesn't support it yet which we're using as our primary db for backend and search operations across multiple regions
i have brought up implementing vectorchrod in neon to their devs, hope they consider it soon
Determinant@reddit
Nobody uses p50 numbers to analyze production performance. No company would accept horrendous performance for half of the interactions.
At a bare minimum, you should display p90 performance and most likely p95. Even better, display p50, p90, p95, and p99 to show how the system scales.
K3NCHO@reddit (OP)
it’s already displayed in blog post
Tyrannas@reddit
I see a lot of these benchmarks never inclusing Faiss, any reason for that ? Is it because it's only an index and not a real db ?
K3NCHO@reddit (OP)
yes, it's a library not a db. some vector search engines use FAISS under the hood
Tyrannas@reddit
Oh ok thanks for the clarification !
mangooreoshake@reddit
I'd love to see a Just Use Postgres version of "Just use HTML"
K3NCHO@reddit (OP)
just use Postgres to make just use HTML