Scaling a Websocket server with a Kafka consumer
Posted by Bayul@reddit | ExperiencedDevs | View on Reddit | 18 comments
Hello everyone,
I'm currently building a real-time service where a user sends data over a WebSocket, which gets sent out to downstream services via Kafka. Then, the results are received also via Kafka, and sent back to the client over the same WebSocket. I was wondering how you would scale such a system horizontally.
The biggest issue for me is when I have multiple replicas of the WebSocket server coming in how do I ensure the correct one reads the Kafka message with results that belong to that client (and thus connection)? Tips and reading material would be greatly appreciated.
Dodging12@reddit
Another talk that may be of use: https://www.infoq.com/presentations/linkedin-play-akka-distributed-systems/
yoggolian@reddit
What’s your Kafka partitioning scheme? If you get that sorted, the answer to you question might be more apparent.
Bayul@reddit (OP)
That unfortunately is guarded under lock and key by our ops people who are very paranoid about people touching their configs so I'm trying to find the best way to move forward with minimal interaction with them
yoggolian@reddit
That sounds like Kafka is not the right solution for you right now then - partition design is part of making Kafka-based systems and if you can’t do it then you can’t do it.
armurray@reddit
Partitioning was also my first thought here. If your input and output topics have the same number of partitions, you could use the web socket identifier as the partitioning key and you should end up sending and receiving the input and output to the same socket.
Things would get gnarly if you need to scale up the partition count, though-- any active connections would get jumbled. An interesting idea either way.
kbn_@reddit
Fwiw, what you’re trying to do here is very hard at scale. This is one of those problems where the ideal solution for lol-scale is much much more complex and really unnecessary at small scale, and medium scale is another thing again. My advice is basically two fold:
(source: I’ve built this exact system twice, at two different companies, both at lolscale, and it was very hard, though modern ALBs are so great compared to what we used to have to do)
Bayul@reddit (OP)
We're aiming for \~10k DAU with this, we've made a toy example for a PoC but the Kafka/WS dynamic seems a bit tricky. Do you use Redis pub/sub to forward the messages between the replicas?
kbn_@reddit
Redis pub/sub feels like the best place to turn at that scale. Doable in one AWS region, which vastly simplifies things. In both systems I built, we needed to rely on peer connections (even cross region) to forward a final hop while maintaining low latency, but I think you won’t need those shenanigans.
Also worth deciding what sorts of delivery guarantees you want to have, particularly server-to-client.
Bayul@reddit (OP)
Thank you,I appreciate the insight, we do need to look into the concurrent users and guarantees for sure.
GopherLearnsSt4t@reddit
u/kbn_ - What’s lolscale?
kbn_@reddit
Both of these were “household name” services, so think tens or hundreds of millions of concurrents globally.
madprgmr@reddit
Not the person you asked, but my guess is that it means ridiculously large scale.
Like, large enough to presumably need to route events from a websocket through multiple data centers - including routing the response back. This could be due to something like reducing latency/connectivity issues for users, a need for strong cross-DC consistency (although you want to avoid this whenever possible), or cross-DC failover when using external websocket proxies (ex: cloudflare).
There may be other reasons for doing this, but I am tired and brain is mush.
becuzz04@reddit
We did something sorta similar. We had all our consumers hooked into Redis and when a consumer would read a message it would broadcast it through Redis. Then the right consumer would send the response back to the client. It was reliable enough for us. Your situation may work with that or it may not.
tparadisi@reddit
Yes. Using redis here can ensure it. I was going to suggest the same thing
morswinb@reddit
Got something like this running in prod.
Used a kafka table to store subscription counts data, and assigned fixed partitions per web socket server.
jscheel@reddit
I have built a “realtime” updates system doing this exact thing, complete with kafka and websockets. Use SocketCluster, it will make your life much easier. Once you consume your response message from the kafka topic, you publish it to the socketcluster exchange and it will take it from there.
shakedownstreet420@reddit
You need a persistent registry that maps each userId/deviceId to the hostname/IP of the webserver holding the websocket connection to that client. Then the consumer should do a lookup for every message and send it to the right host, which can send it to the client. Here is a blog post on how netflix runs their websocket backend https://netflixtechblog.com/pushy-to-the-limit-evolving-netflixs-websocket-proxy-for-the-future-b468bc0ff658
juan_furia@reddit
For one i’d separate those into two services. You can scale indefinitely on small sink and source nanoservices servicing that flow.
The only blocker I see is whether the websocket is unique, in that case you need to address a bit of HA for that, but you can’t scale that bit.