Kafka - question about compression

Posted by jageran@reddit | learnprogramming | View on Reddit | 2 comments

Hi,

I hope someone help me with my confusion. I am quite new to kafka so I am trying to find some answers.

I work with an iPaaS from a vendor. The vendor also provided me kafka functionality which they itself get as a managed service from Aiven. I pay for everything under 1 contract to the iPaaS vendor.

The vendor is moving from the aiven hosted kafka to a different provider. And with this migration to the new broker, they are asking me to pre-compress my payloads using lz4 and send it to Kafka instead of sending to kafka with the compression.type=lz4 setting.

Now my question is, what is the advantage for them in me doing pre-compressing the payloads? I feel like they are not being transparent about this.

I would appreciate your inputs.

Thank you.

[-]

BeginningOne8195@reddit

Your suspicion is understandable because normally Kafka producers already handle compression pretty well with compression.type=lz4, and that’s the standard approach most setups use.

Successful_Drawer467@reddit

Pre-compressing before sending definitely seems unusual, and you're right to be suspicious. Normal Kafka compression at producer level is way more efficient because it can compress batches of messages together, not just individual payloads. By making you compress beforehand, they're probably trying to reduce CPU load in their Kafka clusters since they won't need to handle compression anymore - basically shifting that cost to your side instead of theirs