A Kubernetes Best Practice Question

Posted by failsafe-author@reddit | ExperiencedDevs | View on Reddit | 12 comments

I am pretty inexperienced with Kubernetes. We have an infra team that handles most of that end of things, and during my first year at the company I was working on non-product software: tooling and process stuff. This is stuff that didn’t get deployed the way our main apps do.

The past few months, I’ve been working in various code bases, getting familiar with our many services, including a monolith. Today, I learned about a pattern I didn’t realize was being used for deployments, and it sounds off. But, I’m a Kubernetes noob, so I’m reticent to lean too heavily on my own take. The individual who shared this to me said most people working in this code don’t understand the process, and he wants me to knowledge transfer, from him to me, and then I take it out to others. The problem is, I don’t think it’s a good idea.

So here’s what we have- in the majority of our service repos, we have folders designated for processes that can be deployed. There will be one for the main service, and then one for any other process that need to run alongside it in a support role. These secondary processes can be stuff like migrations, queue handlers, and various other long running processes. Then, there is another folder structure that references these first folders and groups them into services. A service will reference one-to-many of the processes. So, for example, you may have several queue handlers grouped into a single service, and this gets deployed to a single pod- which is managed by a coordinator that runs on each pod. Thus, we have some pods with a single process, and then several others that have multiple process, and all of it is run by a coordinator in each pod.

My understanding of Kubernetes is that this is an anti-pattern. You typically want one process per pod, and you want to manage these processes via Kubernetes. This is so you can scale each process as needed, they don’t affect each other if there are issues, and logging/health isn’t masked by this coordinator that’s running in each pod.

This is not just something that’s been done- the developer shared with me a document that prescribes this process, and that this is the way all services should be deployed Most developers, it seems, don’t even know this is going on. The reason I know it is because this developer was fixing other team’s stuff who hadn’t implemented the pattern correctly, and he brought it to me for knowledge sharing (as I mentioned before). So, even if this isn’t a bad practice, it is still adding a layer of complexity on top of our deployments that developers need to learn.

Ultimately, I am in a position where if I decide this pattern is bad, I can probably squash it. I can’t eliminate it from existing projects, but I can stop it from being introduced into new ones. But I don’t want to take a stand against an established practice lightly. Hence, I’d like to hear from those with more Kubernetes experience than myself. My assumption is that it’s better to just write the processes and then deploy each one to its own pod, using sidecars where they make sense.

It’s worth noting that this pattern was established back when the company had a dozen or so developers, and now it has 10 times that (and is growing). So what may have felt natural then doesn’t necessarily make sense now.

Am I overreacting? Am I wrong? Is this an OK pattern, or should I be pushing back?

[-]

CooperNettees@reddit

its very difficult to justify this pattern. one process per pod means you can set resource limits on that particular process specifically. having multiple in one container means you cannot prevent those two processes from contesting the overall pods resources.

the only situations where I might do something like this is:

the main container cant function without the sidecar.

and

the main container itself is a shim, script or some "hack" solution to a problem that isnt meant to endure long term.

this is not a good pattern and its reasonable to push back.

binaryfireball@reddit

it may not be horrible but its not good, feels like they wanted something "simpler" without thinking it through all the way

failsafe-author@reddit (OP)

Wanting things to be “simple” is kind of a mantra with this developer.

its not bad but it cant be dogma and it actually has to be simple. there is inherent complexity that comes with kubernetes. the kicker is that a lot of companies dont actually need kubernetes and the infra can be simplified by migrating off of it, but that's a question for you to ask yourself.

if you're stuck with kube then talk with him about how this actually is creating friction, be prepared to back up your arguments though

Yeah, that last statement is what I’m trying to do.

(And we definitely need Kubernetes. This is a company that is exiting startup mode and going into “wildly successful”‘mode, which being ether kinds of challenge you want to have!)

DeterminedQuokka@reddit

So I can’t be 100% sure what your code looks like but this might be normal depending what it looks like in production.

We have a similar system where we have basically one core pod that runs in about 6 different modes. And when it’s released it runs 2-3 of most of the modes, one singleton and a one off task (migrations). Every pod runs a single thing but they all use the same base because it’s all just commands on that code which is shared. That’s pretty normal from my experience.

Basically we have a terraform file that is named after the pod then lists out all the deployment classes and their configurations.

If you are saying they deploy a single pod and run 4 processes on it that would seem strange to me.

It’s deploying many pods running multiple processes on a subset of them (so the main service is the only one running in its pod, though it is still run via the orchestrator binary).

I mean one of your main service freaks me out because when it resets at midnight there are none for some period of time.

There isn’t really a reason to run multiple processes per pod, you can make as many pods as you want. But I also don’t think it’s the end of the world.

dogo_fren@reddit

Yes, it does sound weird. Is the Pod a bare Pod or something? That would be even weirder.

lurkin_arounnd@reddit

You typically want one process per pod, and you want to manage these processes via Kubernetes

really you want one process per container, but you got the right general idea

if they are time limited processes and not long running services (such as db migrations), it’d be pretty typically to run them in init containers in the relevant pod. or perhaps in your entrypoint script.

if you’re talking about multiple long running services, you probably do wanna separate them into different containers in the same pod or separate pods

OGicecoled@reddit

One container per pod is a pattern, but it’s not the only one. There is also the sidecar pattern which sounds like your org is using.

originalchronoguy@reddit

Your hunch is mostly right. There would have to be some weird justification for that approach.

Migration services, backing services should run and terminate. Update the DB, copy /rsync the file and be killed. Less running services with exposed attack vectors.

He just doesn't know how to architect things to turn on or off at deployment. You can do that with environment variables and configuration conditionals in helm chart to deploy or NOT deploy a service.

And your health check concern is spot on. Every developer needs to know what processes run and not hidden in some black box. I don't need to know my API is running an internal crontab. I need to know that from a high level I can quickly see in a repo level from a known place (aka the charts or config).