No, Really, Bash Is Not Enough: Why Large-Scale CI Needs an Orchestrator
Posted by someone-very-cool@reddit | programming | View on Reddit | 64 comments
Posted by someone-very-cool@reddit | programming | View on Reddit | 64 comments
Academic_East8298@reddit
I prefer to use makefiles, that are orchestrated using the current CI/CD system. It seems to solve all the issues mentioned in the article, while still remaining relatively simple.
GreatWoodsBalls@reddit
Do you mind elaborating a bit and maybe give some pointers for me to google further?
fragbot2@reddit
The system I work on has a process like the following:
Local development and CI build are fundamentally the same.
Academic_East8298@reddit
There is nothing to elaborate really. The makefile contains all the CI steps. The most basic example would be - build, test and deploy. These can be called by the dev locally and by the CI engine.
gmes78@reddit
How does that address any of the problems raised in the article?
Academic_East8298@reddit
Let's see.
Artifact handling - makefile has all the logic to generate test, benchmark and build results. CI logic is used for caching and reuse.
Combinatorial sprawl - makefile contains the logic for all the indivdiual units. CI handles the dependencies between individual steps and parallelism.
Isolation - run the individual steps on different CI agents?
Language safety - eh, good luck trying to debug CI issues using the CI orchestrator. Bash is much easier to work with, when you can test all the individual CI step locally.
Port problem - CI engines handle the logic of where and when to run something, makefile contains the logic of what to run.
I have lived through a company changing its CI tools enough times, that I today I try to decouple my CI logic from a CI tool as much as possible.
lizardhistorian@reddit
Even that works exceedingly poorly in contemporary CI/CD tools.
The overhead they've created is stupid.
We just cross-compile in the build.
lizardhistorian@reddit
We add a phony 'tools' target which installs the required tools for the build as well.
So make creates the docker image for the CI build step.
fragbot2@reddit
I fully agree. You get the following naturally:
Downsides:
effyouspez@reddit
Switched from make files to Just
lizardhistorian@reddit
Bazel is ass.
Preposterously complex scripts to configure it.
Use CMake to manage your build. Have it use ninja if you'd like.
make is always the backbone.
effyouspez@reddit
Oh I share your hatred of bazel, but it's what I inherited,
CJKay93@reddit
Makefiles are worthless as soon as you need structured data, like "which tests does the current environment configuration support?"
lizardhistorian@reddit
If you need structured data for your builds you fuct up a long time ago.
CJKay93@reddit
Academic_East8298@reddit
Umm what about: make test --os [input os] --browser [input browser] --output [some output file for ci reports]. The makefile spins a docker container with the specified parameters and runs the tests. Can be called by a CI agent and by the dev locally.
ThingPossible1971@reddit
Gold standard
Sorry-Transition-908@reddit
I think the idea is not literally make files. the idea is to keep proprietary services as dumb as possible and write portable code
mpyne@reddit
Makefiles may not be the best, but it's not due to structured data. Turning the cartesian product of a test matrix into make targets isn't Make 101 but it's not rocket science either.
o5mfiHTNsH748KVq@reddit
God damn, I used to run a DevOps/platform team and it was a real fight to get them to use a real language that they can actually write tests for. Ever since, I’ve never hired a DevOps engineer that can’t code.
mx2301@reddit
What were the languages in this case? On my last project, we used nu and nushell to deal with the bash issue.
slaymaker1907@reddit
Nushell has the same problems. Untestable, logging is a pain, no debugger, etc.
mx2301@reddit
It does have a test framework and a dedicated setting for logging in a debug environment. So i would not sah it is that Bad.
Worth_Trust_3825@reddit
From first glance it looks like powershell but with friendly error messages. What are the other selling points? How's the IDE support?
mx2301@reddit
In our case specifically.
We needed to work on Linux and windows and wanted to write templates and scripts that would work on both platforms. Some team members knew powershell only and some only bash/zsh and nushell was similar enough to bash and powershell that team members could adapt. Also it got a plugin system, that is quite neat(but we only used it sparingly).
The IDE support is pretty solid. It ships with an language server, a VSCode Extension exists and I think there is also an Extension available on the Jetbrains side of things.
Worth_Trust_3825@reddit
Looks promising
Lalli-Oni@reddit
Oh lawdy, just reading that made me excited. Best language people are sleeping on.
Although, not sure why you'd need it twice.
mx2301@reddit
Once to write a nushell script on Linux and windows and execute them via nu script.nu. And the second option is to eventually use it as a normal shell, when going into specific containers.
slaymaker1907@reddit
I’m fighting currently fighting that battle. Yes Python can be slightly more verbose, but it’s a lot easier to read, you can easily add logging, and as you mentioned, it’s actually testable.
lizardhistorian@reddit
And now you need to V&V your build tool.
Fails Concept FMEA.
slaymaker1907@reddit
You shouldn’t immediately reach for it, but I think you should really start considering a real scripting language when your “simple bash script” grows beyond a dozen lines or so.
kerakk19@reddit
Hope by real language you don't mean Python, because this shit will give you more trouble than bash
slaymaker1907@reddit
The only problem I see with Python for this purpose is that u/kerakk19 doesn’t like it.
lizardhistorian@reddit
Unless you are using 3.13t and just wrote a new amazing build system then you cannot possibly be correct.
o5mfiHTNsH748KVq@reddit
I have a preference for python and go, but more often than not teams are using typescript for AWS CDK where I'm at. I personally like typescript with pulumi.
yknx4@reddit
If you hired a DevOps engineer that can’t code. You hired a Ops Engineer.
o5mfiHTNsH748KVq@reddit
I used to try to build balanced teams with dev and ops backgrounds, but I've come to learn that a developer can learn anything ops can do. Especially with AI. They just don't want to, but it's called infrastructure as code, so i'm not sure what people expect.
booch@reddit
There's nothing wrong with that. I don't enjoy doing ops-related things. I do enjoy software development. There's overlap, but it's entirely possible to enjoy one but not the other.
o5mfiHTNsH748KVq@reddit
That’s always been me, but I don’t think you have to love every part of your job. It’s kind of neat to step out of this conversation and consider that I was always someone that hated infrastructure work, but ended up running DevOps orgs. It was my job to make those processes code driven.
I try to give people the tasks they have an affinity for, but on the whole, I firmly believe every dev team needs to fully own their architecture from top to bottom. I happen to mostly work with apps in cloud providers, so it’s really to everyone’s benefit that our engineers deeply understand the services they’re using when building software.
I’d have a less solid take on this for a team that’s managing actual on-prem hardware. Probably don’t touch that.
And lastly, if this is Grady, I think you’ve earned working on whatever you want ;)
Hopeful-Ad-607@reddit
To be fair, most "DevOps" engineers can't do actual ops outside cloud providers, which are basically Lego playgrounds anyway.
IndividualPants@reddit
you mean the "dev" in devops actually means something??
valarauca14@reddit
Having worked in devops/dev-infra, the number of times somebody would come along and just comment out a pre-submit check that ran our bash unit tests, style checks, and commit checks was amusing.
Ended up moving a lot of that to its own repo, which got dragged in as a sub-module, so if somebody wanted to change it it required a separate change request.
o5mfiHTNsH748KVq@reddit
fast track to a pip on my teams
valarauca14@reddit
Off shore contractors, replace them all you want they rarely improve
ProtoJazz@reddit
I once took over a project previously worked on by an offshore team.
It had a ton of tests. Like a ton. They took 30-60 minutes to run. And they ran a lot. Pre push hook, then again on ci before you could merge, and then again after you merged but before it deployed.
And the worst part was most of the tests were just
"Mock x to return y. Assert x returns y"
calmingchaos@reddit
Sounds like a certain auto company I used to work for.
mr_dfuse2@reddit
I used to be in the same situation. Best devops engineers come from dev, not ops. (generalization) We used Groovy as our glue language back in the day.
Ill_Fun5415@reddit
The dependency graph problem is real. Once you have more than a handful of jobs with cross-dependencies, bash becomes a state machine simulator. The article's point about error recovery being the real differentiator resonates - retry logic in shell scripts quickly becomes unreadable.
MooseBoys@reddit
Immediately stopped reading when I saw this. Everyone knows
#!/usr/bin/env bashis the proper way to do this.lizardhistorian@reddit
No, I do not want your jacked up environment sucked into my deterministic builds.
MooseBoys@reddit
If you consider environment variables to be untrusted then you're hosed anyway.
Worth_Trust_3825@reddit
If they did former wouldn't appear.
Dramatic_Turnover936@reddit
the part nobody says out loud: CI scripts are production code, they just live in a directory nobody reviews. bash gets messy because it starts as a 10-line shortcut and ends up as 800 lines of state management that two people understand. an orchestrator doesn't fix that habit, it just makes the mess more structured.
programming-ModTeam@reddit
No content written mostly by an LLM. If you don't want to write it, we don't want to read it.
Kanegou@reddit
Couldn't finish the article. Author is way to arrogant to take him seriously.
mattinternet@reddit
Incredibly long winded too, get to the point
Absolute_Enema@reddit
How so?
Kanegou@reddit
The first chapter is nothing but talking down at others. His adressing of previous responses is nothing but appealing to authority.
case-o-nuts@reddit
Also, too AI.
d33pnull@reddit
skill issue
urbrainonnuggs@reddit
Fun fact. Bash is older than the majority of programmers today
Hopeful-Ad-607@reddit
I use argo workflows and it's pretty cool. I build my shit in my build container, test my shit in my test container, build my image in my image build container, and deploy on my deploy container.
Everything is isolated by default. Any dependency has to be explicit.
I cache all dependencies so I can resume from any place.
There's nothing fundamental about argo workflows that's super complex or anything. It's an "orchestrator" that's pretty simple in theory and in practice.
Recommend if you're already doing most stuff in a cluster anyway.
melvin_mouse@reddit
There's some good information in there but the argument seems very "straw man"ey
valarauca14@reddit
This article feels way too disorganized to be AI written.
The author has a laundry list of complaints (build caching, container-image caching, resource allocation/sharing/semaphores for test runners). Which yeah, obviously bash alone can't handle.