No, Really, Bash Is Not Enough: Why Large-Scale CI Needs an Orchestrator

[-]

Academic_East8298@reddit

I prefer to use makefiles, that are orchestrated using the current CI/CD system. It seems to solve all the issues mentioned in the article, while still remaining relatively simple.

[-]

GreatWoodsBalls@reddit

Do you mind elaborating a bit and maybe give some pointers for me to google further?

[-]

fragbot2@reddit

The system I work on has a process like the following:

make start    # login to artifactory, start the build container and compile some protobuf files
make tgt      # build the primary object files and link the main executables
make ut       # run the unit tests
.
.
.
make stop    # remove the docker container

Local development and CI build are fundamentally the same.

[-]

Academic_East8298@reddit

There is nothing to elaborate really. The makefile contains all the CI steps. The most basic example would be - build, test and deploy. These can be called by the dev locally and by the CI engine.

[-]

gmes78@reddit

How does that address any of the problems raised in the article?

[-]

Academic_East8298@reddit

Let's see.

Artifact handling - makefile has all the logic to generate test, benchmark and build results. CI logic is used for caching and reuse.

Combinatorial sprawl - makefile contains the logic for all the indivdiual units. CI handles the dependencies between individual steps and parallelism.

Isolation - run the individual steps on different CI agents?

Language safety - eh, good luck trying to debug CI issues using the CI orchestrator. Bash is much easier to work with, when you can test all the individual CI step locally.

Port problem - CI engines handle the logic of where and when to run something, makefile contains the logic of what to run.

I have lived through a company changing its CI tools enough times, that I today I try to decouple my CI logic from a CI tool as much as possible.

[-]

lizardhistorian@reddit

Combinatorial sprawl - makefile contains the logic for all the indivdiual units. CI handles the dependencies between individual steps and parallelism.

Even that works exceedingly poorly in contemporary CI/CD tools.
The overhead they've created is stupid.

We just cross-compile in the build.

[-]

lizardhistorian@reddit

We add a phony 'tools' target which installs the required tools for the build as well.
So make creates the docker image for the CI build step.

[-]

fragbot2@reddit

I fully agree. You get the following naturally:

a DAG that details your build system.
defaults to an automatic stop on error.
they're simple to compose.
if you use gmake, they have "just enough" internal programmability for automated rule generation.
parallelism is essentially free.
its usage is easy to document and remember [ed. note: this one's under-rated as there's a massive number of README.md files that could be simplified with a phony one-time-setup target.]

Downsides:

most people changing Makefiles barely understand make and hack at something they copied from another repo until it sorta works.
debugging is unpleasant especially if there's a parallelism issue.

[-]

lizardhistorian@reddit

Bazel is ass.
Preposterously complex scripts to configure it.

Use CMake to manage your build. Have it use ninja if you'd like.
make is always the backbone.

[-]

effyouspez@reddit

Oh I share your hatred of bazel, but it's what I inherited,

[-]

CJKay93@reddit

Makefiles are worthless as soon as you need structured data, like "which tests does the current environment configuration support?"

[-]

lizardhistorian@reddit

If you need structured data for your builds you fuct up a long time ago.

[-]

Umm what about: make test --os [input os] --browser [input browser] --output [some output file for ci reports]. The makefile spins a docker container with the specified parameters and runs the tests. Can be called by a CI agent and by the dev locally.

[-]

ThingPossible1971@reddit

Gold standard

[-]

Sorry-Transition-908@reddit

I think the idea is not literally make files. the idea is to keep proprietary services as dumb as possible and write portable code

[-]

mpyne@reddit

Makefiles may not be the best, but it's not due to structured data. Turning the cartesian product of a test matrix into make targets isn't Make 101 but it's not rocket science either.

[-]

o5mfiHTNsH748KVq@reddit

God damn, I used to run a DevOps/platform team and it was a real fight to get them to use a real language that they can actually write tests for. Ever since, I’ve never hired a DevOps engineer that can’t code.

[-]

mx2301@reddit

What were the languages in this case? On my last project, we used nu and nushell to deal with the bash issue.

[-]

slaymaker1907@reddit

Nushell has the same problems. Untestable, logging is a pain, no debugger, etc.

[-]

mx2301@reddit

It does have a test framework and a dedicated setting for logging in a debug environment. So i would not sah it is that Bad.

[-]

Worth_Trust_3825@reddit

From first glance it looks like powershell but with friendly error messages. What are the other selling points? How's the IDE support?

[-]

mx2301@reddit

In our case specifically.

We needed to work on Linux and windows and wanted to write templates and scripts that would work on both platforms. Some team members knew powershell only and some only bash/zsh and nushell was similar enough to bash and powershell that team members could adapt. Also it got a plugin system, that is quite neat(but we only used it sparingly).

The IDE support is pretty solid. It ships with an language server, a VSCode Extension exists and I think there is also an Extension available on the Jetbrains side of things.

[-]

Worth_Trust_3825@reddit

Looks promising

[-]

Lalli-Oni@reddit

Oh lawdy, just reading that made me excited. Best language people are sleeping on.

Although, not sure why you'd need it twice.

[-]

mx2301@reddit

Once to write a nushell script on Linux and windows and execute them via nu script.nu. And the second option is to eventually use it as a normal shell, when going into specific containers.

[-]

slaymaker1907@reddit

I’m fighting currently fighting that battle. Yes Python can be slightly more verbose, but it’s a lot easier to read, you can easily add logging, and as you mentioned, it’s actually testable.

[-]

lizardhistorian@reddit

And now you need to V&V your build tool.

Fails Concept FMEA.

[-]

slaymaker1907@reddit

You shouldn’t immediately reach for it, but I think you should really start considering a real scripting language when your “simple bash script” grows beyond a dozen lines or so.

[-]

kerakk19@reddit

Hope by real language you don't mean Python, because this shit will give you more trouble than bash

[-]

slaymaker1907@reddit

The only problem I see with Python for this purpose is that u/kerakk19 doesn’t like it.

[-]

lizardhistorian@reddit

Unless you are using 3.13t and just wrote a new amazing build system then you cannot possibly be correct.

[-]

o5mfiHTNsH748KVq@reddit

I have a preference for python and go, but more often than not teams are using typescript for AWS CDK where I'm at. I personally like typescript with pulumi.

[-]

yknx4@reddit

If you hired a DevOps engineer that can’t code. You hired a Ops Engineer.

[-]

o5mfiHTNsH748KVq@reddit

I used to try to build balanced teams with dev and ops backgrounds, but I've come to learn that a developer can learn anything ops can do. Especially with AI. They just don't want to, but it's called infrastructure as code, so i'm not sure what people expect.

[-]

booch@reddit

They just don't want to

There's nothing wrong with that. I don't enjoy doing ops-related things. I do enjoy software development. There's overlap, but it's entirely possible to enjoy one but not the other.

[-]

o5mfiHTNsH748KVq@reddit

That’s always been me, but I don’t think you have to love every part of your job. It’s kind of neat to step out of this conversation and consider that I was always someone that hated infrastructure work, but ended up running DevOps orgs. It was my job to make those processes code driven.

I try to give people the tasks they have an affinity for, but on the whole, I firmly believe every dev team needs to fully own their architecture from top to bottom. I happen to mostly work with apps in cloud providers, so it’s really to everyone’s benefit that our engineers deeply understand the services they’re using when building software.

I’d have a less solid take on this for a team that’s managing actual on-prem hardware. Probably don’t touch that.

And lastly, if this is Grady, I think you’ve earned working on whatever you want ;)

[-]

Hopeful-Ad-607@reddit

To be fair, most "DevOps" engineers can't do actual ops outside cloud providers, which are basically Lego playgrounds anyway.

[-]

IndividualPants@reddit

you mean the "dev" in devops actually means something??

[-]

valarauca14@reddit

Having worked in devops/dev-infra, the number of times somebody would come along and just comment out a pre-submit check that ran our bash unit tests, style checks, and commit checks was amusing.

Ended up moving a lot of that to its own repo, which got dragged in as a sub-module, so if somebody wanted to change it it required a separate change request.

[-]

o5mfiHTNsH748KVq@reddit

fast track to a pip on my teams

[-]

valarauca14@reddit

Off shore contractors, replace them all you want they rarely improve

[-]

ProtoJazz@reddit

I once took over a project previously worked on by an offshore team.

It had a ton of tests. Like a ton. They took 30-60 minutes to run. And they ran a lot. Pre push hook, then again on ci before you could merge, and then again after you merged but before it deployed.

And the worst part was most of the tests were just

"Mock x to return y. Assert x returns y"

[-]

calmingchaos@reddit

Sounds like a certain auto company I used to work for.

[-]

mr_dfuse2@reddit

I used to be in the same situation. Best devops engineers come from dev, not ops. (generalization) We used Groovy as our glue language back in the day.

[-]

Ill_Fun5415@reddit

The dependency graph problem is real. Once you have more than a handful of jobs with cross-dependencies, bash becomes a state machine simulator. The article's point about error recovery being the real differentiator resonates - retry logic in shell scripts quickly becomes unreadable.

[-]

MooseBoys@reddit

#!/bin/bash shebang

Immediately stopped reading when I saw this. Everyone knows #!/usr/bin/env bash is the proper way to do this.

[-]

lizardhistorian@reddit

No, I do not want your jacked up environment sucked into my deterministic builds.

[-]

MooseBoys@reddit

If you consider environment variables to be untrusted then you're hosed anyway.

[-]

Worth_Trust_3825@reddit

If they did former wouldn't appear.

[-]

Dramatic_Turnover936@reddit

the part nobody says out loud: CI scripts are production code, they just live in a directory nobody reviews. bash gets messy because it starts as a 10-line shortcut and ends up as 800 lines of state management that two people understand. an orchestrator doesn't fix that habit, it just makes the mess more structured.

[-]

programming-ModTeam@reddit

No content written mostly by an LLM. If you don't want to write it, we don't want to read it.

[-]

Kanegou@reddit

Couldn't finish the article. Author is way to arrogant to take him seriously.

[-]

mattinternet@reddit

Incredibly long winded too, get to the point

[-]

Absolute_Enema@reddit

How so?

[-]

Kanegou@reddit

The first chapter is nothing but talking down at others. His adressing of previous responses is nothing but appealing to authority.

[-]

case-o-nuts@reddit

Also, too AI.

[-]

d33pnull@reddit

skill issue

[-]

urbrainonnuggs@reddit

Fun fact. Bash is older than the majority of programmers today

[-]

Hopeful-Ad-607@reddit

I use argo workflows and it's pretty cool. I build my shit in my build container, test my shit in my test container, build my image in my image build container, and deploy on my deploy container.

Everything is isolated by default. Any dependency has to be explicit.

I cache all dependencies so I can resume from any place.

There's nothing fundamental about argo workflows that's super complex or anything. It's an "orchestrator" that's pretty simple in theory and in practice.

Recommend if you're already doing most stuff in a cluster anyway.

[-]

melvin_mouse@reddit

There's some good information in there but the argument seems very "straw man"ey

[-]

valarauca14@reddit

This article feels way too disorganized to be AI written.

The author has a laundry list of complaints (build caching, container-image caching, resource allocation/sharing/semaphores for test runners). Which yeah, obviously bash alone can't handle.