Conversely I have experienced essentially the opposite - programming teams with little supervision who strike gold independently doing blue skies projects with extremely little business interest and oversight. Once those projects materialize as products a flock of corporates descend from the heavens to take ownership of the potential profits.

The idea of a blue skies project is itself a bit misleading. Generally such a project is to meet a need of some sort that is currently being done very inefficiently. When a project materialises, what that actually means is it gets into the hands of the people using it, and those people will often have particular needs that you may not have accounted for.

Once those people start using this system, they will experience certain amounts of friction, which will translate into complaints. These complaints may not be coherent enough to form into working code, but that's because they are coming from people that do not understand software, or the thought process that goes into designing it. This is why part of the job of a good developer is to actually understand the needs of a stakeholder. Often times when they say they want something, what they really mean is they want something that fulfils the needs, and it's really up to the technical person to help them realise what that specific task, and those specific needs those actually are.

If you instead take everything they say literally, and try to develop exactly what they are asking for, then you will likely end up misunderstanding their needs at some point, and delivering contradictory features. Then you're going to piss them off by claiming you did what they asked, because they thought they were asking for something different, and the technical team didn't spend enough time to actually discover that. Essentially, if this is your experience, then the problem is the fact that you have two groups of people speaking completely different languages, with nobody to translate between the two.

Such communication should be a lot more like a mix of negotiation and interrogation, with a clear process for making for decisions, and clear consequences for each one, even if those consequences are "Ok, your story gets put above the CTO's then."

Essentially, if you treat these people as some hostile other element that's here to ruin your perfect codebase, then that's what they will be. However, if you treat them as generally rational people that are all trying to do something using the best tools they can find for the job, but who have not spent the years and decades that you have to learn all the ideas, terms, and workflows necessary to actually crate such tools, then you will find that they're often a lot more competent then you give them credit for. In fact, if you treat them that way, they will often treat you better in turn, in part because you will actually be delivering the things they need rather than your first interpretations of something they said.

Essentially, look at it this way. If these people are investing into having a group of programmers working for them, then they at least understand that there is value in having such a team. Rather than focusing on how they don't speak real smart, treat them like they are smart in their own thing, and simply don't know your field as well as you do. It's ok to ask clarifying questions, and offer options and alternate solutions in order to better understand the requirements.

Obviously sometimes that's not possible. In some cases the leadership team really is too stubborn and irrational. However, in that case there's no reason to waste time in such an environment. In my experience such a personality is very much a minority, and once you've been around such people they are quite easy to spot from a mile away. If you are in an environment with such people... Why?

The tools that programmers make have a real market value determined by customers. Finding those customers and marketing them categorically has a business value but it is hugely overvalued when their sales tactics involve writing cheques they know they can't cash and then dumping the debt onto engineering teams to work miracles for comparatively little compensation.

While the tools that programmers write obviously have value to customers, I think you vastly underestimate how difficult it is to actually convince people to give you large amounts of money in order to become customers. A lot of time, in order to even get to the point where they can promise a feature that they know doesn't exist yet a person might need to spend months writing emails, making calls, taking people out to lunch, and then a series of bigger and bigger meetings, until maybe they might get a chance to promise a feature that will be a huge pain on you, in return for a very large sum of money. I certainly would not call it a lesser contribution than the actual work of writing the software.

In terms of compensation, the big difference here is that these people generally know how to negotiate, and as part of their contract they demand a percentage of the contracts they sign. Fortunately when you're a sufficiently senior developer, you can also negotiate for more than just your salary. It's just for some reason a lot of developers don't even try, even when they have the clout for it.

Equally, I've seen teams and even entire companies totally paralyzed by an unworkable codebase that can't be evolved into new products.

Maintaining a large codebase, and planning a long term strategy for how to grow it is a very complex task. This requires skill on the part of both the technical team, and the business team. A good technical team isn't going to let the code get into such a state in the first place, and will know how to fix it if they get into such a mess, and a good business team will understand that developers often underestimate complexity and overestimate capacity. A company with neither will be paralysed, because a bad business team will lead them blindly wherever their gut leads them, and a bad technical team will take such leadership as gospel.

If you think just "taking some guidelines and transforming them into working code" is inherently a low value activity then it's a low value activity that is bleeding these companies, often with hundreds of employees, tens of thousands of dollars daily.

On the skill tree that is the Software Developing profession, transforming guidelines into code is literally the tier 1 skill that you're expected to have in order to actually be considered a professional.

The thing that's bleeding these companies try usually isn't the "transforming guidelines into code," it's the "transforming stakeholder blabber into guidelines" skill set.

Of course you should always be humble and recognise that your work is only made profitable by the systems that exist around it which make it valuable, but in my experience the balance is skewed far more heavily in the opposite direction to what you imply here.

Certainly the balance of compensation is not favourable to the developer, but I was not claiming otherwise. This is why I have several sections on how you can translate a more careful, negotiation based approach into contacts and political power. If you want power, the way to do it is not to go, "Nah, developers actually ARE the most important part of the organisation after all, the rest of you suck." Even if you don't say it out loud, to people that have to politick day in and day out, you might as well. Needless to say, they aren't likely to react well, as you have described.

Instead you have to play their game, at least a little bit. Actually listen to them, try to really understand what they mean by engaging with them, rather than just sitting and hearing the words they say while creating your own image of what they might want in your head. Treat them like they are serious professionals, and like you are a serious professional here to do business with them. That is, after all, quite literally what you, and they are.