Feels like the context window at my company has gotten full, because we've gone full RDJ

Posted by ninetofivedev@reddit | ExperiencedDevs | View on Reddit | 48 comments

I'll start by saying there was always just obvious tropes that everyone pretty much universally accepted.

LoC is a horrible metric and a really dumb thing to care about when evaluating performance of a developer outside of the ends of the spectrum.
Comparing story points across teams makes zero sense. Also using story points as a metric for productivity also has huge gaping flaws.
Counting commits or PRs is also easily gamed and useless.
Hero culture is bad.

Suddenly, AI has hit the scene and it's the opened the flood gates. Not only have we decided to start tracking all these ridiculous metrics against individuals, but also throw in token burn.

What in the actual flying ... What has happened?

What are your companies doing? Is it just me?

[-]

jungle@reddit

I saw this trend towards measuring every possible thing and use it for performance management before AI came into play. Tools that analyse things like LOC changed after a code review comment on git, used as a proxy for "impact"... Trying to reduce everything to numbers.

From a manager's point of view, it's very tempting to be able to point to a metric and say "you're below expectations, compared to your teammates or to others at your level in other teams". But, as you say, metrics can be gamed, and improving them doesn't mean doing better work.

And now we have new metrics! Yay! :(

[-]

Direct-Pipe9555@reddit

token burn is such a weird metric to focus on

[-]

bwmat@reddit

It's especially stupid because it's a cost, and there's more & less efficient ways to 'spend' them, so you could be 'punished' for being efficient

[-]

ninetofivedev@reddit (OP)

The message is don't be efficient. Don't optimize cost. Because the metric for measuring efficiency in this case, at scale, is going to be really fucking hard to define.

If you have a team of 100 engineers, and you're trying to determine their AI usage efficiency, anything you come up with is probably going to be bullshit.

[-]

Tacos314@reddit

The metric being tracked is how much got done and was AI used.

[-]

Tacos314@reddit

It's a metric of use, why would you not focus on it? The goal is to increase productivity by using AI. If you're not using the AI tooling and not increasing productivity, that's an issue.

[-]

jungle@reddit

Let's measure average CPU temp. I'm sure that can't be, erm, gamed. ;)

[-]

Effective_Hope_3071@reddit

It's hilarious because 80 percent of my token burn is just asking googlable questions I'm too lazy to tab out of terminal for

[-]

Kjufka@reddit

From a manager's point of view it's very tempting to be able to point to a metric and say "you're below expectations compared to your teammates or to others at your level in other teams".

fuck managers who do that

100% of the time when something like happened in any of my jobs, whenever it was to me or to any other team member - it was never about performance - it was always about manager trying to prove his usefulness to the higher ups. And they always fired someone who was NOT, in fact, underperforming.

[-]

jungle@reddit

There's also companies where someone high up (VP of eng, CTO, etc) buys a product that measures all those things and expects all line managers to use it. What's a manager supposed to do? The higher up is also looking at the same numbers and expects to see things happen.

One of the reasons I quit. Not the main one, but one.

[-]

ninetofivedev@reddit (OP)

From a manager's point of view it's very tempting to be able to point to a metric and say "you're below expectations compared to your teammates or to others at your level in other teams". But, as you say, metrics can be gamed and improving them doesn't mean doing better work.

Why is our industry allergic to peer feedback? Whoever thought it was a good idea to create "measurable" "objective" measures was an idiot. Why can't we just have a team of 8 people who say "Yeah, we like working with Anthony. He's always helpful. He has good intuition. He takes time out of his day to help me understand the problems I'm stuck on." or "We don't really like working with Jonathan. He's an asshole and honestly, nobody really knows what he does all day".

But what if I'm not likable?

Be fucking likable! Really all you have to do is be helpful. That's what people like. It goes for ICs and Managers. All a good manager has to do is provide direction and support your career growth. It's not that hard.

[-]

jungle@reddit

peer feedback

Many companies do that. But it's a very uneven and easily unfair system. Some people don't like to give negative feedback or any kind of feedback, some people can use it to settle grudges, and the personality of the person on which the feedback is given can play a huge part in the quality of the feedback. It also puts a burden on the team to spend time writing something about each other that they know has consequences and is on record. It doesn't contribute to a good team culture in my opinion.

I think a much better system is the manager knowing the people they work with, having good conversations in their 1:1s about the individual's work and about the team's work and dynamics, watching the team work together, looking at the PRs, communications on public channels, the reasons for delays, how they collaborate internally and across teams, noting when they catch issues before they happen, when they share knowledge, their level of ownership (including how they own their mistakes), etc.

But it's so much easier to just have a piece of software give you hard numbers that you can point to while shaking your head (syh?)...

[-]

Tacos314@reddit

The metrics are not bad. Token burn is to ensure you’re using the AI; with the AI, you’re supposed to be able to produce an increase in LOC (Lines of Code), and your velocity should increase. These are all metrics and signals that can be used to judge the AI rollout and how productive it’s being.

Saying that I would be super annoyed but why are you not burning tokens? IS there an issue your having?

[-]

Wonderful-Habit-139@reddit

We're really seeing software architects linking an increase in LOC to velocity.

More lines of code is not better. Code is a liability.

[-]

Tacos314@reddit

LOC and velocity go together; if LOC increases and velocity goes down, something has gone wrong. LOC is how much work someone did and velocity is what got done; ideally, lower LOC and higher velocity would be ideal. If you believe LOC does not matter, sorry, you're wrong.

The true metric, though, is products released.

[-]

vocal-avocado@reddit

It’s absolutely insane that you are an architect with 20YOE and you think LOC is a good metric. And you probably earn 4x what I do. The world is so unfair.

[-]

Tacos314@reddit

Because the metrics are not about you or me; they’re about how the new rollout and policies are being implemented. These metrics are not being used to say who is productive, but rather who is using AI and what type of productivity gains are being made (or not being made).

I also knew this would be an unpopular take, because too many people look only at themselves and not how the organization views things. Managers and CEOs are not stupid (at least enough of them are not) and know very well the issues with pure LOC metrics.

[-]

experienceddevsb@reddit

This flair is only allowed on wednesday, saturday (UTC). Please repost on an allowed day. Intentionally trying to circumvent this rule will result in a suspension. See: https://www.reddit.com/r/ExperiencedDevs/comments/1rfhdrg/moderation_changes/

[-]

ninetofivedev@reddit (OP)

This was posted on Wednesday you stupid bot.

[-]

eufemiapiccio77@reddit

RDJ?

[-]

another_dudeman@reddit

Robert Downey Jr

[-]

FoolHooligan@reddit

...and what exactly does that mean in this context?

[-]

NaBrO-Barium@reddit

Lots of cocaine and then rehab

[-]

Kjufka@reddit

Oh, I though it was "jarvis, write this line of code for me"

[-]

doberdevil@reddit

But why rehab? Blow helps with the LOC metric.

[-]

vinny_twoshoes@reddit

they are using it as a replacement for the r word, which everyone decided is cool again i guess

[-]

anubus72@reddit

google "RDJ full _"

[-]

another_dudeman@reddit

Tropic Thunder

[-]

eufemiapiccio77@reddit

lol exactly is it the vibe coding meme scene from iron man but again make it clear for us

[-]

forgot_previous_acc@reddit

Never go full

Tropic thunder

[-]

bwmat@reddit

I refuse to chase metrics, and bring up ways in which they are dumb or causing perverse incentives every chance I get (especially in meetings including my manager)

Luckily I'm in a position where I'm not worried about losing my job, but to be honest, I don't think I'd behave differently if I was (I've never claimed to be responsible...)

[-]

chickadee-guy@reddit

We are tracked on Github comments, code autocomplete %, prompts per day, and time spent on VPN. Lovely company culture

Offshore are full throttle with pushing slop too

[-]

roodammy44@reddit

Time to start driving full speed towards the edge of the cliff. People who try to hit the brakes are fired.

[-]

Dreadmaker@reddit

We’re solidly in ‘learning mode’ right now. I’m part of a startup that’s at around 150 people now - so a little past the ‘startup’ phase I guess, and into the ‘bureaucracy phase’ - but still, it’s a startup, and we have a lot of very flexible people.

We went ‘ai first’ and ‘all in on ai’ for building things on march 1st of this year (we have no user facing ai in our products and likely never will). We’re not holding anyone accountable for metrics or anything of the sort, and everyone for now is acknowledging that all of the stuff you brought up - the counting LoC, story points, tokens, etc - all of them are dumb. Instead we’re really focusing our energy on how exactly to wield the firehose that is Claude.

We’ve found, as I’m sure most companies have, that our dev team has accelerated way faster than product or QA. So QA is backed the hell up and some dev teams are starving, waiting for product to refine the stuff that needs doing and make decisions there.

We’re experimenting. We just launched a really nifty experiment where we have a super minimal team - one developer and a couple of support staff - where we’re breaking all of our typical workflow rules and doing homework on whatever we need to do to go faster in that way. There’s no product person - the dev is really the product person as well, in that setup - and we’re just trying to experiment with what that might look like and whether the overall pipeline makes sense with a small ‘break out’ team like that.

I can see a world where the tokens or whatever else start to be metrics that get attention paid to them. But for now, we’re approaching it in a pretty solid way, honestly, and it feels good so far.

[-]

NoChemical7637@reddit

sounds like your company's running on chaos theory

[-]

Bodine12@reddit

I use AI to come up with rationale for elevated story point estimates then use it to 'npm install', then check-in node_modules so I'm winning on every measure.

[-]

MissinqLink@reddit

I have a cron script that runs every 15 minutes and translate one paragraph of the silmarilion to to French then back to English and commits it to a ln internal repo. I’m just waiting for someone to bring this up.

[-]

Murky_Citron_1799@reddit

I do the same with merge requests and commits. I have more merge requests than the next 11 developers combined. 3x the next best person. It's been like this for almost a year and nobody has noticed. If they notice Its all fake I'll just say whoopsie, it was a hackathon project I forget to turn off. Oh well no big deal since we don't look at such metrics for performance, right??

[-]

People that aren’t modernising at all are in danger though

[-]

Sheldor5@reddit

the most degenerated idiots run IT companies (or are managers at those)

how? why? nobody ever told them how funking dumb they are?

[-]

NaBrO-Barium@reddit

Pretty sure we’re all degenerate idiots to someone in our lives 😁