Any actual success stories in measuring teams' performance, efficiency and quality across an org?
Posted by horserino@reddit | ExperiencedDevs | View on Reddit | 74 comments
My company's upper management is currently restarting the cycle of re-defining how teams should work, how to improve the company's productivity, improve quality, bla bla bla.
Part of this is rethinking and imposing how teams work (rituals, meetings, etc) and how we measure teams' performance.
But when I think about my work experience (10+ years), I don't think I've ever seen a success story where a company implemented performance and quality metrics that were actually meaningful and that could be leveraged for tangible improvements.
In practice, I mostly feel that process and team improvements are often not measurable in any way that is useful.
Has anyone got any actual success stories on this topic?
flavius-as@reddit
Pick a pain point.
Start measuring it for a few months. Tell everyone their income depends on it.
Now you got people more aware of that problem. Potentially improved things.
The goal was never to measure performance, it was to get things done.
Now change to the next big pain point.
przemo_li@reddit
Toxique.
Nowere here is mentioning of budget for changes. This risk wishful thinking / bullying, when improvements are unrealistic.
That budget is usually secret ingredient to changing failing process into good process.
Mornar@reddit
... well fuck me sideways.
This is genuinely the first instance of useful approach to measuring performance in any way, shape or form in software development that I've seen.
RusticBucket2@reddit
I genuinely can’t tell if you’re being sarcastic or not. Bravo.
Mornar@reddit
You know what, I'd love to clarify but at this point I really can't. On account that I don't want to.
Key-County6952@reddit
Its funny because we can tell that you're lying. It isn't on account thay you don't want to.. seems you genuinely aren't sure.
Mornar@reddit
You ain't got nothin' on me. You can't prove anything, copper.
RusticBucket2@reddit
Understandable.
Potato-Engineer@reddit
You're right: if you suddenly say This Thing Is Important, everyone knows several Things that can be fixed immediately. And then, after a few months/years, you've hit all the low-hanging fruit, you've done some good damage to the medium-hanging fruit, and you're approaching the more-trouble-than-it's-worth fruit.
I've heard this comes up in industrial situations: a lot of companies have some kind of "make an improvement every month" or "report a safety issue every week" program. Once the program has been running for long enough, everyone is making up things to say (or "improving" things by toggling back and forth between two states), and the reporting/improvements are no longer useful.
TheOneWhoMixes@reddit
I think this largely depends on the culture, how it's communicated, and how much trust IC's have in "the system". I could see some organizations where the above "game the gamers" approach would work great, because by grabbing the low-hanging fruit you've given your teams the opportunity to hone certain skills related to The Thing and to get better at measuring The Thing.
But in others it would just feel like constantly moving goalposts with no tangible goals or purpose.
flavius-as@reddit
That's a great point. Added "game responsibly".
AhoyPromenade@reddit
I like the way we work at my current place. We ditch all of that. Use base camp’s shape up thing. Cycles of 6 weeks on feature. Pitches written up in advance, estimated, adjust teams on them to match the scope or push back if beyond capacity.
In terms of getting features out, customers like the predictable cadence. Devs like the working on one thing uninterrupted. Product bought into all this so there’s no desperate ´we need to do X next week’. Requires mature people.
przemo_li@reddit
How does that work for companies that found success but have kitchen sink product? To many things to do, to few devs. Splitting features for to basics and stringing them in one-of-each dreams to be only way to keep progressing at sufficient speed.
Potatopika@reddit
I worked in a place that used shape up. We did not have mature product people. Even though we had pitches nobody knew what should we do and by the end of the sprint we always needed more things in the last couple of days while during the middle of the sprint there was no feedback whatsoever
Maxion@reddit
Never leave that company. You've found Shangri-La
AhoyPromenade@reddit
Downside is that salaries aren't brilliant... only reason I'm browsing the market currently, I really like it here otherwise.
congramist@reddit
What we talking here? Because I will sacrifice salary in the 10s of thousands for the right place.
phoenix1984@reddit
If you make enough to live comfortably, money ain’t everything. I know a lot of people who deliberately make less than they used to because it makes for a better life, myself included.
Note, being poor sucks. If you’re struggling to get by, then go get that bread.
loganbrownStfx@reddit
Shape up is the best. Loved doing it at my old place
Byzaboo_565@reddit
Fuck
wyldstallionesquire@reddit
How do you handle tech debt and support requests?
AhoyPromenade@reddit
We have an team that deal with day to day customer data stuff, then we rotate round on support for those guys so maybe 1/8 weeks we’ll do a week of that. There’s 1 FE and 1 BE on that at a time.
At end of cycle we do 2-3 weeks of refactoring/bug fixes while some estimation stuff is also done.
drungleberg@reddit
I just joined a company where we do shape up, they have done it for years but I've came from generic Jira sprint background. It is quite a change, but I enjoy no estimates or refinements.
What I struggle with at the moment is product is very hands off, to the point where we are almost defining the product roadmap just based on what we think is needed.
How have you found it with product etc?
ashultz@reddit
You guys need to fire those product people and get ones who do their job. The best development processes in the world are pointless when the projects going in to them are "I dunno, just build some things".
I've had great success with shapeup like processes with an engaged product team. When product is checked out then you just finish the project and find out it isn't right.
AhoyPromenade@reddit
I was similar, pretty skeptical. it's definitely not perfect, we do have some issues where the high level estimates are way out, usually underestimating actually.
Product are generally very good, technical people, so it hasn't been an issue. I should say that the product is technical, maths-y, so having generic product owner from generic consumer facing products wouldn't work at all.
loganbrownStfx@reddit
Shape up is the best. Loved using it at my old place
bombaytrader@reddit
Wanna tell you a funny story . My company started tracking time to close cycle item for tickets . Now to game the system , ppl started closing tickets at end of sprint and opening a new one for same work .
Perfect-Campaign9551@reddit
Why would you even want to try to? Sounds like a possible toxic thing
BigPurpleSkiSuit@reddit
The root issue is that most of the time, leadership tries to solve a problem from working backwards on increasing output, and not by asking developers where they're being slowed down, and then solving those problems. Usually leadership starts from DORA metrics, measuring output in the form of PRs/Storypoints, stuff like that, and they view surveys as too soft/not hard data.
Start by surveying/asking your developers about their processes, learn where they're being slowed, i.e. are build times taking too long? Are you in too many meetings which reduces your ability to focus on deep work? Is documentation poor? How's the codebase experience? Then go in and attack where the problems are across the org. Reduce build time, reduce meetings if necessary yada yada Then and only then should you measure some form of output, but don't make it a goal, anywhere. People are smart and will happily turn 1 PR into 10 if they know they're being judged on it.
I've always hated the phrase, 'you can't improve what you can't measure' you can definitely make improvements and feel their effects even if you're not measuring them. I.e. you build 40 widgets a day instead of 30 but don't measure that you're building more, you still get more money from selling 40 widgets. It's the same with reducing inefficiencies. The way you do that with engineering IMO is by asking where time is being lost/wasted/squandered, where the developers are being slowed down and not able to do the work they usually want to do.
Source: I work for a company that does pretty much exactly what you're asking about with pretty good levels of success, depending on how much the company wants to put into solving the issues we surface, and can help fix based on our recommendations.
danielt1263@reddit
You might want to give the DORA metrics a try... https://dora.dev/quickcheck/
Top_Violinist5861@reddit
Agree.. we measured LTTC across a fairly large organisation, I think it got teams releasing to production more often, which is generally a good thing.
Sheldor5@reddit
upper management is 90% useless that's why they have to create work for themselves to have a justification for their jobs
octopus4488@reddit
Upper management here.
You are right. Half of my competition for CTO jobs are old MBA guys using Microsoft Edge and calling it world-wide-web. What else would they do? Set up 12 new recurring sync-meetings per day and produce insightful reports.
spaceneenja@reddit
“Our job is to compare all of you! But in the ene if cuts are made we will keep the teams of the people we like anyway.”
incredulitor@reddit
I've seen it really work once... in semiconductor manufacturing validation. The output wasn't software.
Some independent variables were:
Dependent variables looked like:
The closest I've seen software get to these kinds of issues are devops metrics. This experience led me to push for more frequent measuring of test coverage and more traceability between bugs and development practices at later software jobs, to varying degrees of success but usually with some measurable improvement.
Another isolated case where I saw something work more like this was a long-term effort to improve log messages in an enterprise product. This involved much more direct collaboration and information-sharing between support, professional services and developers than other work on the same product ever did. Results were smaller logs, lower cognitive overhead for customers, PS and support to find what they were looking for, faster turnaround between that and issue resolution, and shorter lifecycle of bugs discovered in the wild.
ImYoric@reddit
I seem to recall that Google had a great project to try and do that, at least 10 years ago, and concluded that nothing worked.
Izacus@reddit
Google and other big companies constantly measure team, product and process performance and then implement changes to improve them.
"nothing worked" is a meme.
spaceneenja@reddit
This is more about product performance and reliability though, not measuring “output”.
It’s actually useful because of that.
Izacus@reddit
There's plenty of "output" measurements as well - blameless postmortems are a way reduce amount of bugs in the code and had that as a metric.
The teams you still try to pretend that don't exist are usually called engineering productivity ("eng prod") and regularly have OKRs that are output metrics you claim are useless. At several companies.
spaceneenja@reddit
Lol, you’re so quick to argue with me that you miss the point of my post and I guess this whole post.
I will reiterate: the stuff you mention is actually useful because it’s not arbitrarily comparing meaningless “engineering output” measures (read: jira velocity, loc written, etc…).
SRE is more about controlling for product quality and preventing teams from sacrificing that for feature bloat, which is a problem in 99% of engineering organizations.
Blameless postmortem are antithetical to measuring engineering teams by their very nature.
ImYoric@reddit
And how do they measure?
Adorable-Fault-5116@reddit
No. At least, from the perspective of an IC, no.
The teams that have felt the most successful to me from my perspective, were the ones with the smallest amount of oversight, made up of people who give a shit.
No amount of process shuffling gets you either of those properties.
imagebiot@reddit
It’s a sign upper management is doing jack shit.
Maybe they should like, do some real work. Like the rest of us.
Inside_Dimension5308@reddit
We did try using metrics like story points, PR count, lines of code, bug count etc to measure performance. And I had predicted that this will not work out. After 6 months of filling the metrics, we discarded it. The reason is that the metrics can be gamed. As a lead I know for sure that the metrics has been gamed because I know what everyone is capable off. People were clocking 30-40 story points while average was 20. Nobody bothered to review the story points. If you have to verify the metrics credibility, the metrics itself is a failure.
Just let the team lead do the performance review based on his observations and also ask the developer to self review. Day to day interactions are enough to determine how a developer is performing.
Objectively parametrizing the performance review process is something I am aligned with.
traderprof@reddit
In my experience, focusing on one clear metric that impacts customer experience rather than dev productivity has been more successful. For one team, we measured "time to close critical bugs" rather than generic velocity metrics.
What made it work was that everyone could see the direct link between this metric and customer satisfaction. We didn't dictate processes - each team could determine how to improve their specific metric.
The key was that teams owned their improvement method rather than having it imposed from above.
PanZilly@reddit
Have you read https://itrevolution.com/product/accelerate/ ? The basics are there. Key take aways: * measure outcome (customer satisfaction), rather than throughput or output (deployment speed, stability etc) * with that, when measuring something, always use several metrics to answer a question, and combine facts with perception. Use a combination of surveys and system metrics * don't go for levels or maturity matrix, but look for trends and improvements * context is everything, so be careful how teams are compared
There's loads of reading and research to be found following this book. Dive into DORA / state of devops, and developer experience metrics frameworks like SPACE or DevEx. Also learn about the basics, lean flow and continuous improvement.
Metrics like SLOC or mttr or deployment frequency alone will encourage devs to game the system, they get better performance for management (their numbers up) but quality will suffer and no one will feel except the customers. If you read up on reaearch and you can explain to your management why and how using just DORA and agile velocity is going to get the company in trouble, then you're on the right track.
That said, it is difficult and takes time to implement but management always wants results faster which is impossible. You'll want to convince them to start small, some metrics on some team that is willing to explore this.
Context is everything not just between teams but also between companies. If some company says we were successful, follow our steps. And you copy their exact moves. You'll fail
Potato-Engineer@reddit
Yeah, if you measure actions rather than outcomes, then you're going to completely ignore "glue work" that keeps people and teams working together effectively. (Related: the infrastructure work, and the dev-quality-of-life work, always gets deprioritized.)
Triabolical_@reddit
Pretty much any measure is a surrogate measure and can therefore be gamed, and devs tend to be good at games.
I did have a friend who used average bug age as a goal and got reportedly wonderful results. I am a zero bug advocate and that seemed like a nice way to incentives it.
Sworn@reddit
Ah yes, the zero bug policy. This was forced on teams at a previous job, but no feature development pause was allowed to actually spend time fixing the two plus decades of legacy which produced the bugs. Instead, bugs were simply closed ("because it was so long ago it was reported "), turned into "feature" tasks which weren't tracked, or passed along to some other team which could possibly also own the code producing the bug.
Triabolical_@reddit
Yes. We used a sliding scale based on the age of code and gradually expanded it.
I do believe in closing bugs that you truly are never going to fix.
Potato-Engineer@reddit
I find that sad, but it's the way it is. If it's low-enough priority, then there's always some higher-priority work.
Why can't we just have infinite amounts of time to polish the codebase until it gleams like the chrome bumper of this truck that's about to run me over?
sadFGN@reddit
That's the best way to get rid of bugs. Just close the task that has been sitting on backlog for months and wait to see if anyone raise it again. lol
ConsulIncitatus@reddit
What?
I'm a senior leader and I do not care about any of this at all. I care about results. I don't care how they get things done.
horserino@reddit (OP)
Lol. That is BS, or at the very least a surprisingly naive take.
The usual dance is: Top execs put pressure on upper management to improve business outcomes and upper management translates that into "improving operational efficiency" which is then turned into "how do we measure that improvement" which is then pushed down to teams to implement according to whatever BS guidelines are spawned in this particular cycle of this dance.
If you've never seen this I sincerely doubt you being a "senior leader".
So my question here is if anyone has actually any success stories on these reworks/reorgs/metrics/guidelines dance in the real world.
IMO this dance is mostly a waste of effort and misguided, but came here with an open mind to get other people's experiences, but so far most people agree with that measuring team's "productivity" is pretty useless.
So tell me senior leader, _how_ do you care about results? And if you're not getting them, how can you tell and how do you deal with it?
ConsulIncitatus@reddit
I report to the CTO and have three directors reporting to me.
Keyword being outcomes. Doing scrum better isn't an outcome, it's a mechanism to achieve an outcome. My boss could not care less whether we use agile or something else, because the methods are not important. If he were to start doing that, it would come across to me as micromanagement. An outcome is "we sold more product", "we made an important customer successful with our product", etc.
I have, much earlier in my career when I reported to ineffective senior management. When a company is pointing inward and describes key results as "we increase the rate of motion", it's not an actual key result, in the same way that throwing your car in neutral and pumping the gas increases the motion of the engine and the car goes nowhere. The key result needs to be tied to an objective, and a bad objective is "increase the efficiency of our software shop." Why are you trying to increase the efficiency? Why does it matter? What does that help you achieve?
This is a pretty good book on how to set business objectives and track results.
What you have are junior leaders who achieved senior positions well before they were ready, because they are focusing on the kinds of things that make a good 1st level manager and not the kinds of things that senior leaders should be thinking about.
horserino@reddit (OP)
Now that is an actual answer instead of platitudes. Thank you for it.
metaphorm@reddit
we just eliminated a whole lot of "pro forma" meetings and switched to "be an adult and use your words. if you've got something to talk about schedule something ASAP. don't wait for standing meetings".
seems good so far but I dunno what we're measuring. it certainly freed up a lot of time on everyone's calendars though.
angrynoah@reddit
None of this is measurable.
Measuring a thing requires holding up an objective yardstick to it. What is the yardstick for a team's "performance"? "Efficiency"? "Quality"? Doesn't exist.
That means any claimed attempt at measuring these things has to begin with a made-up proxy measurement. It may be a good proxy but hahahhaha no it won't. Death by Goodhart's Law is inevitable.
We have to collectively recognize that software development is a creative activity. It can't be measured or modeled or standardized or predicted.
GoTheFuckToBed@reddit
For individuals I like to look at completed and delivered work (not just closed tickets) over three months.
For teams I review bug, incidents, and general velocity. And a few secret metrics ;-)
Sevii@reddit
If you want to measure cross company productivity you need teams to be consistent in their processes over time. I've yet to work anywhere where middle management didn't constantly change the processes.
Tundur@reddit
This has to be qualitative, unfortunately. When is a team performing really well, and when is it just having an easier time of it in terms of complexity and blockers? When is a team performing poorly, and when is it just an impossible task?
You can use quantitative measures to guide investigation (i.e highlight performance smells) but the investigation has to be manual. You need leadership with a deep understanding of their team and their work, who are fully focused on commercial outcomes, and who can mould the team around them.
If a team's output can be effectively measured quantitatively, then that team's work can probably be fully automated in the next 12 months, because it's not creative work.
lantrungseo@reddit
2, I can throw any metrics in the report, but that just serves as a pure report to upper management. Upper managers can enforce some other metrics, but as long as they're relevant and do not disrupt anything, I dont care.
Because I did (1) so well, (2) is always a good, and any actual output from the team is of the highest quality. If I do (2) without (1), the metric becomes the target, and there would be no good output.
roger_ducky@reddit
Any measure done for continuous improvement baselines and NOT tied to performance are good.
Anything used to compare teams or used for personal performance will be gamed to death.
So no, nothing defined by management will ever return useful data. But, the reason management does it is to reset their performance baseline, too. They’ll get promoted for their new initiative, before anyone finds out it doesn’t work.
Ab_Initio_416@reddit
Changing processes can dramatically improve productivity and quality but I’ve never seen such a program work in software development. Mostly developers learn how to game the system.
Relevant quotes and stories:
Not everything that can be counted counts, and not everything that counts can be counted.
- Albert Einstein
A drunk is crawling about on the sidewalk under a streetlight. A police officer asks him what he is doing. The drunk says he is looking for his car keys. The officer asks where he lost them—the drunk points down the street. The officer asks why he is looking here. The drunk says, “The light is better here.”
He uses statistics like a drunkard uses a lamppost—more for support than illumination.
- anon
When a measure becomes a target, it ceases to be a good measure.
- Charles Goodhart
pydry@reddit
DORA metrics are often good red flags but there are no good productivity metrics.
Critical_Bee9791@reddit
ok not metrics but what works then? team performance review where you hold each other accountable? context - i've never worked in a dev team, genuine question
Izacus@reddit
I think you shouldn't put too much weight on posts of people that never led larger teams or managed larger companies here ;)
forbiddenknowledg3@reddit
Clearly you compare velocity and story points.
SiegeAe@reddit
Team's performance? No.
People's performance? No.
Product's performance? Yes.
System's Performance? Yes.
Process's Performance? 50-50
Every single time I've seen individuals or teams have KPIs introduced in some form, I've seen shortly after all of the absolutely best employees either laid off or leave due to stress.
Reverent@reddit
You can't measure productivity, you can only measure results.
Make sure the results are measured well enough and at regular times enough to hold teams accountable.
Incentives are the next layer down. How do you make people care about results? It's not the problem of the people reporting results, it's the problem of the teams who have to produce the results.
thisismyfavoritename@reddit
nope
nutrecht@reddit
Nope but with why ;)
Antares987@reddit
Nope. Not once. Ever. In my 35 years of development. It’s like judging one’s organization of their notes in math class to predict their SAT score.
AngusAlThor@reddit
Any metric which becomes a target ceases to be a good metric. This kinda thing always fails.