Dealing with tech debt caused by other teams
Posted by trojan_soldier@reddit | ExperiencedDevs | View on Reddit | 58 comments
In our company, we have few, small product teams which are given the privilege to touch different parts of the codebase to implement new ideas and run experiments. Imagine a group of Rockstar developers doing hackathons to unlock more revenue streams.
Understandably, these folks have limited context and time to implement clean code. So whenever their experiments are done, often times they have to move on to other highly demanding, fast-paced projects. But this leaves our teams to be the one responsible to clean up their tech debt because we are the true owners.
While our leadership understand and give approvals to address the tech debt based on our proposals, I cannot help but feel envy about this: it just does not feel right that our own team now have to address the tech debt and potentially dealing with regressions when doing so. Often times the tech debt is blocking our own future projects, so we have to deal with this first before.
How do you guys deal with this envy? Is this typical across tech companies? Even though leadership is fine with all of this, I have a sense that this is blocking my own career progression because a decent portion of my work time now is dedicated to audit and address tech debt.
angrathias@reddit
Pick one
Peppy_Tomato@reddit
You focused on the wrong thing though. That team doesn't call themselves rockstars. The OP has assigned them that label and has therefore derailed their entire message.
There's nothing unusual about this arrangement. Some teams have to move fast and be on the bleeding edge. Some teams have to turn those experiments into an actual product that the company can sell or support for the entire organisation.
We could argue about what kinds of different challenges both teams have to face and which are more fun, but it doesn't matter.
It's like theoretical physicists or scientists vs engineers. Which are more important? Hard not to imagine how cool it is to daydream most of the time and perform fancy experiments the other times, versus being on the 20th iteration trying to get the performance of a joint just right... Or the timing of the activation of that stepper motor.
If you don't like the team you're on, you find yourself a new team. If you haven't, it probably means you're already at the place most attuned to the way your mind works, and it's just a grass is greener sort of thing bothering you.
I often find myself wondering why my CEO gets paid orders of magnitude more to fly first-class everywhere and have fancy dinners at fancy places with other CEOs 😁, but then I think about business school and the idea of buying a new e-ink display to hack another todo list for my kitchen starts to feel more palatable.
Comprehensive-Pea812@reddit
those hackathon folks tend to refer to themselves as rockstars.
honestly all respected developers I know never refer themselves as rockstars. they are more like master yoda.
PragmaticBoredom@reddit
It doesn’t matter what people call themselves. You shouldn’t adopt their self-descriptions when they’re wrong.
Professional_Bat9174@reddit
I learned a great deal from a kindly Welsh man who drank tea and typed with only his index fingers. He had a gift for taking poor business requirements and coming up with a solution that made the stakeholders much happier than anticipated.
SideburnsOfDoom@reddit
Trashing the hotel room is totaly on brand for a Rock star.
effectivescarequotes@reddit
I haven't read the post yet. I just saw your comment and upvoted.
trojan_soldier@reddit (OP)
I am tempted to downvote you for acting like those folks..
effectivescarequotes@reddit
Also, for the record, I upvoted your post and your comment calling me out.
I liked the original comment because Rockstar developers and coders who think their hackathon code is fine to go to are not the same people.
I liked your comment calling me out because, it was just a great comment. I'm sorry. I know you're suffering. It's one one of those being too excited about kindred spirits on my part things.
angrathias@reddit
I will give you my nuanced opinion.
When I first started my career 2 decades ago I had this same problem with another dev who had been there before me, awkwardly my friend. He was always viewed as a ‘rockstar’ dev to the boss because he’d do all nighters and spew out an mvp that then others would need to come in and clean up the mess.
Eventually I became his boss, and I had to recognise how destabilising this was. He’d be essentially causing everyone else to look slow, because they were deploying actual working software and wasting time cleaning up his half hacked together shit.
Today, I liken this to having AI generate everything for you. It’s completely unfair to have someone knock together something made of duct tape and toothpicks and hand that over. It’s fine to create PoCs using this method that are intended to be disposed of.
Your envy can then boil down to: I want to be on teams that do exploratory work that isn’t very deep.
And you should ask yourself, what is important to you? Deep understanding of mechanics, doing architecture and system design, building out deep working projects, or is it playing with the next thing, being able to pragmatically decide to ignore all the good design knowledge you have in place of high velocity ?
You can’t really do much about the envy, what you can do, if it’s a path you want to take, is show that you can do it to your boss and try to maneuver onto that team. Do something off the clock and show it. Don’t show your technical chops, show your business value chops. “Here’s something I put together in X hours, here’s how it could be in the future and how much it can deliver in $$”
Hope that helps
effectivescarequotes@reddit
Fair, I read the post and have a more nuanced response. Give me a minute
Beginning_Basis9799@reddit
Change a perspective is it legacy if customers still use it?
effectivescarequotes@reddit
Yes, that's part of the definition of legacy code. If it wasn't in use, we could just bundle it up and shoot it into space. The pain of legacy code is you have no choice but to deal with it.
Beginning_Basis9799@reddit
The term legacy is a bs term it means the team is just not choosing to do modernizations on the product.
This concept is like saying the wife is legacy before a divorce and testing her as such. If a product is being actively sold it's not legacy
effectivescarequotes@reddit
Buddy, this isn't a debate. You're just wrong and doubling down. Legacy code is an industry term that means old and outdated code that we cannot get rid of because it is important to the product and too difficult or expensive to change. You don't have to like the definition, but that's what it is.
trojan_soldier@reddit (OP)
It is still in use
serivadeneiraf@reddit
I've been through something very similar! What’s helped me is systematically documenting the technical debt before diving into it. I use a tool called Blar that analyzes PRs and helps identify tech debt, design issues, and potential improvements. It makes it easier to justify the time spent and prioritize what really matters
ImmemorableMoniker@reddit
I have been on both sides of this, so I will offer direct perspective.
Where we sit in an engineering organization influences our perspective on what is valuable.
Being on a product or feature team, one is focused on that particular code space. We focus on customers of that space. We put effort into the code we work with every day. We can value craftsmanship a little more.
Being on a...I'll call it a Discovery Team...one is focused on new opportunities. One is focused on getting a new value booted up quickly so it can get put out to potential customers to see if it is actually valuable. The goal is learning quickly. There is a decent chance the duct-taped product does not have a market, and it must be changed drastically or thrown out. Craftsmanship is set aside to facilitate speed. The better a software engineer is, the more craftsmanship you get, but craftsmanship is not the driving value. It is speed.
Of course, this leads to friction. Existing product teams have one set of values, and discovery teams have another. This is normal and healthy. If the business was not pursuing new opportunities then it would decay. You can only scale existing products so far.
The friction you are feeling is normal and healthy. The perspective that helped me came from reading Innovator's Dilemma. If you're looking to widen your perspective then I suggest reading that book.
trojan_soldier@reddit (OP)
Thank you so much for sharing. From a career perspective, which team gets more recognition or visibility for bonus and promotion?
I understand that both teams have different expectations and difficulties, but at the end of the day which one is more valuable to the business? From what I've been reading on the internet, the Discovery Team is more guaranteed to deliver more measurable impact
ImmemorableMoniker@reddit
They're both valuable for different reasons, as you say. I encourage you to think less about "which is better" and think more about "which feels like where I want to be."
It's hard for me to speak on bonus or promotion. That is very company-specific. I encourage you to speak with your manager or higher-ups if that's your curiosity.
termd@reddit
Part of the handoff for your team owning their work instead of their team should be cleaning up all the tech debt. Talk with your manager about making that a transition requirement.
No-Economics-8239@reddit
A company's codebase isn't yours. Trying to add possessive feelings about it might be a good conversation between you and your therapist, but I don't see it being very useful in the workplace.
Tech debt happens for a multitude of years. It is a natural consequence of any long-term codebase, even without rogue teams of hackathons with commit access. Companies set priorities about how they want you and others to spend their time. That you don't feel like it is the most productive use of your time... is, again, your feelings.
If you want to focus on paying down this tech debt, you need to be able to articulate the business value in doing so. What does the business get for your efforts, and, more importantly, why is that the best use of your time?
So, you first need to explain small and short-term efforts to incrementally improve the code base. Break down a road map to get there. Then, highlight the benefits to the company each of these improvements will offer. Then, you have the task of competing with your priorities over those of the business.
Alternatively, you can try and argue that letting rock stars without sufficient domin knowledge loose in 'your' codebase is a bad idea. Offering alternatives about why it would be 'better' for your team to make such improvements instead. You just need the political clout to be taken seriously, sufficient communication skills to successfully convey your point, and appropriate negotiation skills to see your vision executed.
nutrecht@reddit
This is not what this is about at all. The problem is that they have basically a caste system of developers where "good devs" get the fun visible work and the rest of the mere mortals can then clean up the mess they created.
Awesome for you if you're one of the "rockstars", but utterly toxic for anyone else.
No-Economics-8239@reddit
Fun? Not a word I would often use in a business context. There may be types of work you enjoy more than others, but this seems pretty subjective and personal. I find it interesting you think some work is inherently more enjoyable than others. That might apply to green field coding versus legacy coding. But getting tasks to muck around in legacy and unfamiliar code bases doesn't sound like my cup of tea. YMMV.
And if this situation is a problem, I'm not sure it in the context you describe. Rock star devs have definitely become an anti-pattern. But I'm not sure that's what is being described here. Tiger teams are sometimes a useful way to solve problems in business. I agree the concept can get taken too far, and trusted developers can be granted too much power and leeway, and that's exactly why rock stars *are* an anti-pattern.
And... there is always a caste system inside a company. Soft skills and politics set the grade. The perception of the value of your skills comes secondary to both. There is always multiple levels of hierarchy in business. Be they ascribed, perceived, enforced, situational, or others. I don't see this as a fundamental problem. Having clear lines of leadership are typically a very good thing in business, and figuring out early where the buck stops inside your particular organization is always a priority for me.
nutrecht@reddit
OP is describing teams that are doing PoC/hackathon work. Because the figuring out how to get shit working part, without the being responsible for your choices part. To me that's 'fun'.
If you don't think that's the case our views are too far apart to be able to have a discussion on this subject I'm afraid.
Idea-Aggressive@reddit
Interesting
RusticBucket2@reddit
Disagree.
OP, don’t ever get involved in a debate with stakeholders about refactoring. You were hired because you know how to write code and they don’t.
You do the refactoring because you’re a professional and that is part of writing good code.
Do not ask for permission to do your job correctly.
trojan_soldier@reddit (OP)
Thank you, I believe everyone is on the same page here: We just want to negotiate the timeline of the refactor, not asking for permission if tech debt should be addressed or not - they should be addressed once it becomes cumbersome to build on top of it
MoreRespectForQA@reddit
You shouldn't negotiate the timeline either. The only dial you should give product/management control over is product quality - i.e. how much time *in total* to spend refactoring.
binaryfireball@reddit
thing i would angle for if you got the wiggle room - their contributions should be only viewed as PoC, meaning that when they're done, your team will get the sprint capacity to make things real. ymmv
No-Economics-8239@reddit
My point to OP was that I was picking up a lot of language about feelings and not a lot about facts. So, I focused on advice around how to handle those feelings in the workplace. It wasn't meant as completely generic advice for all audiences.
Knowing how to do your job 'correctly' is the trick, isn't it? I agree that grossing over the quality of the code base is not typically time well spent. You don't need to ask permission to refactor code. But you do need to own the results. Making improvements under the aegis of an existing task is an easy sell. Making improvements on your own, unrelated to 'approved' tasks means if anything goes wrong, that is on you. And that is typically not career enhancing behavior.
There is always a compromise to be had. Somewhere between running rogue in the code and having to nitpick and argue for every code improvement. Finding what works best for you at your workplace... also part of the trick.
trojan_soldier@reddit (OP)
Thanks for the thoughtful response.
For the first part, I do not have any problem with raising concern, drawing out the plans, and negotiating the timeline with management. They are supportive as long as it is clear that the tech debt is blocking future enhancements.
For the last part, yes it can get political if I argue that my team should do experimental work. Our team is not designed for that purpose. We can do sort of experimental work if the majority of the changes are on our own codebase, but as you can see it limits our opportunities to learn new things or make meaningful impact.
large_crimson_canine@reddit
You hope the executives like your manager more than the other team’s manager
archialone@reddit
That basically makes you the cleaning lady. The rock starts are doing 20% of the work, and you are left with the other 80%, cleaning after them and maturity their solutions.
trojan_soldier@reddit (OP)
That's what I feel.
From a business and resources standpoint, I agree that it isn't feasible to ask them to address everything because they had to touch multiple places owned by multiple teams. If they have to deal with them for each project, it means the company will lose opportunities to strike new opportunities while investors are getting impatient waiting for results.
I guess I wouldn't complain about this if I am one of those folks who have a code maintainer mentality - only keeping the lights on and not learning new things, and get paid for normal WLB.
After talking it out, this sounds like my own ego and the only solution is to switch teams or companies where I can be the rockstar
TheOneWhoMixes@reddit
Honestly, I think your second paragraph is sort of telling. Why does code maintainer == not learning new things and only keeping the lights on?
I've learned a lot by building new systems, but I think I've learned more by jumping into existing systems and figuring out how to balance resolving major tech debt, keeping the existing functionality in place, and addressing the inevitable new feature requests. Sure, you could treat it like just keeping the lights on, but why?
I think that there totally exists a spectrum, and some developers trend closer to "builders" and some closer to "maintainers". But I think that trying to assign more value to one side is the wrong way of looking at it - a successful organization needs a decent mix of both types. Too many builders, and everything's always on fire. Too many maintainers, and you'll be in refactor hell after a while.
archialone@reddit
Sadly a higher value is often assigned to the "builders" that build new cool service with the newest tech, rather than the poor schmuck maintaining a that 5 year old cutting edge POC grade project that grew into a product.
archialone@reddit
Nah, your ego is sometimes right, it keeps you from being mistreated.
You can play the game of writing documents, open jira tickets for the rockstar team, and ask them to fix it.
And bring it up with your managers, so it will be in plain sight for everyone to see.
binaryfireball@reddit
i dont envy them, their grind is probably brutal and they never get to write 'clean' code. i would hate writing hacky shit day in and day out.
trojan_soldier@reddit (OP)
That is true too. I guess the grass is always greener on the other side. The only thing I am dying to know is - when it comes to evaluation time, which team will be in a better position to get a bonus and promotion? Would love to hear the manager's perspective
Esseratecades@reddit
Unfortunately probably them. They can say they implemented valuable features in a variety of the company's problem spaces so they look like an asset to the company while you only look like an asset to the team.
guardian87@reddit
Hackathons across domain boundaries need to involve people from all relevant teams.
It doesn't sound like this should be a permanent setup at all.
MoreRespectForQA@reddit
Getting really good at refactoring tech debt is often not a respected skill, but it is nonetheless a very, very valuable skill. Valuable skills that are not respected today have a habit of suddenly becoming respected one day.
effectivescarequotes@reddit
So, I've built my career by dealing with tech debt I had no hand in creating. It sucks, but it's also part of the job. That addage about coding like the next person who is an axe murder who knows where you live hits harder than it should for me.
To answer some of you questions, and this comes with the huge caveat that it really depends on the company:
That last bit is key. Your job will teach you to opportunisticly refactor code as part of your normal duties. If you embrace the boyscout refactoring rule, and are willing to teach others, you will be fine. Most of the job is dealing with other developers' interesting code choices.
trojan_soldier@reddit (OP)
Thank you for the follow up! You brought great points. If I am looking for learning opportunities, I should look for new jobs.
For the "will this hold back your career" part, from an incentive perspective, wouldn't the rockstar team automatically get more recognition because they brought more tangible impact compared to my team? When it comes to promo time and management does stack ranking, how to sell the tech debt fixing work over the revenue work?
effectivescarequotes@reddit
If you stay in the same company, maybe However your job is a business relationship. If they no longer align with your goals, it's fine to leave.
And based on what you've said so far, that's probably what you'll need to do to advance your career. It'll probably be better for your overall mental health as well
nutrecht@reddit
So they have a team that gets all the visibility and praise, and then the rest of the org have to deal with the mess they create?
I'd love to have a chat with your manager about this.
pbvignesh@reddit
I would say at the end of the day you should be behaving like an engineer than a regular person feeling envious.
Did they actually say this to you? Or are you just assuming things? Why is reducing their damage and addressing technical debt seen as non impactful? Wouldn't you have a dashboard or anything at all reporting the amount of errors you've been covering and fixing.
You have a good problem to solve "How do we enable experimental teams to jump into codebases and make changes quickly while also not causing too much damage in their wake". It can be as simple as teaching those teams to write really good code right from the beginning. I recently read a paper from Google on how they do research (Not sure if I am allowed to link to external stuff feel free to ping me for the paper) but the idea was that they start writing "production ready code" from day 1 of their experiment.
This is not an easy thing to do however it is teachable and possible and is a skill in and of itself that could be useful for your company. You don't have to embody every single thing that Google does but you can however take the portions that could be useful for you and then proceed.
Another one such idea is that you build a small sandboxing environment for your application where new teams can quickly whip up a test instance of that application and write code for PoCs, demo it to leadership and then hand it over to you guys for a proper planned production release instead of causing regression bugs.
Its not an annoyance but it is a good opportunity to speed up research at your company and you should take it
trojan_soldier@reddit (OP)
Thank you for sharing your experience! If we are looking at impact and promotion criteria, wouldn't it be much easier to estimate by scope and revenue impact?
Compared to my team, highlighting and addressing tech debt simply to unblock other tasks do not seem impactful enough - or pretty hard to measure. When it comes to employees stack ranking, wouldn't it be easier for managers to prioritize the rockstar devs for promotions over my team?
pbvignesh@reddit
Idk how it works at your company but at my company it is upto the individual developer to come up with the metrics to show proof of work and your manager to review them. If I think I deserve a promotion it is upto me to come up with data to show to my manager and it is his job to review and verify it.
This way the promotion is entirely in my hands and my skills in collecting data and knowing how to communicate them properly. If i cannot collect the data or know how to communicate then am I really doing a good job ?
Just curious if tech debt is hard to measure how do you and your team decide which of these tasks to take up next in priority? We only take tasks if we know it is important / will cause X% increase in productivity / reduce error rates by Y%.
My recommendation would be that if you don't know these numbers then quickly huddle together with your team and discuss on how you should be prioritizing tasks going forward. Also having a good reliable north star would make everyone motivated (Everyone wants to work on impactful tasks and working on tasks that you aren't even sure is moving a needle can be demotivating)
Being a rockstar dev is not necessarily related to building cool stuff, you can have rockstar QA analysts, you can have rockstar SREs, you can have rockstar platform devs.
To me a rockstar dev in your role is a guy who proactively prevents tech debt from forming and resolving the tech debt that did form at a very rapid pace to the point where your team has bandwidth to run new experiments / build new stuff on your own and share your learnings with other teams. Ideally you could even aim to convert that rockstar team into building a stable releases on their own without relying on you
No_Bowl_6218@reddit
I'm in a somewhat similar situation. My team has been building a frontend for four years using good practices: dependency inversion, Tanstack Query, and test coverage.
Another team was formed three months ago to create a new module, and naturally, they're working independently without consulting us at all. To try and maintain homogeneity in the quality of the code produced, I spend a lot of time working on pipelines to automate rules that our team has put in place.
I use Sonar, ESLint, and other in-house scripts to help with this.
This is the only way to avoid spending an insane amount of time reviewing PRs that don't directly concern me.
commonsearchterm@reddit
Why are you not thinking of looking for a new job? You now have a clearer idea of what you want in a job now. Look for a job while you have a job.
All those stupid HR questions, you can answer pretty honestly right now. What are you looking for in your next role, why are you leaving, what does your ideal team look like etc... You have clear answers to these.
Downtown-Ad-9905@reddit
guard the code that you own better. engage with them before their hackathons to help them build context and align on approaches
DeterminedQuokka@reddit
You make your team the code owner and don’t merge until it meets the standard
trojan_soldier@reddit (OP)
For other product teams - that's the case. Unfortunately not applicable for the privileged teams.
maria_la_guerta@reddit
You need harder lines of delineation and ownership.
Most companies have split product and platform teams. The product guys are going to move fast and break things because that is their mandate. Platform guys are going to be pissed about it because that is their mandate.
These teams need to work together. Both sides need clear lines of ownership they can use to ensure that non-negotiables are respected. Once you have those, escalating is both your weapon and safety after that.
trojan_soldier@reddit (OP)
Thanks, that would be easier. Unfortunately we're all product teams. Just at different level of breadth and dept.
behusbwj@reddit
The company needs an away team model that doesn’t incentivize bad engineering. Generally this is implemented via “paybacks” where the away team is required to pay off technical debt on the host team proportional to how much work they did already. However, this should exclude the technical debt accrued during their work and it should just be assumed that they will fix that on their own