Struggling to move fast enough at work
Posted by ghostphreek@reddit | ExperiencedDevs | View on Reddit | 8 comments
Hello all, I am a senior data engineer with \~5+ years of experience. I recently joined a new org (4 months in) and I am realizing that I am having a hard time keeping up with everyone around me. I was leveled as a senior when I interviewed for the role but everyone seems smarter and faster then me at this org. A vast majority of the IC's in my part of the org are L2's there are one 3 L3's (including me) but all the L2's feel really experienced for that level.
I constantly struggle with getting in my head on the right way to approach a problem. For instance I have 2 OKR's this quarter one of which is to clean up our snowflake instance. Thing is I haven't done much work on that front because I keep going back and forth on how we should structure our roles or how we should name our warehouses. Or take today as an example. There is this process that we need to productionize since it currently exists as a jupyter notebook. I went back and forth all day. Should I just try to force it to conform to our dbt patterns? Should it be its own service? What are the trade off for each? How much tech debt is this if I slam it into our existing DBT pattern?
By the end of the day I was able to use Claude to produce two prototypes. But then the data scientist just let me know that we would refactor it in Q3 and we can just run it as a notebook for now. I felt like I wasted a bunch of time but based on the number of times this thing needs to run he's kinda right. But also I think I am kinda right since if I go out someone else is gonna need to run it so it might be nice to have a paved path for them. But with Claude these days they can just ask the agent to run the notebook and there will be no issues....
On and on that process goes and I feel like I make so little progress each day.
TL;DR: I constantly just stare at my screen thinking of all the possible ways to complete my work. But I struggle to just pick a path and move. How do I get over this without constantly looking like an idiot picking the wrong path.
UnderstandingDry1256@reddit
Leverage all the smart mates in your team, do not try to be “smarter”.
Ask them how they would approach the problem, and after some time you’ll get enough confidence to decide yourself without hesitation.
That’s what all the most successful ICs and EMs do.
tenthousandants44@reddit
You spent 1 day prompting a chatbot to build a POC. You're supposed to throw that away. It would be a waste of your time to try to productionize the slop
MindlessTime@reddit
Is your DS team held in high regard at your company? It may be that DS is rewarded for moving fast, even if that means they produce sloppy, fragile, un-reproducible work. “…we would refactor it in Q3 and we can just run it as a notebook” gave me that impression.
If that’s the case, what you’re dealing with is a company culture or a systemic issue, not a skills or code issue. You need to read the room a little, see the discrepancies between what people say they want to see and what is being praised and rewarded day-to-day. It may be that creating a solid data pipeline is what managers say they want, but “just get it to me in two days” is what they actually reward. They won’t make the time to take the extra steps to do it differently. They’ll just let one team be fast and sloppy and make you accountable for the downstream failures.
What you do is document. Diagram the data flows. Find the pieces that can be cleanly separated by some boundary like an API or a data contract. Enumerate those pieces. Call out that some pieces (like the notebook ones) are important for velocity and you don’t want to slow people down, so put those on the backlog. In the meantime, build monitoring and alerting. If DS wants a notebook solution so they can not spend the time doing it right and get a pat on the back for quickly showing progress in the next weekly meeting…fine. You’ve defined what those notebooks need to produce (schemas, latency, uptime) and created alerting in a public slack channel that tags the DS team if something is wrong. You can force them to own their problems that way. Make the failures loud and public and part of their workflow. Management needs to see the pain they cause, not just the quick results they put out for show.
ghostphreek@reddit (OP)
Yeah this is the follow up that I am following. Close the OKR, document the tech debt, put it in the backlog, move on. But I’m trying to speed up the loop from when I make that judgment call instead of just sitting in my head.
Flat_Shower@reddit
4 months is still ramp. You don't have enough context yet to feel confident, and that's normal.
The notebook call: the DS was right. Shipping something you'll rip out in Q3 is not momentum. Knowing what not to build is literally the senior-level skill.
For the Snowflake stuff, pick any reasonable convention and ship it. The naming scheme doesnt matter. The cleanup being done matters.
ghostphreek@reddit (OP)
Yeah this is tail spin that I’m trying to pull out of. I keep hearing things like “that is the thing a senior would do” and thinking shit I wouldn’t have done that. My judgement is fucked up which is why I think I am incorrectly leveled. Leading to more anxiety. Leading to more bad judgement calls. Etc….
xcVosx@reddit
Discuss your proposal with whatever LLM your company gives you. Start with a line like: I'm a very indecisive data engineer and need guidance on which option to pick.
It sounds like you're getting into a decision paralysis loop.
If you can't break it yourself lean on tooling or make tooling to help you.
Software isn't perfect so our solutions don't need to be either. Finished today but needing to be refined later is better than never finished.
ghostphreek@reddit (OP)
Yeah this is fair but lately I’ve gotten better at debating the llm. Since some of its suggestions are flawed since it lacks the entire context or suggests something that doesn’t fully suit what I need.
But this is a good point. I do try to lean on llms for this