How do you deal with "I have no clue what is the problem whatsoever" moments?
Posted by Affectionate-Mail612@reddit | ExperiencedDevs | View on Reddit | 66 comments
Something doesn't work when it should, or something works when it shouldn't (like bug reproduction).
Once you crossed 5 yo, it feels shameful to say that you just don't know what's going on. Sometimes the solution is adding more logs, but if it's networking or OS quirks, then I'm pretty much in the dark here.
It doesn't happen often, but it leaves me shaken a bit when it does.
HowTheStoryEnds@reddit
I like the ones that I don't know and have to chase, at least they're generally a pleasant distraction.
The ones I don't like and which are more plentiful are those 'I told you so' things that you warned about in advance, management waved away but then I get to fix when they inevitably occur. Those fill me with resentment and dread for a longer time: resentment at the moment and dread long before since I knew it'd be coming.
coderstephen@reddit
It should not be shameful. Nobody knows what's going on with everything all the time.
Do you know what impresses me more as a staff engineer? When you know you know, and when you know you don't know, and admit as such.
But as far as how I deal with it:
Sulleyy@reddit
If you are a person who can figure stuff out, you will be asked to figure out harder and harder problems that no one knows the answer to
WorkFrmHomeAstronuat@reddit
You shouldn't feel like you have to have all the answers because of some amount of career experience. I felt this way at 5 YoE, now I have 10 and I just accept that I can't know what's going on all the time. At my company there are principal engineers with twice as much experience who don't understand things, but they know how to figure stuff out, and they're not ashamed to communicate transparently.
BraveResearcher3037@reddit
I don’t think I’ve ever had a bug I couldn’t figure out via some combination of stepping through the code locally or a bunch of logging except for concurrency issues mentioned before or other issues that only happen at scale. The second class of issues you have to have some type of harness to do load testing.
Oh and the third type where my machines environment is different than production. That can often be alleviated with Docker.
ikkiho@reddit
fwiw the "i have no clue" moments stopped scaring me a few years in. they're a signal to back off and binary search what i've assumed. last bad one was a service hanging on tuesdays only, ended up being a misaligned cron on a sibling box keeping a stale tcp connection open through our LB. took 9 days. honestly i felt dumb until i realized everyone else on the team had walked past it for a year too.
Shazvox@reddit
Having no clue what's going on is fine. As long as you know how to get a clue.
ericmutta@reddit
Indeed. And investing in knowing your debugger inside out is a great way to get many clues (for example the VS2022 debugger is legendary).
BraveResearcher3037@reddit
Which doesn’t help if the issue is a concurrency bug and only shows up at scale…
ericmutta@reddit
Concurrency bugs are the devil's work :)
Though the VS2022 debugger can help if you only want to pull out just half of your hair on such bugs, as opposed to all of it (e.g. you get parallel stack view, task view, threads view, ability to freeze/thaw threads, conditional breakpoints, multiprocess debugging, attaching/reattaching to running processes...all for free in the Community Edition).
DeepHomeostasis@reddit
the times I can't get a clue are usually when Im debugging the wrong layer. For networking or OS stuff that means putting down the code and watching the boundary, tcpdump or strace shows you what left the box instead of what the code says it does. Then bisect the system rather than the source, does it repro with a raw curl, without the proxy, on localhost, each answer halves the search space
Keeyzar@reddit
I don't know but I'll find out is a valid answer.
If it's taking longer, you can either ask for resources (colleagues, new software, ..) or ask for more time.
If your pain point is network, then ask for time to learn networking basics etc. etc.
Icy-Roll-4044@reddit
Yep half the job is being comfortable with not knowing for a while
beaverusiv@reddit
It is the main thing I look for when hiring engineers is if they can say they've made mistakes or don't know something
Gunny2862@reddit
Disconnect. Do something else until your brain is ready to take another look at it.
engineerFWSWHW@reddit
That's the time you need to brainstorm with other folks.
On my previous work, the software devs have faced a unique and very rare issue (happens once or twice a year) where the recorded timestamp is always the same. This is an embedded industrial handheld device running windows CE and c#. But when they changed to a new device, everything works. The software devs just blamed it on the SQL CE.
When i got hired, a customer reported this. I looked at the hardware schematics of their board and looked at the hardware RTC. On the bad unit, I saw that the 32khz oscillator clock isn't oscillating. When i physically pressed the RTC (BGA), it started oscillating and the timestamp starts incrementing again. It seems it needs to be reflowed.
If they had involved everyone at the beginning, the issue could have been resolved many years ago.
SecretaryAntique8603@reddit
People have already covered the political/optics angle.
Actually solving the problem is more like tactics. You need a structured approach. Break down the various layers in the solution where there could be a failure. Think about how a failure could look in those respective layers, and how success looks. Inspect each boundary. Isolate the failure.
Write down all your findings, questions and theories in a stream-of-consciousness document. Make a table of confirmed and suspected instances. Find data points and look for correlations. Make a custom dashboard visualizing relevant datapoints to look for correlations. Form hypotheses and reject them based on what you find. Look for recent changes or deployments that could be related.
Quick wins: Turn off the caching. Rollback suspected components to a previous version. Use debugger breakpoints and git bisect.
carrboneous@reddit
There's no shame in not knowing, and there's no reason not to ask. And sometimes even if you just ask someone else who has no idea to sit and step through the issue with you, things click into place.
But if it's your responsibility to know and nobody else is equipped, I think the way to proceed is to mentally step through it, making sure you know what happens at each stage (whether it's layers of the stack or portions of your code) and map it out until you can see what might be the problem.
Going back and reading the error message/stack trace more and more carefully also tends to help. Sometimes it's right there and you've just screened it out because of the unfamiliar (or over familiar!) jargon, but when you read it, it just says what's wrong.
And then there's just copying the error message and/or a description of the issue straight into Google (or an LLM) and seeing what comes up.
not_a_db_admin@reddit
Saying it out loud helps more than I thought. First time I told a staff dev "I genuinely have no clue what's happening" I braced for judgment and got "cool let's pair on it." Seniors I respect most are the ones quickest to admit they don't know yet.
Noah_Safely@reddit
Wrong mindset. A new challenge is something to appreciate not fear. Tired of solving the same damn class of problems year after year. Over a long career I'd say that's one of the biggest drags.
The job is really cultivating a set of core skills with deep knowledge of basics, then being able to jump in deeply to any rabbit hole & later forget it all. Hopefully while taking decent notes..
No one knows everything, those that front like they do are just being insecure and it's a huge problem. Not just professionally but personally. It's really easy. "I'm not sure offhand but I'll do some digging and find out".
Highly recommend giving the book "Thinking, Fast and Slow" a read.
plugiva@reddit
On the ground, I stopped seeing those moments as a sign that I don't know enough.
Most of the time they're a sign that one of my assumptions is wrong, incomplete or simply invisible to me at that moment.
What usually helps me is shifting from:
"I need to find the bug"
to
"I need to find the gap between my mental model and what the system is actually doing."
Sometimes it's a missing log. Sometimes it's networking, caching, timing, permissions, OS behavior or something else entirely.
But I've noticed that the feeling of being completely lost often appears right before I discover an assumption I didn't even realize I was making.
skeletordescent@reddit
“ Once you crossed 5 yo, it feels shameful to say that you just don't know what's going on”
I’d challenge this, personally. I think many people put way too much pressure on themselves to say “I know what’s going on” or “I understand these systems” when they’re very complex. I’m 8 years or so in at this point, and there are absolutely things I’ve built which I don’t remember exactly how they work and need a refresher when they break, there’s no shame in that.
mpanase@reddit
It's surprising how few times I hear "I don't know". Or even "I didn't understand what you just said".
In my current company there's only 3 engineers who I've heard say either of those 2 phrases: the only 2 I trust at all and myself.
skeletordescent@reddit
The more time goes on the more it's a giant red flag (sometimes) when someone I know doesn't know enough says "Oh that's easy" and starts typing. Had a DevOps person blow past some pretty big warnings and nuked an entire environment because of this when we were all asking them to please slow down so we could fully understand what was going on.
mpanase@reddit
Oh, I have one of those.
His favourite phrase is "This will fix it. I guarantee it. I'm 99% sure".
No, it doesn't fix it. It's actually a stupid action to take and it will get so many clients (and leadership) so angry when you tell them "it's fixed" for the fifth time.
He is a team lead, though. Can't stop him without getting into a war.
AlpacaNuts@reddit
100% agree.
If anything I feel less and less shame for not knowing something as I gain more experience. It happens to me, and I see it happen to others I respect, frequently enough that it's not a big deal. And I have enough evidence to know I will figure it out. Or maybe the shame can only start to kick in after the 5th year. :)
_predator_@reddit
This. Liberate yourself from the idea of having to know everything all the time. "I'll find out" is what matters.
Also, if admitting you don't know something leads to others shaming you for it (OP didn't imply that, but saying it anyway), leave that environment. That's no place to be.
skeletordescent@reddit
Knowing how to find out, how to understand the solution and the problem, and especially how to explain it to higher ups and others, is the real skill. Knowing everything all the time cold is a fool's errand.
fixermark@reddit
Usually when I'm in that state, what's really going on is "There's a very deep-seated assumption I have about what's going on that is wrong." Finding it will be as hard as a fish finding the bad part of the water, because you're swimming in your assumptions. But it can be found; if it didn't exist, the thing would be working already.
(Personal favorite example: had a 3D shader in a game engine we were working on that, for our CEO's laptop, made everything black. I was stumped, completely. Our senior software engineer, looking at the shader, multiplied the final output by 1.00000000000000001. Color came back.
I asked why he thought that would work, and he said 'The shader compiler is written by the same people who make the chipset. Sometimes, they aren't the best software engineers. I've seen the compiler just... Be bad before. I guessed the 0 was coming from it optimizing out the line that sets the color and figured if I forced a math operation on that line the optimization would be disallowed.')
downshiftdata@reddit
I interview for this. I ask specific questions, not to test what you know, but to test how you react when you don't know.
If you try to bluff me, won't admit that you don't know, etc, then you fail the interview.
If you are humble, honest, and inquisitive, then you pass.
I learned this from someone who interviewed me over 17 years ago, and it's stuck with me ever since. And I'd like to think I have a very good rate of return on my interview recommendations.
My advice is to be comfortable with not knowing. Be comfortable admitting that you don't know, that you need help. And when you get help, learn from it.
latamakuchi@reddit
As others have said, you let the stakeholders know it'll require investigation, timebox digging time and get to checking wtf is happening. Perfectly valid to say you don't know... yet.
As to how to dig in? The first thing is knowing what is failing, then backtracking from there: reading code, documentation, asking other devs if it's a part of the codebase you don't know. Knowing what variables and paths are involved helps debug things and informs you as to possible failure causes. Then come up with a hypothesis of what's the issue and test against it.
Usually that's it: either you need more info about the problem or you're making assumptions. The key is making sure you're testing what you think you're testing and having an hypothesis to test against. And also changing one thing at the time or you might not know what fixed it.
x-jhp-x@reddit
I've found that if I've exhausted all options, and gotten additional feedback from others who don't have an easy/great solution & are also stumped (without having them go through everything directly, just perhaps maybe going over my thoughts about potential problems & how to track the issue down), going for a walk or sleeping works well. I've found a lot of my coworkers do the same too.
mpanase@reddit
I need a thread to pull from.
The reporter needs to give me a timestamp, a correlation of data points, and event, ... something.
And/or, logs+telemetry.
If we don't have it now, the next release is just a bunch of new logs+telemetry. And reporter, please tell me asap when it happens again; we'll catch it on the second try.
CowboyBoats@reddit
The Richard Feynman method:
Super glib but when you've got zero ideas, alls you can do is write stuff down. Write down what you notice about the problem. Write down steps to reproduce the problem. Can you isolate it to a unit test? Look through the git history. Can you isolate it to a specific commit or a time range of commits? Talk to people who have eyes on the problem and get them to show it to you, look through it together.
sirspidermonkey@reddit
Once you're past about age 5, there's this weird expectation that admitting you don't know what's going on is somehow a failure.
I'd argue the opposite.
Systems fail in new and unexpected ways all the time, especially when hardware is involved. If you already knew exactly what happened, the question would be why you allowed it to happen in the first place. Investigation is part of the job. Most professions rely on diagnosis. Doctors order tests. Lawyers research case law. Engineers collect data and troubleshoot. If you've never seen a problem before, you'll need time to understand it. Even if you've seen similar symptoms before, the underlying cause may be completely different.
Expertise isn't knowing every answer instantly. Expertise is knowing how to find the answer efficiently, recognizing the most likely causes, and ruling out possibilities in a systematic way.
To borrow a medical analogy: if a patient shows up with a headache and muscle aches, the possibilities range from a common flu to something far more serious. An experienced doctor doesn't magically know the diagnosis on sight. They start with the most probable explanation, gather evidence, and adjust as new information comes in. They'll treat the suburban soccer mom for flu and not start isolating for ebola.
Good engineers work the same way. The value isn't in never saying I don't know. The value is in being able to say I don't know yet, but I know how to find out.
DigThatData@reddit
first and foremost: you need to push back against this.
When you are assigned a task, your responsibility is seeing that the task is resolved. Full stop. If you don't know how to solve the problem, that doesn't change your responsibility at all. It just means that rather than implementing the solution, your responsibility is now tracking down who has that ability and working with them. Maybe this means asking them for help. Maybe this means straight up delegating the work.
Your responsibility is seeing that the task is resolved. If you don't know how to resolve it on your own, being transparent about that isn't shameful: it's professional. If you don't have the humility to ask for help, the work won't get done.
The failure here isn't your personal want for skill: it's the ticket that is still open.
For added context: I'm a "unicorn" MLE. I've been coding since I was in elementary school in the 90's, have been a data/ml/ai professional for close to two decades, hold an MS in math/stats, and it's generally been the case that I'm both the best engineer and most knowledgeable about the math on pretty much every time I've participated in. That said: ML IS A GIGANTIC FIELD AND IT'S IMPOSSIBLE TO KNOW EVERYTHING.
It does not matter how skilled or knowledgeable I am. I do not and cannot know everything. The field moves extremely quickly. The state of the art changes daily. The standards of practice and tooling shifts faster than anyone can keep up with them.
You are a member of a team. Your responsibility is to your team. Your teammates have skills you don't and know things you don't and have heard about things you haven't heard about and have done things you haven't done. You have skills and knowledge they don't, and you don't (or at least shouldn't) hold it against them when that is revealed and you have an opportunity to fill gaps for them.
An effective teammate is someone who has the humility to ask for help and doesn't waste time putting the request out there. If you don't know how to do X/Y//Z thing: it would be great if you can teach yourself quickly, but if it's going to take two weeks to learn it on your own and someone in your proximity already has that knowledge and could resolve the thing in a few hours, you are being professionally irresponsible trying to tackle it completely on your own.
Being assigned an issue isn't like a homework assignment where it's important that it's "your work". What's important is that the thing you are assigned gets done. This is why managers get to take any credit for anything: even if they don't touch a line of code, their responsibility is seeing that certain objectives are achieved. Full stop. If they delegate that work to someone else, the objective was achieved and their responsibility was satisfied.
It's ok not to know things. There's too much to know. You are not expected to know everything. You are expected to get things done. That means you are expected to ask for help. Your teammates want you to ask for help.
Be shameless = be effective. Don't be ashamed when you're struggling. Be ashamed when you delay communicating that you are struggling.
HalfSarcastic@reddit
I've had many moments like this. I took me quite some time to learn how to preserve status in such situations.
First step - don't try to look like you know everything. Don't try to impress everyone. Technologies are like vehicles - any can have any kind of issue. Some issues can obvious, some will require lot's of work just to find the reason, especially if it is not your codebase.
Second step - don't be afraid to ask stupid questions to get as much details as you can. Some of those details might be the key to the solution.
Third step - be honest and let them know that it can take some time even for you, because usually such issue doesn't happen. It is not a standard situation and you don't encounter it every day, so you might need to do some additional research and analysis to figure this one out.
diablo1128@reddit
Moments? lol .... Unless I've done some investigation I have no idea why a bug is a bug 99.99% of the time.
I say I don't know to everybody all the time. I let them know I have to do some investigation and get back to them. It's never been a big deal to anybody I've worked with in my 15 YOE.
oditogre@reddit
It's never shameful to ask. I'm near 15 YoE, I still openly, shamelessly ask experienced colleagues in situations like this. I still sometimes get that little flutter of anxiety when asking a question I think others might think is stupid, but I've never been made to feel that way except very very rarely by people who are just assholes in general.
What's truly shameful and will get you rightly looked down upon by your peers and leaders is letting an easily solvable engineering problem turn into a difficult or unsolvable timeline problem because you wasted days or weeks spinning your wheels researching something somebody on your team knew the answer to.
I've seen that happen. It's a clusterfuck and absolutely shreds your reputation; you'll have people micromanaging you for a long time after that, and not unfairly. I never wanna be that guy.
Just suck it up and ask. It's the easier path in the long haul, and most decent seniors / leads / architects will respect you for being right up front with "I don't know this thing, please help."
metaphorm@reddit
I'm at 15 years of experience and I still run into things like this on a regular basis. there's no shame in not knowing things. this feel is vast and deep, and production systems are very complex.
what I have learned is to be disciplined about forming hypotheses and testing them. Going through a rigorous process of formulating falsifiability criteria and then gathering the telemetry to falsify or validate a hypothesis.
SteveMacAwesome@reddit
Once you get to 5yoe you need to get more comfortable saying you don’t know, not less. You’re never going to know everything and the skill that justifies your salary is your ability to work the problem, find out what’s going on and then correct it.
“I don’t know” is the beginning of the story, not the end.
gemengelage@reddit
My gotos are:
Keep in mind that you're probably working on the difficult task because it's a difficult task and you're the guy for difficult tasks. If it were an easy task, you'd give it to a junior and if a junior had the task, he'd probably hand it to you or ask you for help.
The task is expected to be difficult. Not immediately having an answer is not a personal failure on your side but rather an intrinsic property of the task. You're also (hopefully) not alone. Ask a coworker. You can't be the subject matter expert for everything and everyone has a bad day from time to time.
actionerror@reddit
If you don’t know, usually there’s always someone who knows better than you. Communicate and use problem solving skills. At the very least, I can pinpoint as to where the problem is and I’m never left that clueless.
SurpriseOk6927@reddit
at 19 i gave up pretending to know everything. the devs who act like they have all the answers are usually the ones shipping bugs. honest confusion is a feature not a bug
Uhgley@reddit
There's something oddly comforting about the fact that even people who've been doing this for years sometimes have absolutely no idea what's going on. Once the problem gets solved, everyone tells the story like it was obvious all along but most of us are just guessing our way through it..
thelowtreadmill@reddit
the trick is just being upfront about it early, like "i'm gonna need to dig into this but here's my plan" beats sitting there pretending you know for three days.
TwisterK@reddit
Client = “We will investigate and update the findings before lunch break”
Team = “investigate and update to me and I will deal with client, long term wise, our system observability are not good enuf so it took us more time than expected, find way to improve it”
Me = “look into the system from higher level and see if that any gaps that our tools missing or didn’t connect that other members can’t see then guide members toward that direction rather than giving them solutions straight away.”
pathema@reddit
nice breakdown. thank you
ianpaschal@reddit
Be honest. I try to always give _some_ answer even if it’s “I’m afraid it might be one of several things and I won’t know for sure until I’ve eliminated a couple of the options.” Better if you can say what your plan of attack is in terms of investigation but that’s not always possible.
It feels obvious but I’ve learned it makes such a difference to sort of be transparent about your problem solving approach and other parties are extremely grateful for updates even if they’re not all good news.
AttemptNo499@reddit
It happened last week with a bug we were having, I could only reproduce on the UAT server. On the next morning it did not happen anymore, i replied with what we could eliminate and what could possibly have fixed the issue.
sean_mu@reddit
here, 20+ years in engineering. Nobody knows everything, you just happen to find something you yet to discover. and keep in mind that, the more you know, the more you don't know...
Todo_Toadfoot@reddit
These situations are why I am a SWE. Everyone else saying a weird problem exists and no idea what it is. I have a look and debug it and show exactly where and how to fix the problem. Best stuff there is. Have found many bugs this way in third party libs or "features" for build tools and apps. Also love these types of things to get introduced to a code base because you learn what is actually going on across multiple layers and domain knowledge this way.
elperroborrachotoo@reddit
Once I crossed 10 yoe, it felt liberating. You don't have to wait for so long.
*
domepro@reddit
why would it be shameful? its a normal part of work, especially when you're dealing with large and/or old codebases. It's much better to be the guy that says he doesnt know, finds out and communicates than a guy that can't say I don't know (this also includes no and other variations) and then everything just falla apart when push comes to shove.
Politics aside (this one will usually be employer-specific) the most important thing is to communicate early, often and with brevity rather than make shit up and just go bury your head deep in a problem and disappear from the radar completely for a while.
Inf3rn0_munkee@reddit
18 yoe, I'm honest about it.
To the customer: we're looking into it.
To the manager: I have no clue what's going on YET, probably need more logging/telemetry. (Helps to explain the next few steps too)
hooahest@reddit
Take a deep breath and try to look at the big picture again. Make assumptions, or re-examine some assumptions that I had before
J_revolution@reddit
"I'll look into it" is my default response. Or "I'll get back to you".
If it's transient, just say you're monitoring it.
Difficult-Vacation-5@reddit
It was a typo.. it always is 😭😭😭
mRWafflesFTW@reddit
You need to speak the litany against fear.
I don't know but I can figure it out.
brazzy42@reddit
Focus on how to get more information. If it's a.networking problem, that could mean getting familiar with Wireshark. If that seems too difficult, you may need to consult an expert. Depending on how big or urgent the problem is, paying a high hourly rate for an expert freelancer may still be better than dicking around aimlessly or taking weeks to acquire the necessary knowledge yourself.
randomInterest92@reddit
Just recently we had this and being able to assertively say "i don't know yet, but I'm still looking at various possibilities" is important. Obviously if you are really completely out of options What'd be causing it, then you're probably lacking some knowledge that someone else has to help you with.
Recently we had a customer complaining about some pages loading forever. I was still new in the company and something I've asked pretty soon is "do you use browser extensions, proxy, firewalls, anything?" And they said no (spoiler: they lied).
Then i went down a rabbit hole on our end just to get told 3 weeks later that the issue was a proxy on their end. I couldn't tell because the issue was a rare occasion. All the traffic that got to us looked fine. There was nothing for me to find.
mixxituk@reddit
You got it! Crime scene forensics is the way
The world for me collapses until it's put back in order again 😅 I wouldn't be able to sleep at night
Thank god for app insights
Immediate_Rhubarb430@reddit
These are the most exciting moments
YahenP@reddit
If you think you know what the problem is, you're most likely wrong. For me, every mistake starts with the thought, "I have no idea what's going on, and I'm not even sure there's a mistake." This is perfectly normal and professional. Rushing to judgment is our enemy.
sxeli@reddit
Its natural when you don't have an answer immediately to the problem. It has nothing to do with your experience as an engineer but only how long you have known that particular system.
I like to take a step back and ask about other related side effects to the problem and as some have pointed here - you should ask other colleagues about it as well. However, its important to have a plan to figure the problem out rather than just "oh idk"
M_dev20@reddit
Reset all my assumptions and start debugging from the very beginning. For the stakeholders, this translates to: "I am currently investigating the issue and will provide more information as soon as possible."