How to handle bugs you can't reproduce?

I've worked on logging/tracing solutions for various products for the past 18 years. Your advice, in my experience, is not optimal. Again IME, when devs rely on logs to find an issue it is because of two major red flags: 1. their work is not built to be testable. Most often, people resist attempting to reproduce something with the phrase "i can't reproduce it", and when digging into it i almost always discover the lack of capability to set up a test case for reproduction creates an amount of work so big, that nobody wants to put on an estimate, and nobody wants to "delay" their work for, and nobody is particularly excited about. This can manifest as lack of granularity in APIs, as confused levels of abstraction, common anti-patterns (stuff we consider simple-to-avoid like global state cause a magnificent amount and variety of problems) and sometimes just a skill gap (e.g. someone is unaware what they can do with instrumentation and fault injection). 2. A telltale sign of this second thing is when people follow this advice, to dump as much internal state as possible in logs and try to work backwards, attempting to divine how a certain state combination might have happened. IME not only does this take an incredulous amount of time, but also it reinforces a lack of understanding (and lack of willingness to understand) the workload the software is put under. I see this a lot with library authors, and issues one might label as "performance problems". My personal favorite is when the issue someone attempts to find is a deadlock / priority inversion issue which results in no (additional) logs, leading to heaps of dead code written to detect a problem which is waiting for the right tool to be used. I would (and I know many of you would not) also state this as a deficiency in testability and testing. I know most devs see tests as demeaning, not fun, and (heard this once, what a day) "non-creative" work and yet ever-growing lists of regressions don't ring an alarm bell to them. I would also at this point like to point out that one cannot see in a log something that is not already thought of by the programmer who wrote the logging statement in the first place. Unsurprisingly, the weakest link is always our assumptions. Here is what I do, OP, in light of the fact that I might not have the time to re-design an app in order to make it decent: * first, work with the problem reporter to establish a description, as precise as possible, of the steps they took and/or their set up. Access to their product/testbed/software is more crucial the harder to detect the issue is. * if you have a problem description that you feel is precise enough, state some possible causes. If you cannot state possible causes it most likely means the cause is one of your assumptions. If you keep a list of assumptions, congratulations, you are far closer to the goal than you think. If not, I wish I could tell how to retrace them but I cannot. * if you have some possible causes, start thinking about how to examine them in technical terms - first without modifying the system in question (i.e. don't insert logging code), then with if the former fails * you mentioned "pour over the code to try to find anything that might cause it" and frankly you need to abandon that kind of thinking. If you read a book over and over again, you are likely to limit your perspective and gain dogmatic opinions rather than open its subject to criticism and counter, and it's the same with a codebase. You need to look at how the code is exercised and in what environment that happens. Some people on this path are fans of techniques like "the five 'why's", what works for me is asking "how" instead. "How" helps me think about possibilities that I am not aware of, and establishes the need for communication with the problem reporter. In doing so, I also find both parties' technical knowledge improves. The tools you use are mostly secondary, don't advocate for them. Instead just think of what choices do you have available to find out information about \_specific questions you pose\_. PS. i'm posting on a dump acct because i know how responses to my suggestions turn into a polemic, i've seen enough, i'm not interested