Have you ever had to debug the compiler?
Posted by dystopiadattopia@reddit | ExperiencedDevs | View on Reddit | 86 comments
I just read an otherwise attractive job description that mentioned debugging as a key task. Ok sure. But it went all the way from the code and third party libraries to debugging the compiler "if necessary."
This kind of floored me. In my entire career I have never had to debug a compiler. Tbh I don't think I would even know how.
Does this particular requirement indicate a mess of a system and/or workplace, or is this just an unusual but not unheard of task?
mechkbfan@reddit
Best to clarify if this a miscommunication or if legit
It's feels like it's been interpreted the wrong way but the hiring department.
But if the team lead is a 60yo who once found a bug in the compiler back in 1992, and now anytime they can't explain an issue, they start debugging the compiler, then I'd probably look elsewhere lol
xeric@reddit
Yea this feels like a miscommunication - probably meant more like debugging compilation issues or using LLDB
ShoePillow@reddit
I've used gdb, but not lldb. Is it similar/better?
xeric@reddit
I’ve been out of the C++/ObjC world too long to remember well, it’s probably similar
bteam3r@reddit
Yep my first thought was "HR wrote this"
dacydergoth@reddit
Do I know him? Of course I know him. He's me.
alxw@reddit
Once, in 26 years. C# issue with optimisation, worked with a debug build, but not a release build, submitted report to MS, had to deploy a debug build into prod until MS released a patch.
vaughands@reddit
Happens time to time. Rule of thumb though: if you think it's a compiler bug, it's probably not. But I've had it happen once or twice in the 15 years I've been doing this. :)
matthedev@reddit
I've submitted a small patch for an open-source build tool before, but I've never debugged the compiler. I'd wonder what kind of crazy code is being written where needing to debug the compiler would be coming up or whether they're using some in-house or forked compiler.
Presumably, this job isn't a systems programming position.
CodrSeven@reddit
It's never the compiler :)
Unless you wrote it, then it's always the compiler.
moreVCAs@reddit
doing systems work you find compiler bugs occasionally.
idontlikegudeg@reddit
Yes, but I would not call that "debugging".
ninetofivedev@reddit
If you're working in modern web stack? No, you're not debugging the compiler.
If you're writing kernel drivers, embedded, games, etc? You might have to debug the compiler.
tcpukl@reddit
Yeah I've had to before. I work in games.
I've had some random logic bug or crash and when looking at the assembly view out was caused by the compilers optimiser breaking the logic.
Had to send a bug report off to get it fixed, but also massage the code so the compiler didn't break.
idontlikegudeg@reddit
So you found a compiler bug, but did you really debug the compiler?
vivec7@reddit
This sounds like the sort of thing that I'm likely never going to need to do, yet would find the challenge uniquely intriguing, and would hate it if I ever actually had to do it.
SpiderHack@reddit
Its really interesting to find, I found a bug on e in gcc by using $ as the first character of a cpp vsriable name. It passed parsing, but utterly fubar the linker.
I had been writing php recently and hopped into the ACM programming competition as a lark, and whoa buddy did it feel cool to find an issue instead of solving any problem (out team didn't prepare at all, but 1 year we solved 1 problem, which doesn't sound like much, but is better than like half the teams, lol)
hypnofedX@reddit
GCC was the first compiler I ever used and hearing it brings a smile to my face!
garlicNinja@reddit
Yeah dumb question.
Doctors have you ever had to do heart surgery? If your a surgeon probably if your an eye doctor no.
StandardSignal3382@reddit
Maybe a TS to JS transpiler then?
TechgeekOne@reddit
I can confirm it definitely happens in games. The Direct X Shader Compiler (DXC) has a nasty habit of miscompiling on the SPIRV path. It's rare, but it has bitten me me a few times.
Rust does it sometimes too with random ICEs when you use certain nightly feature combinations but if you're using nightly that's expected and usually it's a matter of "update your version" rather than "go debug the compiler".
NewFuturist@reddit
In the old days of web dev "debuggging the compiler" meant realising the API for something is broken in Firefox/Chrome/IE and searching for some way to get around it or hoping the a polyfill will help you.
idontlikegudeg@reddit
Speaking about Java: I often debug through the JDK provide libraries, and I have also already found bugs in the compiler, i .e., the compiler not accepting valid source code in one instance and producing wrong output in another. But I have never actually "debugged" the compiler as in stepping through the compiler source code, and I think people who actually do that are very few.
I think this is probably a case of having someone from the development team specifying a list of requirements for the candidate and someone from HR without technical background putting it together in text form. I would simply ignore it, write that debugging is a task I do on a daily basis, and simply not mention the "debugging the compiler" thing. That’s something that can be talked about in an interview or should someone from the actual dev team call you in order to shorten the candidate list before making interviews.
rorschach200@reddit
I'm a GPU/ASIC compiler backend engineer, so yes, I debugged the compiler a few times.
But on a more serious note, I have seen a notoriously large volume of cases where teams and people call "compiler" just about any script that does something with the code, however basic.
So the people behind that job description are either full of shit, or HR misunderstood something, or the team has is using some sort of basic JavaScript or even configuration file transpiler and if it's homegrown, it's probably buggy. We could be talking 2000 LOC script here. And that's what that is referring to.
Particular_Camel_631@reddit
Microsoft c/c++ 7 had an issue where it couldn’t handle switch statements with more than 255 cases.
We were auto generating the code via a 3rd party tool.
I wrote another tool to turn it into repeated if statements.
This is the only genuine compiler bug I ever found in 35 years.
StandardSignal3382@reddit
If you work for a giant C++ shop for example it might not be uncommon where you may find compiler bugs and may even have to contribute patches
Drugbird@reddit
While it is possible, 99% of the times when people think there's a compiler bug it's instead just a regular bug (often undefined behavior) in their code.
The compiler is really one of the last things you should look at when looking for bugs.
tcpukl@reddit
Though I've found 2 I think over my career in games.
Normally in early console prototype/unannounced level toolchains.
stormskater216@reddit
yup, my sole compiler bug was for an early version of the Android JNI compiler
wisconsinbrowntoen@reddit
/u/Drugberd probably meant you aren't likely to find a bug in the C compiler
Numerous-Addendum816@reddit
sounds like they're setting you up for some serious challenges.
fudginreddit@reddit
Ive never had to debug gcc or anything like that but Ive definitely spent ALOT of time digging through external library code and having to debug that. We even had guys open pull requests on big libraries Im sure everyone here knows.
It really depends on what youre doing I guess but I dont think its an indicstion of a bad workplace
Fish150@reddit
Ive found errors in typescript before... I didn't debug the code, but I helped provide examples to the team at Microsoft.
It was a fun little bug to find.
https://github.com/microsoft/TypeScript/issues/52475
lolCLEMPSON@reddit
If you are writing a compiler, yes.
n4ke@reddit
Are we talking a commong language or an internal DSL? Because for the latter, debugging the compiler is not that unusual, unfortunately.
For generic languages, I don't think I've ever seen much more than figuring out quirks of compiler versions. Unless maybe we're talking really in-depth low-level or performance optimized work.
Skullclownlol@reddit
What I've seen this usually mean is not "the compiler has a bug", but "we have a few different compilers and build option presets, and they generate different outputs, which may mean that the ASM doesn't necessarily align in key ways that are required on platform X but not platform Y or in sim Z". So then you look at the ASM and alignment to validate/fix.
PoopsCodeAllTheTime@reddit
It’s not so daunting if looking at a self hosted interpreter
ArticlePotential5825@reddit
what's the main topic of discussion here
boring_pants@reddit
I've found and reported compiler bugs before.
I haven't run a compiler under a debugger, no. But it's fundamentally no different than debugging any other application.
Why not? If you've got the source code and debug symbols and everything, you can just attach a debugger and step through the code. The tricky part is understanding everything it's doing, but if you have access to the source code you probably also have access to documentation and a group of people who know about it.
nullvoxpopuli@reddit
I work in javascript, so I tell folks "your job doesn't stop at the node_modules boundary"
this is to encourage them to dig in to code that the company doesn't own, and realize that all code is made by humans (at best), and can have errors that may be worth investigating. Working around an issue causes confusing for others, and assessing what type of issue you're seeing could make things way easier for your own work if a problem is fixed in the correct location.
xyz214@reddit
I’ve read the cPython code in one or two occasions - not to bug but to understand its inner workings.
Outside-Storage-1523@reddit
Sounds like a fun job. I'll take it over whatever I'm doing, considering my hobby is hacking the kernel.
Diligent-Floor-156@reddit
Yep had to, about 10y ago. I'm in embedded, at the time working with a niche proprietary MCU having a core that hasn't been used by anyone else in a long long time (8 bit). The toolchain was a gcc port made by a contractor, and was severely outdated (branched out from gcc 2). At some point the flash was full and we noticed the compiler was missing tons of size optimisations, so we asked the contractor to upgrade the toolchain to a more recent gcc version (knowing they hadn't worked on it for years).
Well that was a year full of surprises when compiling the project with the new toolchain. Had to dive quite deep, to understand what's going on and to create simple examples, as I couldn't just send them our whole codebase.
Had no prior compiler knowledge and still doesn't really have any, except for the many bumps I encountered.
landmesser@reddit
You debug the output of the compiler, not the compiler itself.
Most compilers break and behave strangely when given a to big file like >100k lines in one single file.
In C/C++, because of header inclusion, that number can be reached sooner than you think.
Fabulous-Possible758@reddit
When C++ template metaprogramming first came around, it was because they basically discovered that they had implemented a programming language in the type system by accident. Before it became much more standard, you could easily break compilers doing what are now pretty common tricks.
Patrick-Poitras@reddit
I've had about 10 bug reports for the Common Lisp compiler in the past 4 years. It's a small community and my workplace has a significant amount of code vs. the compiler's internal tests. Sometimes someone is doing something weird on our end and it triggers compiler bugs.
The SBCL team is really good at fixing those, and then we all celebrate the compiler being better.
ContraryConman@reddit
What is the actual domain you would have worked in for this?
The_Northern_Light@reddit
Yes, at my first Silicon Valley job we actually had a few verified compiler bugs. We were writing c++ for a VLIW processor, mid 2010s.
newprint@reddit
They are full of shit. You will find just handful of people who really know how modern compilers work and that work req. PhD and some good knowledge of architecture.
I once had to communicate with Intel Compiler team because of some issue with C++ compiler I had. Back in the days, Intel C++ compiler team was in Moscow, Russia.
If you have issues with compilers, first thing is to read assembly and see if you seeing any issues and then message the compiler team
Zenin@reddit
Once upon a time compiler design was basic CS curriculum, not post-grad. Is the Dragon Book no longer a standard text?
One doesn't need to hold a PhD to identify a compiler bug, even in a modern compiler. And while modern mainstream compilers are certainly more advanced than the past, the basic architectures and stages are largely the same as they've ever been. Understand those basic architectures and you're well on your way to at least narrowing down what stage the bug is occuring, which can go a long way to at least identifying a workaround while you wait for the vendor to provide a patch.
Some examples of that can be particular issues in the optimization stages which often have flags available to disable particular optimizations, offering the ability to workaround the bug without sacrificing all optimizations. Or maybe it's a parser bug that can be worked around by a well placed comment or noop statement.
newprint@reddit
> Is the Dragon Book no longer a standard text?
Sure, if you want to write toy compiler, Dragon book is good starting point, but that has nothing to do with actually trying to "debug" modern C++ compiler. Have you ever looked into Clang ? I had a privilege. Did you see all modules it has to optimize your code ? Yeah, good luck with dragon book figuring it out.
Zenin@reddit
Unless you're on the compiler dev team itself, that's significantly farther than users would likely ever need to take a debugging effort.
But to be able to debug deep enough to identify a bug exists and possibly narrow its location down enough to write a half-decent bug report and/or a workaround (such as disabling this or that optimization flag), for that practical user level debugging the Dragon book level of understanding is plenty.
I'm also old, so I'm probably biased here; Back in my day debugging core dumps was just another day that ends in Y for a mid level developer where today that's considered black magic even among so-called "senior" devs.
mjbmitch@reddit
OP is full of shit. It’s an AI-generated post.
newprint@reddit
I'm naive then. God dammit, getting old.
mjbmitch@reddit
One of the easier things you can probably do is ask yourself if the premise of these sort of posts are real and go from there. Analyze it like you might to a post on r/ amitheasshole.
For this one in particular, OP frames the most outrageous requirement in the job description as a requirement to debug “the” compiler (see my note below at the end). What an absurd yet plausible requirement. In the comments, however, they go even further and one-up themselves by claiming the job description also has a requirement to debug the OS. There is no way anyone would ever write a post specifically about a job description having absurd requirements, enumerate them in the post, but not mention one of the most absurd requirements until they bring it up in a comment.
Note: nobody is ever going to write like that (referring to a set of compilers they would need to debug as “the” compiler) unless they’re inexperienced (which makes the requirement plausible if it were written by someone from HR…). If someone is going to paraphrase something like that, they’re either going to capture the intent and write it correctly or acknowledge the absurdity of the way it’s written and write it exactly as it’s written (… neither of which OP did).
Keep in mind that these sort of posts also like to include an “edit” section as part of the original submission. E.g., LLMs have been trained on posts with edits which cause them to in turn create posts that include literal text fragments that say “edit:”.
I don’t think this post was ever actually edited. The edit section claims it was added to clarify questions but I don’t see anyone asking for clarification on the languages. Why would that be? The section was always there. Nobody ever needed clarification because the information was always in the post in the edit section.
happy_guy_2015@reddit
Yes, many times.
Memorably, starting with one of my very first C++ programs! Turbo C++ 1.0 had a bug in the code generation for (default?) copy constructors, where if I recall correctly, the generated code would wrongly try to copy-construct a sub-object from itself, which for the string type would result in a crash. So basically pretty much any code using non-trivial classes with string fields would hit that bug. That was a closed source compiler, so debugging the compiler in this case just meant isolating the issue and filing a bug report.
keelanstuart@reddit
I remember my father, upon hearing me exclaim that it must be a compiler bug, looking at my code and saying "nope, it's you." That was in the very early 90's, using Turbo Pascal. The complexity of compilers, especially with templates in c++, has definitely increased, but I've never found a legitimate bug in one. Sound card drivers, yes. Compilers no. It's fascinating to read about about though.
CrushgrooveSC@reddit
Yes lol.
They often have bugs, or some magical ridiculous hoop you have to jump through in order to get to actually emit the instructions you want. (Looking at you cmovg)
bobotheboinger@reddit
I've worked embedded and developed, delivered, and integrated custom ASICs, and while i have had to debug the compiler a couple times, it really was less than a handful over more than 20 years. I would find it very uncommon. I think they are just being a bit hyperbolic in their description.
If it's an embedded position, I might look for someone who would feel comfortable turning on assembler output to verify that what was compiled generated appropriate machine code, but that would be the lowest id expect most people to get to.
Rough_Response7718@reddit
I work with legacy technology. Id say yes all the time, stuff is very picky and whenever environment slightly alters it breaks.
CheithS@reddit
Did it once - a long time a go - and found a bug in the MS C optimizer.
IPv6forDogecoin@reddit
You do find compiler bugs from time to time. I put my personal rate at about 1 bug per year.
engineered_academic@reddit
Yuuup. Sometimes you need to especially if you think your tools have been compromised. Several attacks have not only compromised binaries but also the tools used to build those binaries so even if you build from source it is still compromised.
teerre@reddit
Depends what you mean "debug". If you mean considering the codegen not being optimal, yes, mostly clang, but also rustc and the ocaml compiler. If you mean "my compiler crashed", then no
UncheckedMoonrise@reddit
In one research project on a supercomputer, I had hit not one but three (!) bugs in a closed-source compiler. I looked through the assembly it generated and guessed what code changes might prevent it from falling over.
So yes, I did have to debug a compiler, though it was a black-bod job, not a “step through a compiler with a debugger”.
jake_morrison@reddit
Normally the rule is that if you think you have found a bug in the compiler then you are wrong.
Compilers for embedded CPUs can be pretty flaky, though. Bugs are common enough, even (especially?) when you are paying $2000/seat for a license.
In my 30+ year career I have found bugs in Perl, PHP, and Erlang, but it doesn’t happen often.
CrazyFaithlessness63@reddit
In embedded systems development (AVR, PIC, ARM Cortex) yes, it's not uncommon. The result is usually not 'fixing' the compiler though, more adjusting your code to avoid the bug or just to optimise the generated code. On AVR chips for example (which are Harvard architecture, separate code and data memory) it's more memory efficient to use nested if statements than a switch statement; the switch statement generates look up tables for jumps which consume precious RAM while nested if live purely in the code memory (EEPROM). Looking at the assembly generated by the compiler helps pick things like this up.
For Java, Go and Node you are very unlikely to hit a bug in the compiler that hasn't already been detected and has work arounds. Debugging the OS is probably more performance tuning than anything else (unless you are writing device drivers or other kernel level integrations). For that stack it just sounds like they are trying to sound more technical, I doubt anyone is inspecting assembly output or spinning up gdb to see what the OS is doing internally.
dystopiadattopia@reddit (OP)
That was my suspicion as well, unless there are other technologies at work that they didn't list. I've spent most of my career as a Java dev and the most I've ever had to do was identify bugs in third party libraries (or bugs caused by improper implementation of those libraries). And a few times I had to dip into the guts of SQL Server, but that's about it.
GhostMan240@reddit
It depends on the compiler I guess. If you are using some niche proprietary thing I could see it. Anything modern and widely supported there is essentially a 0% chance. In my 6 years of experience in embedded I’ve never dreamed (had nightmares?) of such a thing.
janyk@reddit
The closest I came to that was when I was debugging and found out that Oracle WebLogic's classloader wasn't loading JAR files in the correct order. So, custom classes that should override vendor classes weren't being loaded before the vendor classes (this is the Java EE ecosystem, for those who don't know). That's more comparable to debugging a linker, I think, but it's on the same level of hell as "I can't trust a single damn thing about my dev tools now and now I know that I have absolutely no idea what's going on and no control over the most basic code I write".
To answer your question: that was the one and only time I encountered such a problem in my \~12 year long career so far. So, it's rare. But it might happen. And if it does I don't think it indicates a mess of a system or a workplace because they simply don't control the development of compilers/linkers/application servers/JVMs (unless they have custom versions of those).
necheffa@reddit
I have diagnosed multiple bugs in the Intel compilers and GCC.
Sometimes the codebase is pushing the limits of the language standard. Sometimes the implementation is just bugged.
GCC really has a leg up in that most major versions are used to build itself, the Linux kernel, and the entire userspace of a typical distro. So if you are just
apt get install build-essentialsor whatever and using the compilers packaged by your distro, they've been through a very rigorous regression test suite, chances of you finding a bug are slim. If you are a special snowflake, it isn't unheard of though.zugzwangister@reddit
I ran into a weird C++ compiler bug many years ago.
It worked fine with logging. It showed the bug without.
Okay. My logging statement changed something. Obviously.
But it didn't. Turns out, as long as I had a comment right before a certain line of code, it worked. Stepped through the assembly code. It was fine.
Remove that one comment, and the assembly code was different and not fine. Or I could replace the comment with an actual line of code and it would work.
Zenin@reddit
Borland, right? Lots of us hit that parser bug in the late 1990s.
zugzwangister@reddit
MSVC++
Dimencia@reddit
Specifically in C# with Maui in Windows, doing certain things wrong enough will crash Visual Studio when debugging, and you get prompted to debug the crashing debugger (ie, Visual Studio, which is also the compiler) - so maybe they mean something along those lines. It's not even deep systems work, but it is using some specific native APIs that like to break things
ianbanks@reddit
C++ compilers used to have (especially in the 90s) not terribly infrequently internal crashes on constructs that weren't that odd, especially when using newer C++ features. The usual solution wasn't debugging the compiler though, it was finding a way to change your code slightly to stop triggering the internal crash, because it isn't all that practical to manage patched toolchains across a team or to wait for your patch to end up in a stable release.
Recently though I did have to modify clang (to get MC/DC coverage on a particular embedded target working), and my impression of clang's code base was that it wasn't that much different in terms of quality and complexity than any other large non-compiler C++ code base.
zicher@reddit
Yes
Triabolical_@reddit
I was test lead for the VC++ compiler for a few years, so the answer is "yes", at least to the extent of trying to figure out if submitted behavior is a bug.
90% of the optimizer errors that we got submitted were undefined behavior.
I don't know why you would look for this in a normal position.
informed_expert@reddit
I had the misfortune of working at a place using an old version of C++ Builder. The number of bugs that compiler and associated frameworks had was maddening. Even after we upgraded to the latest version, I found myself filing bug report after bug report to the vendor. The best kind was when the bug involved undefined behavior in the compiler such that the result was non-deterministic.
high_throughput@reddit
Lmao, without further information I'd say they're full of shit but in a good way. Basically signaling that they don't shy away from technical challenges. Maybe they even identified a compiler bug once and they're eager to brag about it, even though all they did was file a ticket upstream that was quickly closed as a duplicate.
Maaaybe you'll end up creating an MCVE for a compiler bug report, but I would generously estimate a 0% chance of you ever having to dig into the compiler's codebase.
obscuresecurity@reddit
Debug one no.
Realize one is wrong, and need to figure out what to do, I've had to do it a few times.
PartBanyanTree@reddit
I would take this as a bombastic over-enthusiasm by the job description. Like we're looking for someone so committed they go from A to Z and beyond!
But, I mean, just in case, check, of course. It _could_ be that they're working with some pretty obscure custom compilation targets, custom hardware, where the compiler is not as robust/dependable/secure/established
I've run into a few bugs in the compiler over the years but they're exceedingly rare, to be honest. But I'm usually dealing with vary stable platform choices. Mind you I once did a stint at General Electric working on a compiler software that targeted various custom PLC, so that's the other side, and a very very neglected series of vary custom compilers that have aged very badly (ie, the only way to write code for this custom piece of 15 year old hardware is this even older piece of badly maintained software that currently only has, like, 6-12 dudes kinda working on it). So from that perspective I'm amazed anyone anywhere ever gets anything to compile!
roger_ducky@reddit
I’ve had times when I had to go “yup, specific optimization flags generated the wrong code” before.
Typically it’s not about hooking the compiler up to a debugger but looking at the intermediate output to see if it matches what you meant, and if there are things you can do to force it to generate what you wanted.
MapLarge614@reddit
Yes. Rarely but that happened. The code generator produced invalid instructions in some rare cased. The producer fixed it for us. Twice.
Linaran@reddit
The most I did was notice that some C++ compiler didn't support some feature from the STL and I would switch over to another one. I did write my own compiler for a thesis so I technically did debug a compiler :)
Hixie@reddit
Depends what languages and compilers you work with. I've done a lot of work both in my career and in my personal projects that involve cutting-edge versions of compilers and so it's not at all uncommon to find bugs in them.