Falsehoods programmers believe about null pointers
Posted by mmaksimovic@reddit | programming | View on Reddit | 115 comments
Posted by mmaksimovic@reddit | programming | View on Reddit | 115 comments
lalaland4711@reddit
It does. As the article continues, UB means "you don't know what happens next" (or, in some cases, before), which proves that in fact it is UB.
If all UB was defined to trigger nasal demons, then it wouldn't be undefined.
Xmgplays@reddit
While the article is wrong in it's reasoning it is still true: For example the C standard explicitly calls out
Meanwhile on the C++ side I'm pretty sure that derefencing a null pointers is also defined if you don't do anything with the resulting lvalue, i.e.
*nullptr;
as a statement is not UB.Now neither of these is particularly useful, but still.
lalaland4711@reddit
I like language lawyering, and you got me down a rabbit hole.
So I guess
int* p = nullptr; return (typeid(int) == typeid(*p))
is valid, but since the operand doesn't "point[] to an object or function", non-typeid
seem like UB.basic.compound/3 says that a pointer is either a null pointer or a pointer to an object. (or one past the end or an invalid pointer). I don't think that "or" should be treated as inclusive, so a null pointer doesn't point to an object or function.
For your first example, I think you missed out on quoting the more important section:
So the way I read it I'm not so sure. Basically the standard seems to say that "if you see &*E, then you can just replace it with E" before continuing. It does not say that
*E
is safe.archiminos@reddit
That part threw me as well. Undefined behaviour has always meant just that: "not defined by the standard."
As in, anything can happen. It just so happens that it's usually the implementation still has to do something in these cases so it becomes implementation-defined.
But the whole point of it is that if you, as a programmer, write code that creates undefined behaviour, it's not the compiler's fault if it does something you don't expect.
robhanz@reddit
Sorta. There's undefined behavior and implementation-defined behavior. They're not the same.
Here's a reasonable overview: https://www.quora.com/What-is-the-difference-between-undefined-unspecified-and-implementation-defined-behavior
However, one of the key bits here is that UB, at least in C/C++, allows the compiler to do a lot of things. Since UB can't happen, the compiler is allowed to do things like omit entire branches that can only be reached via undefined behavior.
archiminos@reddit
Also this:
Was it? I'm probably just a bit too young to remember, but really? Was it? I have doubts
ShinyHappyREM@reddit
There was a time when assembly was the standard and compilers (even before C existed) were seen as slow and cumbersome, getting in the way of what needed to be done. Of course it usually involved performance-intensive scenarios, or deadlines.
You can see it still today - when compilers don't have the latest CPU intrinsics implemented, it prompts some developers to put the instructions into inline assembly blocks.
nerd5code@reddit
Often the intrinsics are exactly that anyway, just in a header.
imachug@reddit
I won't say I remember the time when it wasn't, because I'm pretty young and I don't. But I do a lot of software archeology and I love retrocomputing, so I occasionally stumble upon ancient code and discussions. I've read the sources of a couple old C compilers, including a PDP-11 C compiler that I believe was in use at the time (though it probably wasn't the original C compiler), and I've checked out posts on Usenet from back then.
And never once have I encountered the modern notion of undefined behavior there. It has always been interpreted as "certain operations may be implemented depending on what's easier for hardware". The compilers have been incredibly simple, basically the only optimization they applied was constprop and maybe simple rewrites for
if
s, so all the variance you could get was either from hardware perspective or the values being computed in different types in compile time vs runtime. We don't have a name for such a notion today; I guess you could call it "non-deterministic implementation-defined behavior"?The modern interpretation of UB has been ridiculously hard to accept for some folks. These days, there's plenty of talk about how Rust is a cult and memory safety is stupid and borrow checking is an abomination and we all should return to C -- well, imagine the same thing, but for UB. It's been argued as being an unintended side effect of unfortunate wording in the C standard, and personally I also hold this point of view (even though I consider UB to be a useful tool).
Maybe Dennis Ritchie will convince you:
I'm sorry I don't have better (or more) sources -- it's been a while and I didn't think to save links.
Big_Combination9890@reddit
I wouldn't accept this as a general rule.
There is no valid code path that should deref a null pointer. If that happens, something went wrong. Usually very wrong.
It's like dividing by zero. Sure, we can recover from that, and there may be situations where that is the right thing to do...but the more important question is: "Why did it divide by zero, and how can we make sure it never does that again?"
So yeah, unless there is a really, really (and I mean REALLY) good reason not to, I let my services crash when they deref null pointers. Because that shouldn't happen, and is indicative of a serious bug. And I rather find them early by someone calling me at 3AM because the server went down, than having them sit silently in my code for years undetected until they suddenly cause a huge problem.
And sure, yes, there is log analysis and alerts, but let's be realistic, there is a non-zero chance that, if we allow something to run even after a nil-deref, people will not get alerted and fix it, but rather let it run until the problem becomes too big to ignore.
ManticoreX@reddit
The context for this is inside of a section called "Dereferencing a null pointer eventually leads to program termination".
The problem statement is "I am Java and I want to throw my NullPointerException instead of my program terminating"
I agree with your post in concept, but you aren't actually addressing the author's point. Java DOES want to deref a null pointer without crashing so that the problem can be exposed to the developer as a Java exception. It would then by default crash but with a Java stack trace.
The whole point being that if "Dereferencing a null pointer eventually leads to program termination" was true, Java would need to do a null check and throw a NullPointerException. It isn't true, thus Java can leverage this as an optimization
Sarcastinator@reddit
Not that it matters to your point, but Java doesn't do null checks to do this. It relies on the operating system throwing a segmentation fault or whatever it does when the pointer is dereferenced and then the runtime traps that and throws.
Batman_AoD@reddit
It's pretty great that the notice at the top addresses this pretty directly:
johan__A@reddit
I'm confused, the quote says that recovering instead of checking before hand is an optimisation, but your argument has nothing to do with that.
Are you just stating your opinion on what should happen when a null ptr is dereferenced?
Engival@reddit
I believe they're saying that the simple act of dereferencing a null pointer is a design flaw, and it shouldn't happen. You shouldn't need to recover from it, nor should you need to litter your code with unnecessary checks... but you should design things in a way that the expected state is for the pointer you want to be valid.
Think of it like this: Let's say I design a door that sometimes stabs you when you try to open it. Sure, you could start adding safety around the door, like handing out protective gloves to each user before they try to use it... or, you could go design it properly to NOT stab you.
The main issue here is that the original article is talking in grand generalities. It's not a simple black and white problem. There's truth on both sides of the argument here.
Perhaps they're right about the exception handling. Rather than checking if malloc() failed (which it really shouldn't), you ignore it and let the signal handler handle it. Personally, I wouldn't do that, because it's putting too much trust in the signal handler being able to handle all possible weird ways you could use that pointer.... but to me, that's an optimization problem. If you're malloc()ing in a tight loop and are worried about the null return check being a performance bottleneck, then maybe the malloc() is in the wrong place, and it IS the performance bottleneck. Ie: It's a design problem again.
Maxatar@reddit
OPs argument amounts to saying that you should never write bugs.
It's useless advice that superficially sounds correct, and technically it's true, well designed software doesn't contain bugs of any kind and is perfect... I mean that is technically true... but it's entirely worthless advice.
Big_Combination9890@reddit
In case you meant me by "OP"...no, that is not what I am saying.
I am saying that bugs as serious as a nil ptr deref, should fail fast, hard, loud and visible. Because then they get fixed. And a very good way to achieve that, is letting them crash the service where they occur.
I know that's not a popular opinion in a world where cloud providers have taught people that 5-9s uptime and "failing gracefully" (whatever can possibly be graceful about a nil-deref is anyones guess) are the highest ideals of software development, but it has served me really well over the years.
Engival@reddit
I don't see how you got to that conclusion.
I agree with the above poster: Bugs should fail hard and fast, and get noticed.
Nobody is suggesting "Just don't write bugs".
Mikeavelli@reddit
Typically when I need to aggressively check for null pointers, I'm either in an environment where someone else doesn't know how to build Doors that don't stab my hand, or I'm in an environment where a bad door has a grenade attached that will kill people rather than a paper cut that will annoy them.
Yes, deferencing a null pointer is a design flaw, but we work in a field where design flaws exist and are often outside of our control.
Big_Combination9890@reddit
Precisely.
And so that, when the expected state is not valid, aka. a pointer that shouldn't be nil turns out to be nil, the bug (because that is almost always the result of a bug, or as you say, design flaw) becomes immediately visible, loud and clear, by crashing the service.
Rainbows4Blood@reddit
You could argue that derefing a null pointer that comes from outside and then not crashing could be counted as validation.
I'm not saying that I would do it that way. But it is something that someone could consider doing that way.
Big_Combination9890@reddit
Null pointers don't come from the outside, at least I am not aware of a serialization format that allows me to define a null pointer.
But you know that, so I'm gonna assume here that you are talking about validating ingress data into types where nil ptr can potentially happen, e.g. when a component object is optional and omitted.
Valid point, but, in my opinion, different from what I meant above. Ingress data should always be validated, and if de-serialization can result in nil ptr, then of course I have to test for that before using the type. In my post, I am not talking about deserialized input data, but pointers in the logic flow of the application.
Of course one should not let input data crash the service, that would basically be a built-in attack surface :D
pixelatedCorgi@reddit
I am not a formally trained programmer / software engineer but rather a tech artist that taught myself C++ a long time ago while for game engine programming.
What you said about “letting your services crash” rather than checking permission or attempting to recover is always what I have thought made the most sense, but I’ve yet to actually work with anyone who feels the same.
In my head if something I wrote is trying to deref a null pointer, that inherently means something went very wrong somewhere. I don’t want a warning to be thrown in the log that will just be ignored or not even noticed in the first place, I don’t want the program continuing to run in some unknown state, I just want it to crash immediately so I can fix the issue. Crashing in my mind is the perfect way to let me know “hey shit is broken somewhere and you need to fix it now, not later.”
Sentreen@reddit
You would like Erlang/Elixir. The whole execution model is based on isolating running code in different processes that can crash without affecting any other process running in the system. They then add supervisors on top of it, that can automatically restart stuff that started, or crash themselves if enough crashes happen in a short period of time.
This is a fairly good talk that covers the essentials if you are interested.
nerd5code@reddit
Fail-fast is great if you have the infrastructure for it—e.g., Erlang or other BEAM languages—but it’s nontrivial to get that right, and without it, things can get dicey when there’s more than one thread involved.
Big_Combination9890@reddit
This isn't just in your mind, this is actually a paradigm in software engineering, known as Fail Fast.
And it has saved me alot of pain over the years.
Pesthuf@reddit
This is what I used to really hate about PHP (the language has gotten much better about this in recent years): So many obviously catastrphic failures in built in functions that should have thrown an exception or raised an error just triggered a warning and, if you were lucky, made your function return something innocent like "false". Sometimes in functions where you wouldn't think it even has a return value.
This, paired with many PHP developers being allerging to error checking, made programs just chug along, operating on garbage data, write nonsense and mess everything up.
Thankfully, the developers of the language have seen the light and many warnings have been turned into exceptions.
Extra_Status13@reddit
While I see your point and agree with it, I feel like the divide by zero is a very bad example.
When crunching tons of floating point, it is often better to first do the whole computation and then check at the end for NaN rather than killing your pipeline.
After all, that is precisely the point of having NaN and it's weird propagation rules: so you can check it later.
Indeed the quote in this case holds very well: do you want to check everything and just avoid a proper simd pipeline? Go ahead, check every 0 before any div, but it will go slow. (Asking for permission, slow as only one number checked per instruction).
Want to go fast? Let the hardware do the check and propagate the error, check only the result. (Asking for forgiveness: indeed an optimization).
john16384@reddit
Floating point operations don't mind dividing by zero because they have a way to represent that result (NaN). Integer operations don't, so they must alert you in a different way.
nerd5code@reddit
Until C23 or C++…17 I wanna say, integers can use ones’ complement or sign-magnitude representation, in which case negative 0 can be used as a NaN/NaT representation. All hypothetically, of course; but there are also ISAs like IA64 that have NaT tag bits on general regs, and it’s possible to have hidden tag bits on memory (AS/400 did this for pointers), either of which might be used for integer div-zero, min-div-neg1, or log-\<1 results.
Big_Combination9890@reddit
You are not really "asking forgiveness" in your example. You simply do the ingress validation somewhere else.
As has been pointed out in another answer to my post: Validating ingress data (like the input to a computation pipeline) is a different matter. I absolutely do see your point in letting the computation run and check the result for hardware errors...there is no point in letting bad data crash the service, that would essentially be a built-in DDoS :D
nerd5code@reddit
qNaN specifically. sNaN is still/often a thing.
kabrandon@reddit
I work with externally owned APIs often, where the data coming in gets marshaled into a struct with many embedded pointers for different types of data the API may return. In my case, I often need to check for null pointers and I might as well not write code at all if I would have it panic on the first null pointer access.
axonxorz@reddit
Marshal missing data into a canary object. It's better design. Then you can find out if you always have the canary object, or if you sometimes get a null, a bug.
Engine_Living@reddit
This is the basis for the facade pattern / law of demeter. Expose the external data structure through a facade which contains any null checks, instead of forcing code all over place to randomly dereference a obscure data structure.
AbstractButtonGroup@reddit
Then it is a different case, where null is a valid maker for optional data. But in cases where null is not expected to happen it is indeed best to fail early rather then try to carry on in invalid state.
QuantumFTL@reddit
If an API you are using is giving you invalid data, why continue running the program? Whatever you're using outside of the process to ensure robustness should be able to handle that.
Synyster328@reddit
This is a refreshing take to read, I think people's obsession with "failing gracefully" has led to the common practice of "Make it do anything else, just don't let it crash", without enough thought being put into designing that "anything else". So as you said, you have some half-baked fallback path that lingers like a tumor when the real problem happens before ever evaluating the null ref i.e., why did we even get to this state in the first place.
imachug@reddit
Not as a general rule, no. For all intents and purposes, if you have a possibly null pointer, comparing it to
null
before doing anything is almost certainly a good idea, I don't think we're disagreeing here.I think the most common reason where you don't want to do that is when there's an abstraction boundary, where you have two languages (host and interpreted) and you want to translate null pointer dereference in the interpreter to an interpreter error instead of crashing the host. For example, Java converts it to
NullPointerException
, Go throws a panic, etc. For some languages, it's a way of ensuring memory safety, and for others, it's a debugging mechanism.Either way, if the performance cost is so large that you can't insert null checks at every dereference, and if you control the generated machine code and can make sure you never assume dereferenced pointers are non-null, then please feel free to dereference them and set a SIGSEGV handler. That's the approach both Go and HotSpot take, AFAIK.
trelbutate@reddit
I don't understand what you're saying here.
Which is exactly why you check if it's null beforehand. To exclude the code path where you deref a null pointer.
cherrycode420@reddit
I think he's trying to say that the issue is the Null-Pointer itself rather than the Dereference, hinting towards a bigger issue that's easier to find if you just let a Crash happen..
trelbutate@reddit
I don't think there's anything wrong with null pointers in general. There are many situations where an object does not (yet) exist and it's still a valid state.
That's why I like nullable types that clearly indicate whether something is allowed to be null or not (for example, in C#).
Raknarg@reddit
how are they wrong? It is an optimization, they're not suggesting it as a good practice.
pelrun@reddit
There was a reason Novell Netware apps back in the day were extremely robust and performant. They weren't apps, because there was no userspace. Everything was literally modules loaded directly into the Netware kernel and would crash the entire server if anything was wrong. So not only did developers have no choice but to fix all the crashing bugs, it was extremely obvious when something was wrong.
Ossigen@reddit
The guy even put a giant purple banner at the top but apparently it wasn’t enough
Big_Combination9890@reddit
Yes, and my post in no way goes counter that; I have not stated that the article presents this as a general rule. I have given my opinion about this section.
Ossigen@reddit
Yes but, respectfully, I don’t think your argument is strong enough.
Big_Combination9890@reddit
Then you should have started with that instead.
Yes, I know that's what the article presents. I read it.
I simply disagree that it is an optimization.
I build B2B systems for large corporations for a living, so I have no idea how you arrived at that conclusion. Yes, crashes can happen. And in some cases, crashes should happen. nil-deref often are such cases.
Because I have debugged software (things you'd probably call "big systems") that did "recover" from such obvious bugs. Usually, I was brought in, when the "recovering" started to cause a big fukkin problem, that was no longer recoverable. Stuff like the prod server that was running for 4 years at that point, suddenly starting to get OOM-Killed by the kernel every few minutes, halting the entire business. And guess what, such problems are ALOT harder to fix than having to quickly patch a faulty method in a data ingress pipeline at 3AM in the morning.
I don't, and I explained my reasoning, so maybe you should read my post again before making assumptions about the strength of my arguments.
Here is how it works:
I deref the pointer. If it is nil, a panic occurs. I deliberately don't recover from the panic, and accept that the service crashed.
This concept is called failing fast and it's probably one of the most valueable time savers in the development of exactly those "real big systems" you assumed I "never had to work with".
Ossigen@reddit
Haha, I see now, sorry I misunderstood your comment.
flundstrom2@reddit
UB means UNDEFINED behavior, and in C and C++, it means the compiler is free to crash an airplane in your head, despite your code only controls your bedroom lamps.
In fact, it is even free to execute a code-path that doesnt trigger the NPE, including backwards:
might print
when run on a weekdays, and
on Saturdays. EVEN if you are running on a modern CPU with null-pointer detection, signal handlers and what not, since the compiler may inject code which disables the null-pointer detection.
UB is worse than the devil, because the devil is always evil.
robhanz@reddit
At least in C++, UB is not recoverable. Sorry. Sometimes it may seem to be, but that's entirely too blase of an attitude.
Why? Compilers prune dead code. And they're allowed to. And since UB can't happen, they're allowed to presume that code that results in UB can't happen.
In the third line, we dereference p no matter what. This lets the compiler say "well, i'm being told to dereference p. That's undefined behavior is p is null, therefore p must not be null.
Which means that entire condition can be completely erased.
Now imagine it wasn't just an assignment, but some kind of critical function that needed to be called? Now it's not called. Ever. Whether or not p is valid.
This has caused real, massive bugs. It is not safe to just say "UB isn't unrecoverable". It must be avoided.
If by "recoverable" you mean the program might not always crash? Sure. But crashing is, in many cases, the least bad thing that can happen in case of an error.
Guvante@reddit
Most of these are "weird platforms exist" here are some better ones.
valarauca14@reddit
You seem to be confusing the functionality of the limit register (e.g.: any address less than or equal it is a memory error) & offset register (CR2).
The limit register controls if an memory segment error occurs. If a value is less than or equal to the limit register, that value (the bad value) is added to CR2 before the CPU before being handed off to the correct interrupt handler.
What I'm trying to say is the limit register is the first global descriptor table entry. Which is always zeroed on the only modes people use (32bit flat mode & 64bit long mode).
Null is not
0x16
it is0x0
, the interrupt handler sees0x16
because CR2 added0x16
to0x0
.FYI we've had 5 level page tables in the kernel since 4.14 (2016). Now 56bits are usable on a lot of server class CPUs.
Guvante@reddit
I didn't say null was 0x16 I said the actual failure happened due to reading 0x16 not reading 0x0. And that you won't see 0x0 in CR2 for that reason.
I didn't realize they had added bits but good to know.
nekokattt@reddit
No, but you should be treating it as such.
You don't need to worry about UB if you avoid UB.
Supuhstar@reddit
Choose programming languages, which make this not a problem. Like Swift or Rust
imachug@reddit
Ehh, I don't know about that. I can see two interpretations of your claim:
0
and abandons odd platforms, which affects some of the claims. (Not sure what Swift does here.)To the former I respond that sum types are great, but if you have to touch
unsafe
code, then you have to think about such specifics quite often, so it's not not a problem -- it's just a rarely important problem. Maybe a subtle difference, but I very much have to consider such specifics. (But then again, not everyone writes low-level code in Rust, and that's fine.)To the latter, well, IIRC that was a deliberate choice to define and think real hard about all the stuff C leaves implementation-defined, much like provenance, so overall I think it was a good idea. Can't say much else.
steveklabnik1@reddit
Rust and null is in a bit of a weird place. In order:
Dereferencing a pointer produces a place expression, and it is UB to:
What is dangling?
So, nothing about null specifically or its address. The reference does refer to "null pointers" and such, and so it's fairly under-specified.
However, it is true that the core library has
core::ptr::null()
: https://doc.rust-lang.org/stable/core/ptr/fn.null.htmlWhich documents:
So, in that sense, it's vaguely similar to the way it's handled in C; it's often literally zero, but doesn't actually have to be, and if zero is a valid address, it's more that it's legal in Rust but
core::ptr::null
won't return the correct null pointer.However, the Ferrocene Language Specification, which is used for the safety certification of Rust, and is going to be merged into the reference in the future, defines things more explicitly:
With
null
linking to:So I suspect it'll probably end up like that in the end.
I'm not an expert on platforms in which 0 is a valid address, but all of this doesn't inherently mean Rust is unusable on them. For example, on ARM, address zero is the reset vector, but you can access it just fine with inline assembly, you'd never use an explicit pointer to that address for this kind of task anyway.
imachug@reddit
I think having
core::ptr::null
not return a null pointer andcore::ptr::is_null
not check that a pointer is null is a non-starter, personally. The reference doesn't define it unambiguously, but then again, the reference doesn't specify a lot of stuff. I think it's safe to say that0
will remain null.Yeah. I'm more concerned about platforms that define e.g.
-1
as the null pointer. These two properties are related, but not equivalent. The value of a null pointer is fundamentally an ABI thing, so really the only thing to worry about here is FFI, which is probably better handled in userspace than the language itself.steveklabnik1@reddit
I'd agree with all of this, yeah.
Supuhstar@reddit
Point is it's handled at compile time so you don't have to worry about the runtime concerns that this article is concerned with.
Also, Swift uses a similar approach, but doesn't concern itself with what's at the memory at that address until/unless it's passed to code outside Swift (e.g. linked C libraries). In Swift land, there is no "null"; the
nil
keyword is just a keyword that defaults to meaningOptional<MyType>.none
. As long as you don't force-unwrap it with!
, it'll never be a runtime issue, and using!
to force-unwrap that causes a specialized fatal error with the message "Unexpectedly found nil while unwrapping am Optional value". Not exactly a null pointer exception, more of a bespoke system for handling cases where the isn't a value.imachug@reddit
I have no idea what this means. Are you still talking about algebraic types? This post does not discuss anything relevant to ADTs, it discusses machine behavior, the behavior of optimizers and compiler backends like LLVM, and the C standard. Rules enforced (or not enforced) by the first two sitll apply to Rust and Swift. Rust programmers do have to care about nulls when dereferencing unsafe pointers.
Supuhstar@reddit
I don’t like mixing software engineering with hardware engineering.
If you’re writing software, you choose a language to write it in. These days, I struggle greatly with recommending any language which doesn’t guard against these things that compile time.
That's all I’m saying
imachug@reddit
Hardware engineering has nothing to do with this. Hardware engineering is designing microchips. I'm talking about writing software that targets the (already existing) hardware. The distinction you're looking for is low-level vs high-level code, and that I can't argue with: high-level Rust code doesn't have to deal with null pointers. But the post is about low-level stuff, which neither Rust nor Swift can help you with.
nerd5code@reddit
Some notes:
x86 ffunn
It was actually more complicated than “zero is null is the IVT,” because three pointer types were possible to objects or functions (
__near
,__far
, or__huge
), and these would default differently depending on your memory model, and from the ’286 on, the number of architectural-nulls actually depended on the setting of the A20EN line and IDTR and the CPU mode.__far
pointers are what you’re probably thinking of—all-zeroes gave you address zero, which is where the IVT started (’286+: by default). But pre-’286 or with A20EN disabled, you could also hit that address with FFFF:0010, FFFE:0020, FFFD:0030, etc. because the segment (←) and offset (→) were combined as 16×seg+off, and C wouldn’t generally see the high addresses as null even though they aliased. With A20EN enabled (’286+), FFFF:0010 and up were de-aliased, which let you (or rather, DOS) use the 65520 B of RAM that started at the 1-MiB mark, called the High Memory Area (HMA).__huge
pointers were a normalized form of__far
, which generally kept bits 4 through 15 zero-valued, so you only had 16 offsets per segment. All-zeroes was still null in both C and hardware, but you couldn’t reach the HMA. However, if you tweaked the bytes of the pointer directly, you could potentially encode sometime-nulls by keeping segment and bits 0–3 of offset =0, but setting bits in the 4–15 region. It was effectively undefined whether C or underpinnings would see those as null or not—if tested/accessed after re-/normalization, then yes, else no. Similarly, with segment FFFF and nonnormal huge pointers, you might hit HMA or address zero (or not), but C would never see null unless it normalized unexpectedly, and then its idea of null and the hardware’s might differ.For
__near
pointers, you had an implied segment that used whatever was in CS/DS/SS already, and the pointer only represented the offset, limiting you to 64KiB total for code and/or data. Thus, unless you’d frobbed a seg reg, which took a bit of effort or a bug elsewhere, or perhaps running byte 0F on an 808x (which used that for POP CS, not extended opcodes as on ’286+), you wouldn’t generally be in segment 0 contextually, and offset zero would be local to your code and/or data segment.However, there was potentially an important structure there, placed by DOS: Your 512-byte program segment prefix (PSP). That included information about your program and its command line, so frobbing it could have wide-ranging effects. This was especially an issue for the Tiny model used by .COM files and upconverted 8080/8085 code, where code, data, heap, and stack all had to fit into 64 KiB-512 B unless you did up your own far gunk to escape. Address zero must always be CD 20, which codes a DOS exit syscall (INT 20h), because that was how you ended an 8080/8085 program, jmp/call 0. Very few DOS programs actually exited that way, fortunately, and the
exit
function usually used an INT 21h syscall that accepted an exit status.I mentioned IDTR, which was added with the ’286. It was primarily intended for protected mode, but you could relocate the real-mode IVT to an arbitrary address with it—though you wouldn’t, generally, unless you were intercepting interrupts separately.
In protected mode and the i432 ISA its guttiwuts derived from, both GDT and LDT carried an unused null entry, and any offset into that would trigger a fault. Segments were now coded as selectors carrying a table select bit and two RPL bits, so there were 8×65536 possible null pointer codings. But you still had near/far and hypothetically huge pointers, so near nulls were just offset 0 in a valid segment.
Aperture size
The null aperture in flat address spaces (=most, or via x86 near) has a particular size. That means that accidental access to a large non-object (us. an array) via a null pointer can accidentally escape the null aperture and access valid memory! E.g.,
On Linux IA32, you generally have read-only memory starting at the 4-MiB mark IIRC, then after .text and .rodata come writable .data and .bss.
In x86 real mode, the BIOS data area (BDA) followed the IVT, so frobbing things there would break things that used BIOS or DOS (which used BIOS) services. After the BDA, there might be some DOS data, then possibly the ’286 LOADALL area, then more DOS data, so null with a larger offset could be quite dangerous.
Integer-pointer casts
Pretty much any cast between integer and pointer should be viewed as suspect in portable codebases, with or without involving
uintptr_t
—and that requires C≥99 or C++≥11 support (or most C9x or C++0x approximations thereunto), and that type is optional in the first place—e.g., OS/400 ILE in P128 model has none, for example, and it will only round-trip like 24 bits of a pointer via cast to/fromint
,unsigned
,long
, orunsigned long
.You mentioned these casts are ISB, and there’s enough variation in behaviors that it’s a far better idea not to rely on it at all.
The last real necessary uses for it are
implementing
memmove
,implementing an allocator, or
detecting object/pointer alignment absolutely.
Of these, the first two necessarily involve some ABI/OS/ISA assumptions, and the final doesn’t make sense in the pure Abstract Machine models, where objects and functions might be positionless islands unto themselves.
It does make sense to assume some minimal necessary alignment of a base pointer, and then find the alignment of
(char *)relptr - (char *)base
instead (which assumes both pointers are to the same underlying object).C23 does give us an absolute
memalignment
function in both freestanding and hosted impls, so presumably you’d have to know segment base alignments a priori to implement that in 16-bit protected mode, or just limit your maximal considered alignment to 16 bytes (a.k.a. one “paragraph”) or so, which was the allocation granularity of both the DOS and OS/2 kernels.pron98@reddit
It's more than that. People like John Regehr have done a fantastic job educating the public about the horrors UB, but perhaps they've done too good a job because one thing that, I think, is still misunderstood is that UB is always relative to the programming language. From the perspective of the C language spec, undefined behaviour is the end of the line. The C spec cannot assign semantics to a C program with UB. In other words, it can say nothing about what it means. Really, it is not a valid C program. So from the perspective of the C spec, UB is the end of the line; it's the worst thing that can happen.
That is all true, but when we run an executable compiled from a C program, we're not running C code. We're running machine code, and machine code has no undefined behaviour (or, at least, not in the same situations a C program does). Every machine instruction has well-defined semantics, though some may be nondeterministic and the semantics depend on the chosen hardware and OS configuration.
So while the C spec can say absolutely nothing about a C program with a C UB, we can still talk about the behaviour of the machine-code program we actually end up running, and even about the probability that some machine-code behaviour will occur in an executable produced from some C program. It's just that we cannot be assisted by the C spec when doing so.
YakumoYoukai@reddit
After demonstrating how thoroughly ridiculous this thing we call programming is, I cannot tell if this is real or not.
imachug@reddit
It was supposed to be an in-jest joke, but I belatedly realized that some virtual machines do, in fact, encode addresses in floats via NaN boxing.
cdb_11@reddit
Frankly, null pointers should be legal to read from, and only segfault on writes. Then dereferencing a null pointer could act as accessing a zeroed-out object.
Likewise, you could always dereference a null-terminated string pointer, and everything would work out just fine.
This way it'd be possible to write code for the happy path, without doing any branches.
imachug@reddit
The problem with this approach is that, in practice, the pointers you will try to dereference won't be NULL pointers, but rather slightly offset NULL pointers. Suppose that the fields in your
struct String
were reordered: ifs
was null, the fields->data
would be located at address0x8
, and so you'd read from address8
. You could argue that it's fine because we can map the whole page0
, but then you'd have this weird behavior where short structs behave correctly and long structs break down unexpectedly. Not ideal.cdb_11@reddit
That still works. Even today 0x8 still points to a protected null page, and is guaranteed to segfault (x64 linux at least). What I'm saying is to just give that address range a read access.
It's not 100% bullet-proof of course, but that's fine IMO. The exact size of the null page could be a compiler option, or the compiler could pick it automatically based on the widest struct. For dynamically linked programs, the linker could do that, since it's basically its job anyway. But I guess it still could in theory break on
dlopen
, as by that point it may be too late for that.imachug@reddit
What about arrays? Would accessing
array[1]
be allowed, ifarray
isNULL
? That seems like a major issue.I'd be wary of specifying a behavior that cannot be 100% relied upon. If it's just a best-effort attempt and you can still create out-of-bounds "NULL" pointers, every function will have to check for
NULL
anyway, and at that point it's not any better than status quo.In fact, it's arguably worse than status quo, because currently you have a chance to notice that the
if (p == NULL)
check is missing if the program crashes; but if it doesn't and silently goes on, it's easier to miss such checks.cdb_11@reddit
You could say only
array[0]
is legal. But I'm not really arguing for language specifications to make any portable guarantees, but rather for platforms to enable this style of programming. I think it sucks that this style didn't caught on, and now you have to jump through extra hoops to do this, to the point where it's probably not really worth doing it.JiminP@reddit
Colloquially, actually dereferencing a null pointer does "crash the program". Sure, likely there will be a signal handler which will leave a crash dump (C++) or the panic would be recovered for the thread to continue (Go), but as far as business logic is considered, "the routine" will have ended, which is usually what matters.
At least on modern C++, the proper way of handling null pointers is:
nullptr
for a null pointer. Don't useNULL
.Side note: as the blog post has been noted, null pointers point to zero on Rust. This is due to null pointer optimization; for example it is guaranteed that
Option<Box<T>>
andBox<T>
has the same size, andNone
has all zero-bits as its memory representation for this case.mackthehobbit@reddit
The article is a weird take for sure. Either your function allows null pointers in its contract or it doesn’t. If it doesn’t, sure, allow the dereference, UB and probably panic.
Perhaps the only notion worse than exceptions-as-control-flow is segfaults-as-control-flow…
NewPhoneNewSubs@reddit
May I introduce you to drum rotations as flow control?
mccoyn@reddit
Cached link since that website seems to be having problems today.
https://web.archive.org/web/20250724213610/https://users.cs.utah.edu/~elb/folklore/mel.html
AlienRobotMk2@reddit
Finally some real programming.
campbellm@reddit
As are most "Falsehoods programmer's believe about ..." articles, unfortunately.
imachug@reddit
(OOP here) The article is not a guide on abusing segfaults for writing reliable software, that's for sure. It's goal is the opposite -- it's to demonstrate that things you might have been taught or thought were obvious aren't, in fact, portable. And this includes false claims like "dereferencing address 0 reliably causes segfault", which doesn't make much sense in modern C, obviously, but does in machine code or other low-level languages, like very old C. Of course, I'm not advising anyone to dereference null pointers in modern C, or anything like that :)
mackthehobbit@reddit
Rereading the full article makes more sense, and I think a lot of the criticism you're getting is because some of your notes are easily misread. The second paragraph under heading 2 is contributing here:
My first read was "both cases" referring to headings 1 and 2, where heading 1 is talking about segfaults in C, C++, rust and how they can be recovered, while heading 2 is talking mostly about higher level languages. That sounds like a recommendation to segfault on purpose and ask for forgiveness. It's now a bit clearer that "both cases" actually means Go and Java.
In an article primarily discussing C and C++ standards, and various assumptions you shouldn't make about null pointers and what happens if you dereference them, this obviously felt contradictory.
A more careful read, with a bit of critical thinking, reveals a lot. On the other hand: if I assume that I already know what every writer means to say better than they do... how would I ever learn something new?
imachug@reddit
Yeah, makes sense. I guess I really messed up the delivery here, didn't I? Sucks that I didn't catch it before posting and that I didn't realize that was the problem when it was posted the first time. I'll see if I can fix it.
Ameisen@reddit
I've done some interesting things with using signals/vectored exceptions and on-demand data processing via the access of reserved memory ranges...
nerd5code@reddit
It can certainly work if you have tight control over codegen, but if an optimization reorders your code unexpectedly under the assumption UB doesn’t happen, things can break in insane ways.
E.g., people do shit like
rather than just
if
ing properly, under the assumption thememset
will cause a crash. (Which is the incorrect thing to do in the first place—should be a controlled abort at absolute most, but null return frommalloc
is an expected, well-defined outcome, so really the program should terminate cleanly viaexit
orreturn
frommain
.)But
memset
might be inlined or deferred untilp
is actually needed, as might themalloc
, since both of those can be compiler builtins or LTO-inlined, so any intervening code might run first, possibly triggering extrinsic side effects withoutp
in hand. Worse,malloc
thenmemset
-to-0 might just be merged intocalloc
, in which case you’re back to an entirely uncheckedp
.And if it’s at all likely that
p
might actually be null, faulting is just about the slowest, least-predictable way to go; your CPU might be out tens to hundreds of instructions ahead of hard reality, which means all that work needs to be thrown out, the ISA-visible state (probably not caches etc., which can cause security issues) needs to be rewound to just before the fault, and then the CPU needs to vector into the kernel/supervisor, which needs to scan VMAs to determine that no, the fault address should not be deduplicated or swapped in, and then all the signal/exception-raising goop needs to be engaged to flip the CPU back into user mode at the right address with the right stack contents (writes to which from kernel mode must be protected or guarded somehow), and often after the handler (which is often extremely constrained in its actions), a syscall (→kernel mode again) has to be issued to end handling and return again to userspace. And then, ISAs like the x86 will promise a TLB flush on page faults, which is totally unnecessary for this kind of thing.Compared to a predictable conditional branch like you’d get for
if(!p)
, whose only cost on successful prediction is in terms of BTB and brpred history cache space, a fault is incomparably slow.pakoito@reddit
How is UB better than a crash?
Full-Spectral@reddit
It would never be. A crash is the 'happy' path. UB is the "Hey, did someone hit the missile launch button by accident or something?" path.
flatfinger@reddit
In the language the C Standard was chartered to describe, many statements of the form "On platforms with trait X, Y will do Z" were true, and the authors of the Standard allowed implementations targeting platforms with trait X to, as a form of what they called "conforming language extension", treat Y though it were defined as doing Z, without regard to whether the behavior was "officially" defined.
Nothing in the published Rationale suggests any intention to deprecate such practices or reliance upon them, since the authors of the Standard never saw any need to create any viable alternative. If one wants to allocate an array of 1000 initially-null void pointers, there isn't even a portable means of doing something like:
When the Standard was written, it was clear that code which relied upon reads of address 0 yielding a value of 0 was non-portable, and there was no intention to change that. On the other hand, the fact that the Standard says that UB occurs as a result of non-portable or erroneous program constructs, rather than merely erroneous constructs, was intended to leave open the possibility that such constructs may be correct in some execution enviroronments while being erroneous in others, with the programmer being responsible for knowing whether they would be correct in the particular environment where the program would be run.
Supuhstar@reddit
If you find yourself trying to recover from a null pointer exception... you really need to take a good hard look in the mirror and question your life decisions.
ivancea@reddit
This is ridiculous. No senior "believes" half of those made-up "falsehoods", and most of them were taken out of context.
"Dereferencing an null in Java will end up in an exception, and you can catch it!"
No shit Sherlock, that's not the point. You catch it, you're, in general, a terrible programmer
imachug@reddit
You know, maybe I should just stop writing.
That's exactly correct, the point is not that the userland Java code can catch the NPE, it's that the JVM converts a machine-level NPE to an exception and can continue execution without crashing the process or making it unreliable to continue. Bad wording, I guess?
I don't understand. You might say that these falsehoods were taken out of context precisely because they typically hold, but there are exceptions; well, here are many (won't say "all the", of course) exceptions in one place.
Maybe it's just wording, but I don't see how that would be the case.
(1-4) Do all seniors know that the address
0
can be mapped withmmap
-- occasionally, only on some machines -- which can cause null dereference in machine code not to crash the process? Or that, in freestanding environments, there is often physical memory at address0
?(5) Are all seniors familiar with software philosophy back from 1990?
(8) I can agree that this is well-known.
(6-7, 9-12) I don't see how this can be well-known.
ivancea@reddit
The internal implementation is for the VM to decide, and doesn't have to be executed at machine-level. So there's probably no "conversion". That's the fail of the article: Thinking that all the languages work at "machine level", and that everybody thinks that. Languages are allowed to not delegate nearly anything to the machine if they don't want to do so.
Continuing with my first paragraph: A "null pointer" doesn't have an address by default. That's a purely language-dependant decision. Not every language is C. Thinking that a null is a 0 is not something a senior "does". There's not even a sense of address in "nulls", unless you're talking specifically about languages that have such meaning for them.
9, 10 and 11 are purely C-related, so not very interesting. The answer to that is in the standard, not much to guess here. Similar for 12. They all say "On platforms where the null pointer has address 0", which is already a quite vague preset. "Null pointers having addresses, and such addresses being 0, but not really 0". Now we're mixing different layers. Not just the application and language layers, but also the hardware layer. I would add a (13) talking about how a Java null pointer doesn't always have to be made of copper atoms. Just in case!
And returning to the first points:
It's already answered, but as neither a null pointer has an inherent address, and segmentation faults has little to do with a language supporting nulls, I don't see why would anybody think that. And as with the other cases, 2-4 are mostly copies of (1) with slightly different definitions.
So well, in general, the mix of layers in those definitions are what makes them wonky IMO. They feel like "AHA! You thought that huh? But did you know that ACHTUALLY the copper atom may have an extra electron? Gotcha!"
imachug@reddit
I think the problem is that, for whatever reason, you ignored the disclaimer saying "[t]his article assumes you know what UB is and [...] very basic knowledge of how CPUs work" and decided that this post can be meaningful when read from the PoV of a high-level language rather than languages which, you know, actually define what UB is and are somehow related to the CPU.
So C, Rust, and stuff like that, as well as low-level machine code; HotSpot and the Go runtime were just examples of programs written in those languages, not separate languages this can apply to.
The post is even titled "null pointers", not "nulls", so I don't understand how you could possibly imagine it meaning to cover languages that don't even have the concept of a pointer... tunnel vision, I guess?
ivancea@reddit
What? It says it assumes you know what an UB is and how CPUs with. That has nothing to do with "C". They have nothing to do with low level languages. The article even mentions Java NPEs, so either the article is wrong and inconsistent, or no, it's not just for "low level languages".
Don't blame the readers for a badly explained article then? We can't read the writer mind and guess what "their intent" was.
Yet it talks about Java NPEs. Makes all the sense! /s
In summary, if you want to talk about a very specific set of languages, enumerate them and say "this only applies about junior devs that only know about these languages and can't even think about language design at any other level". Because when you mix languages, you're talking about language design. If you think you can talk about nulls of two different languages without taking about language design, that's your falsehood #13
imachug@reddit
I've explained this elsewhere in the thread and in the parent comment as well, but I'll repeat myself: Java is an example of how the JVM itself can catch null pointer dereferences in the JIT code and translate them to NPEs, without crashing the JVM process. It's not an example of how the userland code itself can handle NPEs.
You are missing the point. Languages have idioms, and knowing more languages does not automatically make you a better programmer within those languages. You are not supposed to think about UB when you write assembly -- moreover, that's pretty harmful. You're supposed to think about performance when you write low-level stuff in the kernel despite telling people to optimize for readability when they write high-level Python code.
All too many concepts simply don't meaningfully translate between languages, and that's exactly what happens here: you are expected to treat
NULL
pointers as non-magic when you write C because the language itself forces you to, even if your experience tells younull
should be an unknown sentinel; couple that with language idioms, and you might usememset
to zero a structure because you "know"NULL
is 0 and watch everything break on rare platforms.ivancea@reddit
It's more like "when you learn some language, you don't carry over what your know from others. It may or may not work".
But you surely think about performance when you write high level python, or Java, or whatever, if that's what your solution is related with. In a similar fashion, you don't think about performance in low level languages of your solution doesn't need it.
Anyway, yeah. If the article doesn't expect it to affect to every language, then it should state so, period. Because nulls aren't a C unique trait.
Btw, I don't know who are "you". You're not op, so don't expect people to guess that you're the author of the post or anything like that. Not even the post says who's the author, and I will surely not navigate all its links to find out
imachug@reddit
Yeah, it's an odd legacy to have :/ I tried to remedy this by setting my Reddit display name to "purplesyringa", but I guess people aren't used to reading bios. I registered u/purplesyringa when I posted my first article after years of commenting on Reddit and promptly got banned because algorithms decided I must be spamming, and my attempts to register other accounts got shadowbanned even without any activity. Not sure if I can do anything about this.
EbrithilUmaroth@reddit
Now I just need to figure out what "dereferencing a null pointer" means
QuaternionsRoll@reddit
While this code example is correct, the statements preceding it are not. The address of an object must be distinguishable from
NULL
according to Section 6.3.3.3 Pointers, Paragraph 3 of the C standard:Null pointers can only be "guaranteed to compare unequal to a pointer to any object" if the compiler can ensure that the object placed at address
0
is will never be compared to a null pointer at runtime, at which point the fact that the object's address has the same bitwise representation as a null pointer becomes (nearly) unobservable, and statements about the object having the "same address" as the null pointer become meaningless.imachug@reddit
That's a good addition. My intent was to say that the compiler is allowed to use the same bitwise representation for
p
andq
as long as it optimizes all comparisons likep == q
tofalse
. The comparisons are still allowed, they just have to be lowered in a non-trivial fashion. But you can still theoretically observe that the bitwise representations match by usingmemcmp(&p, &q, sizeof(int*))
.Sidenote: Why not
(uintptr_t)p == (uintptr_t)q
? C defines ptr2int casts as exposing, and the problem with exposed casts is that they're basically impossible to formalize without asserting address uniqueness (yet another good lesson from Rust). So C does the obvious thing and refuses to formalize the semantics, so I can't even claim whether assigning equal addresses would be sound lowering. Compilers don't do this these days because tracking data dependencies is hard, but I don't think the standard explicitly forbids this, unless I'm mistaken.QuaternionsRoll@reddit
/u/imachug, I found your original post, but I figured it would be better to tag you here than reply to a post from 8 months ago lol
This article is pretty good FWIW
One_Economist_3761@reddit
The link didn’t work for me. ;p
feketegy@reddit
There are a lot of falsehoods programmers believe
mr_birkenblatt@reddit
And you posted it to Reddit... smh
curien@reddit
The comparison (
p == &y
) is already UB before you even get to the dereference. You're only allowed to compare pointers that point within (or one past the end of) the same object.imachug@reddit
Comparing pointers for
<
or>
is only allowed within the same object, yes. Comparing pointers for==
is allowed for any object. (But the result may betrue
even though the objects are different if the addresses align by chance).curien@reddit
Ugh, you're right, I misremembered that.
baordog@reddit
Assuming hubristically that we can write an API that excludes the possibility of null pointers entirely is exactly how we got to the practice of paranoid null pointer checks.
Realistically most programmers cannot anticipate all of the cases where the pointer might be null. If your service takes data from remote sources or the kernel you can’t actually guarantee the pointer isn’t null.
XNormal@reddit
In the old DOS days, the interrupt vector table resided in address 0.
I once wrote a Turbo Pascal library that installed a virtual method table pointer at address 0 that trapped any virtual method call using a null pointer and converted it to a runtime error at the call address.
It also didn't disturb the usual function of interrupt 0 (division by zero). I think it only worked if the object did not inherit from a non-virtual base class, but all the major libraries had a common virtual root class.
ShinyHappyREM@reddit
Good ol'
archiminos@reddit
I'm assuming UB is Undefined Behaviour? Is this a common abbreviation? I've never seen it before.
nerd5code@reddit
It is in the C and C++ end of the pool.
alphaglosined@reddit
Normal code paths shouldn't be catching a null dereference.
You can't know what code you called caused the deref. If you did know, you would have done a null check.
To continue on is egotistical at best. Something must die. There must be a sacrifice for the process to continue.
Usually a coroutine.
Not doing this allows logic-level errors to enter the program, putting it into an unknown state.
Also, there is a big difference between a read barrier seeing the null and throwing an exception, and a null deref actually occurring.
Unfortunately, signal handling on null dereference and then attempting to throw an exception from within a signal handler is a known "fun time" generator and is very platform-specific. If this occurs, I suggest considering the entire process dead and preferring null deref read barriers to protect you instead.
Finally, all this runtime protection is the backup; it should never be considered your primary protection against null. Static analysis should always come first to prevent you from doing stupid things. However, due to people not valuing it, it may only be able to catch the really stupid stuff by default and not give as strong a guarantee as DFA can or a type system can offer.
I'm not just stating this for funzies; I have been working on a DFA that will hopefully have the ability to be turned on by default in D, and one of its analyses is to prevent really stupid null dereferences. So far, it's only found one such example in our community projects that are in CI. My takeaway from that is if code survives for a period of time, and it's been looked at by senior developers, it probably is free from such patterns, but it's still better to have the analysis than not.
Jemm971@reddit
The pointers….😂😂😂😂
kreiger@reddit
Yeah no, this is nonsense. A NullPointerException is not caused by a a literal null pointer. It's a misnomer for a Null Reference Exception.
UnDosTresPescao@reddit
Yeah, no. I recently had to track down an issue where the Linux Kernel went from not using one of the arguments in a function call to writing to a field about a hundred bytes into the structure without checking the pointer. We were passing in a null pointer. After rebuilding our driver for a new version of Linux, sometimes it would work, sometimes it would reboot the PC. Pure joy.