I just had a little ghost in the shell moment...

btw, theoretically speaking, I can't see how classic softmax attention could not be able to guess the lenght of text. I mean, Imo it is not something LLMs are able to do, but probably if you train a transformer using RL with the sole purpose of guessing the lenght of its context, it could manage reach an approximation. (assuming it use full classic softmax attention, so not sliding window, DSA, CSA... idk about lightning attention or recurrent formulations of linear attention). Also, modern positional encoding is purely relative, still from each token's perspective there is a continuous concepts of distance toward other tokens, embedded via Rope angle shift, and that would help. ie, a model hidden state could identify the tokens for which is valid the conditions "each other tokens vector is rotated only in a direction compared to this one" identifying first and last token of the context even without taking into account causal masking, and "estimate" the total rotation from the first to last token (or count the numbers of rotations, depending on the rope coefficient used for the model compared to the max context lenght, if this end up being periodic) I'm not saying those LLMs we use are able do that, just that it is not impossible, architecturally speaking.