All of these are like 100x more severe than I'd expect, that's insane.
Instead of the back-end seeing 1.5 requests as usual, it saw exactly 2. I received the first response, but the next user received the response to my smuggled request. The response they should've received was then sent to the next user, and so on. In effect, the front-end started serving each user the response to the previous user's request, indefinitely.
To make matters worse, some of these contained Set-Cookie headers that persistently logged users into other users' accounts. After deploying a hotfix, Atlassian opted to globally expire all user sessions.
This too is nuts, exfiltrating credentials and essentially hijacking entire HTTP requests is scary, but essentially locking the server into a state where it mis-serves user requests and starts authenticating end users under random different users, that's nuclear havoc lol
I've seen it before. The idiots where I worked were shoving user permissions in a static variable instead of passing it as a parameter. The bug was in production for weeks before I figured it out.
The scary part for me was that I was a junior developer at the time and decided to send a mass email to the whole dev team explaining how basic static variables worked and why they shouldn't be using them in a web server. The very same day the idiot who made this mistake was promoted to team lead. So if anyone bothered the check the change history on that file it would have looked like I was attacking my 'better'.
Somebody switched our HTTP request handler from sync to async without realizing threads/threadlocal variables are now shared between simultaneous requests... luckily it's a greenfield project so no customers were affected. Yikes though.
I don’t doubt you’re right but I’d be interested in understanding more about the problem you describe. In this case does “static variable” mean what is also sometimes called a “class variable”? And if so, was the problem that a given users permissions were being given to another user? And if so, how had any notion of concurrent sessions worked?
This was back when .NET was still relatively new and most of the world still ran on PHP or Classic ASP. We didn't have stuff like ConcurrentDictionary, just raw mutexes if you didn't want to break your collections.
I went through the code base and purged every static variable. Thankfully there weren't many.
The flatness of reward fo programmers is remarkable to me. The fact that time-served seems to be the major determinant of position and salary (when attrition in the field is low) is bonkers.
Lots of people will agree that a small number of programmers produce an outsized positive impact, but reward structures seem to rarely reflect this.
It’s partly just the hard problem of assessing contribution. But it also feels like a cultural problem: with programming having become a destination for making money rather than craft the peer-evaluation structure feels like it’s emphasized blind rewards rather than critical consideration. — That’s just feel mind you. As mentioned, actually judging contributions is hard. So I could be wrong.
Lots of people will agree that a small number of programmers produce an outsized positive impact
Perhaps, though I'd say oftentimes, it's a well-oiled team as a whole that makes the difference. And for a team lead to discover that someone in a team is the mythical 10x solo programmer is tricky. The incentive structures are also hairy.
There's part, but there's the whole organization to account for.
Time served, while usually not exciting, it's a stable variable that has some value (someone that spends enough time in an organization should know how to run it better), but it creates a roadblock to nonstandard career paths. And credit is a double edged sword because politics. Toxic bosses will steal credit from their subordinates.
There should be a career path for people who don't want to be leaders, and honesty should be properly valued, not just give it lip service.
Lots of people will agree that a small number of programmers produce an outsized positive impact, but reward structures seem to rarely reflect this.
And conversely, a programmer in an impactful position who's in over their head can create an outsized negative impact. Worse, it can be very hard to recognise such programmers until the damage is already done.
IMO programming politics within an org is a combination of 1) navel-gazing ego, and 2) job stability.
If you get people with not just an ego ("I'm pretty good") but a comparative one in which they put others down ("I'm way better than these idiots"), you get a lot of political bullshit. If people think stepping on someone will lead to their job being safe(r) or a promotion, you get a lot of political bullshit. On the flip side, if people are neither too worried about justifying their jobs on a daily basis nor think everyone else is a goddamn idiot, an org has a pretty high chance of having only the "people care about their work" sort of politics rather than "people have knives out" sort of politics.
I've certainly implemented things in a way I knew at the time to be suboptimal because there were constraints that wouldn't be knowable to a new hire. If later clean up proves impossible, I'll happily explain the constraints to anyone willing to listen so the ideal implementation can hopefully be made real.
If someone, without learning the surrounding context and constraints, called me an idiot for that initial implementation, I take no offense. Only the caller is diminished.
"Static variable" is a very overridden term with quite a few definitions. I really have to stretch to think of a definition where storing auth information is okay. I think it's almost certainly egregious. Still, asking for the explanation exposes the problem just as well as blasting, yet elevates the asker.
Layer 7 load balancers always annoyed me, in part because of these protocol complexities.
Something I'd like to see is web frameworks and cloud platforms with better support for direct networking between clients and servers. In an era where ASP.NET can talk HTTP/3 natively, there are significant downsides to proxying through some slightly incompatible middle-box.
One approach would be to use stateless (random) load balancing for the 'www' domain, and then use random/unpredictable domains with wildcard DNS pointing at individual servers for sticky sessions. The DNS server could use something like consistent hashing to direct traffic at individual VMs.
Good Layer 7 load balancers can be cheaper at serving static files and fighting off DDoS attacks. You don't want to have every client specialize in DDoS protection.
if you let an idiot design your web server and they don't validate the request headers then you could get unexpected results that could lead to exploitable vulnerabilities.
My client's vendor can't even implement CSV right. If you put quote-pipe-quote "¦" in any field, say an account name or transaction description, it will break the banks backend software. They will literally be unable to generate reports.
I won't say the name of the bank or vendor for obvious reasons. But I've already created a paper trail for when it happens.
But really I always find it kind of fascinating that plain old ASCII has a set of characters for this kind of stuff, including 0x1C, 0x1D, 0x1E, 0x1F for file, group, record and unit separators, but the real-world usage seems to be about zero.
I've seen a ton of differently flawed variants of CSV, TSV and whathaveyou, including
one whose vendor claims it's XML, and insists on using a .xml extension, but is in fact values separated by a character, and records separated by a different character; one might call the format "character-separated values" or something
one where the first row isn't CSV at all, nor is it headers; it is a horizontal set of key-value pairs
one where the last row must be ignored, for it is aggregates
many that don't handle whitespace in cells
many that are clearly just implemented with split/join
(As an aside: when opening a CSV in Excel through double-clicking, do not save it unless you're sure you know what you're doing. They may have since fixed it, but for years, if not decades, this would silently overwrite cells with what it thinks is the correct data format. Hope you enjoy your +1 (555) 123 456 7 phone number becoming a float with scientific notation! Instead, open it with Excel's Data tab.)
I've never actually seen a piece of software use the ASCII record separator, etc.
And I think the answer as to why is simple: it squanders the main benefit people see today in CSV, which is human-readability. You open it in a text editor and the meaning of the format is clear as day. Non-printable ASCII chars ruin that. At that point, you might as well use a more sophisticated format.
I think human-editability also comes into it, as in, people likely gravitate towards separators that they can type on their keyboard. So we get stuff like "¦" rather than ␟ and \n instead of ␞
(And now funnily enough we have glyphs for ␞ and the like, at entirely other character points.)
The IETF has made several changes to the HTTP/1.1 spec over the years to cover security vulnerabilities that have been caused by poor implementations.
One instance of this is the removal of arbitrary amounts of white space before and after the colon in HTTP headers.
The reality is that a lot of software is written by the lowest bidder and the specs need to be written in a way that minimizes the likelihood that implementers will make mistakes.
TL;DR specs need to be written so that they're idiot proof.
The IETF has a series of RFCs that document current best practices (you should take a look, they are actually pretty good reads when relevant). As the post mentions there are some parts of the actual RFC that don’t make clear the security impacts of some parts of the spec. A best practices for implementing HTTP/2 and HTTP/2 -> HTTP/1.1 translation could explain some of those pitfalls and good ways to mitigate them. Or at least an errata on the existing RFC
In HTTP/1, duplicate headers are useful for a range of attacks, but it's impossible to send a request with multiple methods or paths. HTTP/2's decision to replace the request line with pseudo-headers means this is now possible. I've observed real servers that accept multiple :path headers, and server implementations are inconsistent in which :path they process:
That's nonsense. You can send a HTTP/1 request with multiple methods or paths. The server will just reject it.
DO THE SAME THING WITH HTTP/2. Don't guess which one to use, just tell the client no using error code 400 BAD REQUEST.
Maybe that's a joke, but in case it's not: how are you thinking of including multiple HTTP methods in a single request? By definition, the HTTP method is the first word in the start-line. In DO THE SAME.... only DO is the HTTP method... then the server should try to parse THE SAME ... as a URI which should fail even if it didn't already fail because DOis not a known HTTP verb. If you get either of these errors, you don't send a 400 response at all, as the request is malformed you should close the connection (otherwise there's no way to know when the next request starts!).
What if your first and second line have the HTTP method. And then for some servers it takes the first occurrence and for some servers it takes the second?
Sounds pretty stupid, doesn't it?
But that's what I'm seeing with this example where the verb or path is set twice and the server doesn't reject it.
What if your first and second line have the HTTP method. And then for some servers it takes the first occurrence and for some servers it takes the second?
By this logic, me writing a stupid server that takes the first line, wraps it in a bash script preceded by "rm -rf *" and then executing said bash script is proof that HTTP/1 allows remote execution of arbitrary scripts rather than saying "a non-conforming implementation of HTTP/1 can have bad shit happen"
I can shove anything I want into the request regardless of what your "definition" says I am allowed to use. That's rule zero of building a server.
Your argument is basically "We're going to assume that all HTTP/1 clients follow the rules, but we aren't going to make the same assumption for HTTP/2 clients".
Due to how HTTP/1 is designed, any server seeing a request with two request lines with method and path, will assume that the second line is an (invalid) header, NOT a header line. This should be obvious to anyone who has written a parser before.
The parsing logic works like this (simplified):
method, path = parse_first_line()
while true:
line = read_next_line()
if is_empty(line):
parse_body()
break
else:
parse_header(line)
HTTP/1.1 only has a single start-line which should contain VERB URI HTTP_VERSION. The next line must be a header line, or if empty, it initiates the body (if any). I can't even comprehend how a HTTP server would get confused with this sort of thing, but I don't doubt after reading the other post titled "HTTP/1.1 should die" (which is all about HTTP parser absurd mistakes just like this).
Accidentally allowing a header to be parsed twice is easy. If you don't explicitly check for it in parse_header, it won't be checked. I don't know how you'd accidentally parse the first line twice. Like, what would you be doing in a loop that could detect "first line" more than once? The way you know that it's the HTTP/1 start line is because it's the first line. That's hard to screw up.
I think the idea here is that it's harder to catch this, because a naive implementation would check data frame length field and ignore the Content-Length field, and if you trigger a downgrade from HTTP/2 to HTTP/1, the Content-Length field ends up being ignored.
That being said, the RFC clearly indicates this should be verified:
A request or response that includes a payload body can include a content-length header field. A request or response is also malformed if the value of a content-length header field does not equal the sum of the DATA frame payload lengths that form the body.
It's a programmer error, 100%. But there is something to be said for architectures that make it easier for programmer errors to happen (hence attacks on Java because it doesn't require explicit nullability marking).
This feels like the person being attacked has to do a lot of stupid in order to make themselves vulnerable.
Say I send a pair of messages to server A and that reroutes them to servers A1 and A2 based on the message type. But I trick it into sending both messages to server A1, so what? Server A1 should fail on any message it wasn't designed to handle. It shouldn't do the 'bad thing'.
Furthermore, what is server A's role here? It's not behaving like a load balancer, just passing along messages. Nor is it behaving like a front-end server that interprets my messages and creates new ones for the backends from which is assembles a response.
No, it "rewrites requests into HTTP/1.1 before forwarding them on to the back-end server."
Why the fuck would you do that? If anything, it should be upgrading requests to HTTP/2 in order to improve backend performance without requiring clients to upgrade.
Well, perhaps you wanted to use the image with software that doesn't support JPEG 2000, like most software, because JPEG 2000 doesn't have much market saturation even 25 years later and JPEG is one of the most popular image formats on the planet. So you convert it to the less "advanced" but more popular format.
That's just incompetence. According to Wikipedia, most browsers implemented HTTP/2 in 2015.
After a decade there's no excuse to have not updated their backends software to be compatible with their front end. What other patches are they missing on those old servers?
The disconnect here seems to be the assumption that because HTTP/2 exists, HTTP/1 is now bad and should be avoided. There are no plans to discontinue HTTP/1. It still works just as well as it has for the past ~30 years, and will continue to do so.
HTTP/2 provides new capabilities that are appealing for high-performance websites like Google, but are less interesting for sites with lower traffic or less dynamic pages (which is most of them). HTTP/2 adds a significant amount of complexity over HTTP/1 and would necessitate significant backend changes to take full advantage of the features, so without a pressing reason to migrate, many businesses haven't. Switching to HTTP/2 is a business decision, not a moral imperative.
HTTP/2 requires TLS, and vast majority of the time you won't have set up mTLS between your internal servers, so HTTP/2 is not possible. Yes there's H2C (http2 cleartext) but it's technically non-standard and not well supported everywhere.
In another thread I read that they may do it to eliminate the TLS overhead in the backend. Though that sounds like they're just begging for an internal attack.
That's "not their decision". It's my compute, my overhead, my decision.
It infuriates me when developers working on something very widely used do something for "their own convenience" or because they can't imagine someone else making a different tradeoff.
The classic example of this is "azcopy", the Azure blob copy tool. For about a decade it insisted on writing tens of gigabytes of log files to disk, with essentially no way to turn it off. Just about anyone who had ever used it got bitten by this, because if you run a scheduled sync job then eventually your disk would fill up and everything would crash.
Why was this so? Why no option to turn it off until recently?
Because it was convenient for the developer for troubleshooting purposes!
No, it "rewrites requests into HTTP/1.1 before forwarding them on to the back-end server."
Why the fuck would you do that?
Typical situtation is having proxy server that terminates https traffic in http2 and converts it to http1.1 to be handled by backend. It is classic configuration of nginx for example. This allows contain all complexity about https and http2 in nginx and write application backend code without doing complex parsing of http and TLS.
And if anyone asks why I transitioning to pure database development, just point them to your post with the additional line "And they're all this bad or worse".
We've seen that HTTP/2's complexity has contributed to server implementation shortcuts, inadequate offensive tooling, and poor risk awareness.
No. We saw incompetent developers making obvious mistakes.
Am I saying that I'm smart enough to make a secure HTTP server? Hell no, that shit is hard. You need a team of experts checking each other's work to get this right.
But I at least know how to do basic fucking parameter validation. And while I'm usually quite happy to call out bad specifications, I'm pretty sure HTTP/2 doesn't say, "When you get an ambiguous message, just go ahead and flip a coin".
Most front-ends are happy to send any request down any connection, enabling the cross-user attacks we've already seen. However, sometimes, you'll find that your prefix only influences requests coming from your own IP. This happens because the front-end is using a separate connection to the back-end for each client IP. It's a bit of a nuisance, but you can often work around it by indirectly attacking other users via cache poisoning.
Again, you have to be doing something really stupid for this to work. Why the fuck is the front end server, which is acting as a load balancer, trying to cache anything?
Obviously this is going to fail. You don't even need an attacker. You just have to understand that "GET /user/account/info" is going to return a different answer for each user.
nicheComicsProject@reddit
So I didn't read the whole thing but all the ones I saw were actually vulnerabilities with HTTP/1 downgrade.
Chisignal@reddit
All of these are like 100x more severe than I'd expect, that's insane.
This too is nuts, exfiltrating credentials and essentially hijacking entire HTTP requests is scary, but essentially locking the server into a state where it mis-serves user requests and starts authenticating end users under random different users, that's nuclear havoc lol
grauenwolf@reddit
I've seen it before. The idiots where I worked were shoving user permissions in a static variable instead of passing it as a parameter. The bug was in production for weeks before I figured it out.
The scary part for me was that I was a junior developer at the time and decided to send a mass email to the whole dev team explaining how basic static variables worked and why they shouldn't be using them in a web server. The very same day the idiot who made this mistake was promoted to team lead. So if anyone bothered the check the change history on that file it would have looked like I was attacking my 'better'.
nemec@reddit
This happened where I work just last week lol
Somebody switched our HTTP request handler from sync to async without realizing threads/threadlocal variables are now shared between simultaneous requests... luckily it's a greenfield project so no customers were affected. Yikes though.
grauenwolf@reddit
I'm so happy that AysncLocal is now a thing in C#. Before it, state management in async functions was a right pain in the ass.
That said, it's an understandable mistake. ThreadLocal is an esoteric technique on its own, AsyncLocal is even harder to understand.
grauenwolf@reddit
I'm so happy AsyncLocal is now a thing in C#. Without it state management with async was a right pain in the ass.
SoPoOneO@reddit
I don’t doubt you’re right but I’d be interested in understanding more about the problem you describe. In this case does “static variable” mean what is also sometimes called a “class variable”? And if so, was the problem that a given users permissions were being given to another user? And if so, how had any notion of concurrent sessions worked?
grauenwolf@reddit
Yep. And in VB it's literally called "shared".
Yes. But only if two requests occurred at exactly the same time on the same server. So it was hard to reproduce without a heavy load.
The value was read almost immediately after being written and never again. So the window of opportunity to trigger the bug was very small.
You need another thread to be running and changing X while my thread is on step 2.
SoPoOneO@reddit
Thank you for the clarity. I am having sympathetic dry heaves imagining dealing with this.
grauenwolf@reddit
This was back when .NET was still relatively new and most of the world still ran on PHP or Classic ASP. We didn't have stuff like ConcurrentDictionary, just raw mutexes if you didn't want to break your collections.
I went through the code base and purged every static variable. Thankfully there weren't many.
MassiveInteraction23@reddit
The flatness of reward fo programmers is remarkable to me. The fact that time-served seems to be the major determinant of position and salary (when attrition in the field is low) is bonkers.
Lots of people will agree that a small number of programmers produce an outsized positive impact, but reward structures seem to rarely reflect this.
It’s partly just the hard problem of assessing contribution. But it also feels like a cultural problem: with programming having become a destination for making money rather than craft the peer-evaluation structure feels like it’s emphasized blind rewards rather than critical consideration. — That’s just feel mind you. As mentioned, actually judging contributions is hard. So I could be wrong.
chucker23n@reddit
Perhaps, though I'd say oftentimes, it's a well-oiled team as a whole that makes the difference. And for a team lead to discover that someone in a team is the mythical 10x solo programmer is tricky. The incentive structures are also hairy.
UnnamedPredacon@reddit
There's part, but there's the whole organization to account for.
Time served, while usually not exciting, it's a stable variable that has some value (someone that spends enough time in an organization should know how to run it better), but it creates a roadblock to nonstandard career paths. And credit is a double edged sword because politics. Toxic bosses will steal credit from their subordinates.
There should be a career path for people who don't want to be leaders, and honesty should be properly valued, not just give it lip service.
pozorvlak@reddit
And conversely, a programmer in an impactful position who's in over their head can create an outsized negative impact. Worse, it can be very hard to recognise such programmers until the damage is already done.
MINIMAN10001@reddit
I feel like most people who care to check change history
They realize that mistakes stand on their own, no one forced them
They were just curious in who was to blame
But I don't actually know about what programming politics looks like
Jaggedmallard26@reddit
I mostly use it to determine who to aim questions at if there's a line doing something I don't understand the significance of
gimpwiz@reddit
IMO programming politics within an org is a combination of 1) navel-gazing ego, and 2) job stability.
If you get people with not just an ego ("I'm pretty good") but a comparative one in which they put others down ("I'm way better than these idiots"), you get a lot of political bullshit. If people think stepping on someone will lead to their job being safe(r) or a promotion, you get a lot of political bullshit. On the flip side, if people are neither too worried about justifying their jobs on a daily basis nor think everyone else is a goddamn idiot, an org has a pretty high chance of having only the "people care about their work" sort of politics rather than "people have knives out" sort of politics.
grauenwolf@reddit
Thankfully I never found out the politics either. I think everyone was just happy to have the bug fixed.
mwcz@reddit
I've certainly implemented things in a way I knew at the time to be suboptimal because there were constraints that wouldn't be knowable to a new hire. If later clean up proves impossible, I'll happily explain the constraints to anyone willing to listen so the ideal implementation can hopefully be made real.
If someone, without learning the surrounding context and constraints, called me an idiot for that initial implementation, I take no offense. Only the caller is diminished.
"Static variable" is a very overridden term with quite a few definitions. I really have to stretch to think of a definition where storing auth information is okay. I think it's almost certainly egregious. Still, asking for the explanation exposes the problem just as well as blasting, yet elevates the asker.
BigHandLittleSlap@reddit
Layer 7 load balancers always annoyed me, in part because of these protocol complexities.
Something I'd like to see is web frameworks and cloud platforms with better support for direct networking between clients and servers. In an era where ASP.NET can talk HTTP/3 natively, there are significant downsides to proxying through some slightly incompatible middle-box.
One approach would be to use stateless (random) load balancing for the 'www' domain, and then use random/unpredictable domains with wildcard DNS pointing at individual servers for sticky sessions. The DNS server could use something like consistent hashing to direct traffic at individual VMs.
perk11@reddit
Good Layer 7 load balancers can be cheaper at serving static files and fighting off DDoS attacks. You don't want to have every client specialize in DDoS protection.
tajetaje@reddit
Honestly I feel like the IETF should put out an RFC about these vulnerabilities
grauenwolf@reddit
But what would it say?
I'm not sure that's going to go over well.
tofagerl@reddit
RFC 10.000:
You SHOULD follow all the previous 9.999 RFCs.
grauenwolf@reddit
I wish.
My client's vendor can't even implement CSV right. If you put quote-pipe-quote
"¦"in any field, say an account name or transaction description, it will break the banks backend software. They will literally be unable to generate reports.I won't say the name of the bank or vendor for obvious reasons. But I've already created a paper trail for when it happens.
gimpwiz@reddit
Some guy 20 years ago: "Yeah, I'll use a pipe as a special character to denote special behavior. Nobody would ever enter that into the text field"
anon_cowherd@reddit
That is literally the exact situation I am dealing with now, except instead of 20 years ago it was 2.
grauenwolf@reddit
Not pipe, quote-pipe-quote. That 3 character sequence is literally the field separator.
And this is fairly new software. It replaces the old mainframe system the vendor used to sell.
syklemil@reddit
must be a sibling of little Bobby Tables.
But really I always find it kind of fascinating that plain old ASCII has a set of characters for this kind of stuff, including
0x1C,0x1D,0x1E,0x1Ffor file, group, record and unit separators, but the real-world usage seems to be about zero.chucker23n@reddit
I've seen a ton of differently flawed variants of CSV, TSV and whathaveyou, including
.xmlextension, but is in fact values separated by a character, and records separated by a different character; one might call the format "character-separated values" or something(As an aside: when opening a CSV in Excel through double-clicking, do not save it unless you're sure you know what you're doing. They may have since fixed it, but for years, if not decades, this would silently overwrite cells with what it thinks is the correct data format. Hope you enjoy your
+1 (555) 123 456 7phone number becoming a float with scientific notation! Instead, open it with Excel's Data tab.)I've never actually seen a piece of software use the ASCII record separator, etc.
And I think the answer as to why is simple: it squanders the main benefit people see today in CSV, which is human-readability. You open it in a text editor and the meaning of the format is clear as day. Non-printable ASCII chars ruin that. At that point, you might as well use a more sophisticated format.
syklemil@reddit
I think human-editability also comes into it, as in, people likely gravitate towards separators that they can type on their keyboard. So we get stuff like
"¦"rather than␟and\ninstead of␞(And now funnily enough we have glyphs for ␞ and the like, at entirely other character points.)
chucker23n@reddit
Yep.
grauenwolf@reddit
Same. I'm surprised that I never came across them being used correctly in the wild.
tofagerl@reddit
I have, but only in legacy software...
tofagerl@reddit
... what...? Is this guy available for children's parties?
(It's a clown joke!)
SputnikCucumber@reddit
The IETF has made several changes to the HTTP/1.1 spec over the years to cover security vulnerabilities that have been caused by poor implementations.
One instance of this is the removal of arbitrary amounts of white space before and after the colon in HTTP headers.
The reality is that a lot of software is written by the lowest bidder and the specs need to be written in a way that minimizes the likelihood that implementers will make mistakes.
TL;DR specs need to be written so that they're idiot proof.
Majik_Sheff@reddit
I'm the time it took you to write that the universe popped out 2 more iterations of a better idiot.
Shivalicious@reddit
I like you practising what you preach by adding an idiot-proof TL;DR to your spec for writing specs.
tajetaje@reddit
The IETF has a series of RFCs that document current best practices (you should take a look, they are actually pretty good reads when relevant). As the post mentions there are some parts of the actual RFC that don’t make clear the security impacts of some parts of the spec. A best practices for implementing HTTP/2 and HTTP/2 -> HTTP/1.1 translation could explain some of those pitfalls and good ways to mitigate them. Or at least an errata on the existing RFC
grauenwolf@reddit
Good point.
grauenwolf@reddit
That's nonsense. You can send a HTTP/1 request with multiple methods or paths. The server will just reject it.
DO THE SAME THING WITH HTTP/2. Don't guess which one to use, just tell the client no using error code
400 BAD REQUEST.renatoathaydes@reddit
Maybe that's a joke, but in case it's not: how are you thinking of including multiple HTTP methods in a single request? By definition, the HTTP method is the first word in the start-line. In
DO THE SAME....onlyDOis the HTTP method... then the server should try to parseTHE SAME ...as a URI which should fail even if it didn't already fail becauseDOis not a known HTTP verb. If you get either of these errors, you don't send a 400 response at all, as the request is malformed you should close the connection (otherwise there's no way to know when the next request starts!).grauenwolf@reddit
What if your first and second line have the HTTP method. And then for some servers it takes the first occurrence and for some servers it takes the second?
Sounds pretty stupid, doesn't it?
But that's what I'm seeing with this example where the verb or path is set twice and the server doesn't reject it.
KyleG@reddit
By this logic, me writing a stupid server that takes the first line, wraps it in a bash script preceded by "rm -rf *" and then executing said bash script is proof that HTTP/1 allows remote execution of arbitrary scripts rather than saying "a non-conforming implementation of HTTP/1 can have bad shit happen"
grauenwolf@reddit
I can shove anything I want into the request regardless of what your "definition" says I am allowed to use. That's rule zero of building a server.
Your argument is basically "We're going to assume that all HTTP/1 clients follow the rules, but we aren't going to make the same assumption for HTTP/2 clients".
vincehoff@reddit
In HTTP2 it is valid syntax but invalid semantics. In HTTP1 it is invalid syntax.
grauenwolf@reddit
There's no reason to make that distinction. Invalid is invalid.
A1oso@reddit
There is an important distinction, though:
Due to how HTTP/1 is designed, any server seeing a request with two request lines with method and path, will assume that the second line is an (invalid) header, NOT a header line. This should be obvious to anyone who has written a parser before.
The parsing logic works like this (simplified):
renatoathaydes@reddit
HTTP/1.1 only has a single start-line which should contain
VERB URI HTTP_VERSION. The next line must be a header line, or if empty, it initiates the body (if any). I can't even comprehend how a HTTP server would get confused with this sort of thing, but I don't doubt after reading the other post titled "HTTP/1.1 should die" (which is all about HTTP parser absurd mistakes just like this).grauenwolf@reddit
And I can't understand why a HTTP/2 server would get confused when it saw two headers.
TinyBreadBigMouth@reddit
An HTTP/1 parser would look something like this:
Accidentally allowing a header to be parsed twice is easy. If you don't explicitly check for it in
parse_header, it won't be checked. I don't know how you'd accidentally parse the first line twice. Like, what would you be doing in a loop that could detect "first line" more than once? The way you know that it's the HTTP/1 start line is because it's the first line. That's hard to screw up.KyleG@reddit
I think the idea here is that it's harder to catch this, because a naive implementation would check data frame length field and ignore the Content-Length field, and if you trigger a downgrade from HTTP/2 to HTTP/1, the Content-Length field ends up being ignored.
That being said, the RFC clearly indicates this should be verified:
It's a programmer error, 100%. But there is something to be said for architectures that make it easier for programmer errors to happen (hence attacks on Java because it doesn't require explicit nullability marking).
grauenwolf@reddit
Honestly, this is probably the core of my complaint. If it was framed as "Common ways people mess up HTTP/2" then I would on board with the message.
Instead of frames it as a problem with the specification and the developers were innocently just following orders.
grauenwolf@reddit
This feels like the person being attacked has to do a lot of stupid in order to make themselves vulnerable.
Say I send a pair of messages to server A and that reroutes them to servers A1 and A2 based on the message type. But I trick it into sending both messages to server A1, so what? Server A1 should fail on any message it wasn't designed to handle. It shouldn't do the 'bad thing'.
Furthermore, what is server A's role here? It's not behaving like a load balancer, just passing along messages. Nor is it behaving like a front-end server that interprets my messages and creates new ones for the backends from which is assembles a response.
No, it "rewrites requests into HTTP/1.1 before forwarding them on to the back-end server."
Why the fuck would you do that? If anything, it should be upgrading requests to HTTP/2 in order to improve backend performance without requiring clients to upgrade.
yohslavic@reddit
Exactly, while reading half through the article I was like, why the fuck would you have a server switch to http 1, who even does that
Mehis_@reddit
AFAIK nginx only supports HTTP/1.1 for proxying. It's very common.
grauenwolf@reddit
That's their excuse but... https://nginx.org/en/docs/http/ngx_http_v2_module.html
sl8_slick@reddit
Nginx supports HTTP/2 to the frontend, but for backend connections it only supports up to HTTP/1.1
grauenwolf@reddit
Why? It's been a standard for a decade?
Assume I'm a complete idiot, or worse an executive, and explain to me why anyone would want to use a technology that's literally a decade out of date.
TinyBreadBigMouth@reddit
Do you use JPEG image files? Why on earth would you use JPEGs when the JPEG 2000 standard came out 25 years ago??
grauenwolf@reddit
No, I predominantly use PDN and PNG files. But let's say U wasn't.
Why would I convert JPEG 2000 files into JPEG files instead of using them as is? What benefit would be for doing that?
Your analogy makes no sense.
TinyBreadBigMouth@reddit
Well, perhaps you wanted to use the image with software that doesn't support JPEG 2000, like most software, because JPEG 2000 doesn't have much market saturation even 25 years later and JPEG is one of the most popular image formats on the planet. So you convert it to the less "advanced" but more popular format.
grauenwolf@reddit
Your analogy is still stupid. What backend server doesn't support HTTP/2? And why would you choose it when so many do?
Can we please talk about the topic at have instead of this weird tangent about file types?
DHermit@reddit
Because the backend likely doesn't support HTTP/2.
grauenwolf@reddit
That's just incompetence. According to Wikipedia, most browsers implemented HTTP/2 in 2015.
After a decade there's no excuse to have not updated their backends software to be compatible with their front end. What other patches are they missing on those old servers?
TinyBreadBigMouth@reddit
The disconnect here seems to be the assumption that because HTTP/2 exists, HTTP/1 is now bad and should be avoided. There are no plans to discontinue HTTP/1. It still works just as well as it has for the past ~30 years, and will continue to do so.
HTTP/2 provides new capabilities that are appealing for high-performance websites like Google, but are less interesting for sites with lower traffic or less dynamic pages (which is most of them). HTTP/2 adds a significant amount of complexity over HTTP/1 and would necessitate significant backend changes to take full advantage of the features, so without a pressing reason to migrate, many businesses haven't. Switching to HTTP/2 is a business decision, not a moral imperative.
grauenwolf@reddit
It's enabled by default in IIS. I literally have to do nothing to benefit from it.
You're acting like we have to hand roll our own HTTP/2 support when really we just need to avoid obsolete software that doesn't fully support it.
MaxGhost@reddit
HTTP/2 requires TLS, and vast majority of the time you won't have set up mTLS between your internal servers, so HTTP/2 is not possible. Yes there's H2C (http2 cleartext) but it's technically non-standard and not well supported everywhere.
grauenwolf@reddit
That's not a decision that Azure Application Gateway and nginx should be making on our behalf.
grauenwolf@reddit
In another thread I read that they may do it to eliminate the TLS overhead in the backend. Though that sounds like they're just begging for an internal attack.
BigHandLittleSlap@reddit
That's "not their decision". It's my compute, my overhead, my decision.
It infuriates me when developers working on something very widely used do something for "their own convenience" or because they can't imagine someone else making a different tradeoff.
The classic example of this is "azcopy", the Azure blob copy tool. For about a decade it insisted on writing tens of gigabytes of log files to disk, with essentially no way to turn it off. Just about anyone who had ever used it got bitten by this, because if you run a scheduled sync job then eventually your disk would fill up and everything would crash.
Why was this so? Why no option to turn it off until recently?
Because it was convenient for the developer for troubleshooting purposes!
grauenwolf@reddit
You make a strong argument.
PainInTheRhine@reddit
Azure for example
angelicosphosphoros@reddit
Typical situtation is having proxy server that terminates
httpstraffic in http2 and converts it to http1.1 to be handled by backend. It is classic configuration of nginx for example. This allows contain all complexity about https and http2 in nginx and write application backend code without doing complex parsing of http and TLS.TheNamelessKing@reddit
Plenty of other things don’t need to downgrade the connection,
Anything based on Envoy for example.
angelicosphosphoros@reddit
Good for people who use Envoy then.
PainInTheRhine@reddit
For example Azure Application Gateway downgrades backend connection to http/1.1 . And there is no way to change it
grauenwolf@reddit
And if anyone asks why I transitioning to pure database development, just point them to your post with the additional line "And they're all this bad or worse".
PainInTheRhine@reddit
Google application load balancer handles http2 on the backend side though. No idea about AWS
grauenwolf@reddit
No. We saw incompetent developers making obvious mistakes.
Am I saying that I'm smart enough to make a secure HTTP server? Hell no, that shit is hard. You need a team of experts checking each other's work to get this right.
But I at least know how to do basic fucking parameter validation. And while I'm usually quite happy to call out bad specifications, I'm pretty sure HTTP/2 doesn't say, "When you get an ambiguous message, just go ahead and flip a coin".
grauenwolf@reddit
Again, you have to be doing something really stupid for this to work. Why the fuck is the front end server, which is acting as a load balancer, trying to cache anything?
Obviously this is going to fail. You don't even need an attacker. You just have to understand that "GET /user/account/info" is going to return a different answer for each user.
nekokattt@reddit
Can't wait for h3/quic