Niquests 3.18 — 3 Years of Innovations in HTTP
Posted by Ousret@reddit | Python | View on Reddit | 82 comments
When I started working on Niquests, I dreamed about a "no-compromise" HTTP client, and three years later I finally made it to the end.
The end goal was, at least for me, to reach what the world of today allows us to, today. And, unfortunately the Python community is often stuck with decade old capabilities. (ie. http/1 only, legacy ssl capabilities, ...).
Niquests started as a fork of Requests in mid-2023. The motivation was simple: Requests is frozen, and millions of developers are stuck with a library that does not evolve (feature-wise). I didn't want to reinvent the wheel, the Requests interface is genuinely pleasant to work with. I just wanted to bring it up to speed with what modern HTTP looks like.
What changed in three years?
A lot. Here's some key things:
- HTTP/2 by default, HTTP/3 over QUIC when the server supports it. No hassle.
- OS trust store by default. No more shipping certifi and hoping it stays up to date.
- Certificate revocation checks. How did we ever lived without this?
- DNS over HTTPS, DNS over TLS, DNS over QUIC, DNSSEC. Customizable DNS resolution per session. And overridable at will.
- Async/Await.
- WebSocket/SSE over HTTP/1, HTTP/2 and HTTP/3 through a unified API.
- Happy Eyeballs algorithm.
- Post-quantum security and Encrypted Client Hello (ECH).
- HTTP Trailers, Early Responses (103 Early Hints).
- In-memory certificates for CAs and mTLS. No need to write certs to disk.
- Network fine-tuning and connection inspection: DNS response time, established latency, TLS handshake delay,
all exposed through
response.conn_info. - Native Unix socket support, ASGI/WSGI app direct usage in sessions.
- Runs in the browser through Pyodide/WASM (experimental, added in 3.18).
- Feature parity sync/async with mirrored interfaces.
- Fully type-annotated.
And it's fast, I mean really fast. In a real-world benchmark (details and reproduction steps in the README) sending 1000 requests to httpbingo.org/get:
| Client | Avg time to complete | Protocol |
|---|---|---|
| httpx | 2.087s | HTTP/2 |
| aiohttp | 1.351s | HTTP/1.1 |
| niquests | 0.551s | HTTP/2 |
Migration starts as a one-liner. Replace import requests with import niquests as requests and you're done.
We maintain backward compatibility with the Requests API. Your existing code, your .netrc, your auth flows,
your cookie jars -- they all work. Even requests-mock, responses, betamax and similar third-party extensions
are supported with minimal shims.
It's getting some real traction lately. We're about to cross the 100k pulls per day from PyPI alone, Niquests appearing more commonly in Github code search engine, and folks creating issues whether they found a bug or just to challenge the solution. That's excellent news!
It's been more than a decade since I started doing open source, and so far, it's nowhere near boring me. I'll answer the community as long as I possibly can.
What My Project Does
Niquests is a HTTP Client. It aims to continue and expand the well established Requests library. For many years now, Requests has been frozen. Being left in a vegetative state and not evolving, this blocked millions of developers from using more advanced features.
Target Audience
It is a production ready solution. So everyone is potentially concerned.
Comparison
Niquests is the only HTTP client capable of serving HTTP/1.1, HTTP/2, and HTTP/3 automatically. The project went deep into the protocols (early responses, trailer headers, etc...) and all related networking essentials (like DNS-over-HTTPS, advanced performance metering, etc..)
Project page: https://github.com/jawah/niquests
wyldstallionesquire@reddit
Tried niquests, but I’m running into issues when niquests overwrites urllib3, and that’s killing docker-py in our test suite. Is there a workaround?
pakeha_nisei@reddit
urllib3.future and how it shadows urllib3 is a deal breaker for me, and would be a deal breaker for a lot more people if they actually knew about it.
fiddle_n@reddit
You can install niquests without shadowing urllib3 (there’s a section in the docs on how to do it) - but that the shadowing behaviour is the default behaviour is certainly controversial.
Ousret@reddit (OP)
I know it is. That's why we spend literal hundreds of hours making sure we aren't causing issues. See https://github.com/jawah/urllib3.future/actions/workflows/ci-dead-things.yml https://github.com/jawah/urllib3.future/actions/workflows/integration.yml https://github.com/jawah/urllib3.future/actions/workflows/packaging.yml along with keeping every single tests from upstream. it's not done carelessly.
and if a champion could find the right, and perfect solution for the main goal behind it, I would welcome it immediately.
regards,
pakeha_nisei@reddit
There is a perfect solution - having only a
urllib3_futuremodule and have libraries like niquests import it directly.Libraries like urllib3 are far too widely adopted and caught up in dependency chains to be simply replaced with an entirely different library, especially when it is opt-out and it requires building from the source package with an environment variable set. That is way too much to expect of downstream users. What about binary-only installations? Expect people to manage their own private pip repository just to build wheels of urllib3.future that don't fundamentally break how Python packaging is intended to work?
The situation with HTTP libraries is as depressing as you say it is. But there is no way I would allow usage of niquests in any of my deployments at my company with the way urllib3.future works. There is simply too high a possibility of extremely difficult to diagnose issues. And I don't think even the warnings you put in your documentation accurately convey that.
Ousret@reddit (OP)
it's the way out(...) as Python permits it today. it's not perfect, but perfectly usable and continuously tested.
That's okay. You are free to choose.
The same things applies to urllib3 itself. Over the last 16 months there was some breaking(...) And only experienced users could find the cause.
I would be careful with that thinking. Would you forbid someone to remove Chrome in favor of Firefox with that exact logic? Rare are those who understand the presence of urllib3. And never before a real swap~able alternative existed.
It would prevent more than 80% of our users to use it as a drop in replacement of Requests (extensions, and so on...). Cancelling the real advantage we have, meaning letting inexperienced to experienced people using Niquests "out-of-the-box", friction less.
Regards,
wyldstallionesquire@reddit
Honestly this is the first time, in 20 years, I’ve run across this shadowing behavior in a library. FWIW.
fiddle_n@reddit
I’m sure no perfect solution exists or it wouldn’t be a choice between two options.
But I am surprised that the trade-offs would lead towards this being opt-out behaviour rather than opt-in. Environmental situations like these feels like behaviour users should explicitly decide with eyes wide open.
I suspect most users of niquests are unaware the shadowing behaviour is even a thing. I wasn’t aware until I had used it quite a bit. For me it felt like a surprise to find out, though ultimately not a big deal - but if I had a different use case (such as writing a library) I might have felt differently.
Ousret@reddit (OP)
Maybe so. But it's not what I saw with the documentation analytics. People do read the FAQ and do read the sections around it.
Moreover, the minimal amount of complaint shows that it just work as urllib3 just work.
The other way around is way more complicated to maintain and explain to users in general and ultimately can result in the same results as today with "extra steps". You'd be surprised how much people are relying on it exactly for the shadowing part.
Anyway, the goal in a relative distant future is to get rid of this situation. But that's require significant work and traction.
Regards,
Ousret@reddit (OP)
That's a shame. Feel free to revisit once it's gaining more traction or when HTTP/2 celebrate it's 20th birthday.
The "if they actually knew about it." is misplaced. Silently omitting that:
The fact is that urllib3-future is more conservative than urllib3 itself and more reactive while still providing a way to isolate it (at the cost of loosing the compatibility with 3rd parties), just in case(...) Finally saying things like "nightmare" or "deal breaker" while consciously omitting the "way out" that is clearly documented reveal intellectual dishonesty.
Regards,
Ousret@reddit (OP)
That's the second time I've heard this. And we investigated why the compat layer would fail here. We've found the potential issue and emitted a fix for it, could you retry with latest version of urllib3-future? If still stuck, could you give us more info on Github?
regards,
wyldstallionesquire@reddit
Its specifically with testcontainers. I will try to nail it down more specifically so you can check it out.
titanium_hydra@reddit
not sure if this is just me, but when i added to our project all of a sudden our testcontainers were hanging. the docker api is still using requests so i'm wondering if there is some sort of compatability issue
Ousret@reddit (OP)
I am not sure. If you could open an issue and provide a minimal reproducible example, I would help.
Regards,
titanium_hydra@reddit
yep, when i get some time to dup i'll post something to github, thanks
Ousret@reddit (OP)
I think we've found the issue and fixed it(...) If you'd try again and confirm that it's gone, it would be appreciated.
regards,
takeda64@reddit
Is Niquests for of httpx, or perhaps both are fork of the same code (like requests), I noticed some similarities so I'm just curious.
This is not some kind of attack, it's perfectly fine to fork open source code, I'm just curious about the history.
Far-Captain6740@reddit
You’re comparing with aiohttp/httpx but I’d like to curl_cffi also :>
Ousret@reddit (OP)
Of course, why not. I tried right now and got this:
Using the async session they provide. I suspect that curl_cffi isn't well optimized for asyncio unfortunately. I don't know them well enough.
That being said, if I understand the goal of curl_cffi, provide a TLS engine that would "fool" firewalls into thinking you are not a bot, there is a possible alternative. Historically Python mandate you to use OpenSSL, and with Niquests you can use AWS-LC without effort
pip install niquests[rtls], and your TLS "fingerprint" would be different enough automatically. possibly making you less flagged than before. (+ using DNS over HTTPS would unlock ECH everywhere, giving you a real chance of being stealth)regards,
Far-Captain6740@reddit
Thanks ^^
bboe@reddit
With all the supply chain attacks recently, will you say more about steps you are taking to minimize the chance of attack against your repositories?
Separately, are you still a team of one?
Ousret@reddit (OP)
We were fortunate enough to be part of the Github Secure Open Source program (session 2) and were mentored by true security experts. So yes, we took the step towards securing at scale our packages; charset-normalizer being the most critical part as of right now, we didn't get the slightest issue so far, and hopefully will remain that way.
And, yes, it's still one maintainer so far. Time will surely help, contributors will arrives, I am hopeful.
regards,
bboe@reddit
Thanks for the response. Do you have a contingency plan for the organization should you be unable to continue work on the project? Not having redundancy was, and still is, still my primary concern with moving to this project.
Ousret@reddit (OP)
Of course, I understand. What I can tell you so far is that I've left instructions to my wife in case of an unfortunate event to promote someone (in a short list of trusted individuals). It's planed.
regards,
twigboy@reddit
That is a level of dedication I have not yet reached.
My wife has absolutely zero technical knowledge of any coding projects I have, let alone being able to manage them in my absence
SwampFalc@reddit
To some degree, you just need a very human capability of detecting good intentions. Anyone should have that capability. As long as you maintain a leadership with those good intentions, the actual workload will be assigned to capable people, either within or without the leadership.
jvlomax@reddit
I've actually been planning on replacing httpx in our code after the recent drama. I might actually give this a go!
nurigrf05@reddit
Can you share the "drama"?, we are using httpx too, mostly because it supports async, Im wondering what kind of situations could make a team need to migrate from it...
Thanks
jvlomax@reddit
https://tildeweb.nl/~michiel/httpxyz.html is the crux of it. It just doesn't seem well maintained anymore
nurigrf05@reddit
Thanks
chub79@reddit
I wanted to praise your dedication for this project. Launching a competitor to requests and httpx back then was audacious. But you folks have stuck to it and now it's timely to replace the other two as they are lost in their own shenanigans.
jryan14ify@reddit
Does anyone have a link for how / why requests is frozen?
JimDabell@reddit
They declared a perpetual feature freeze eleven years ago. It doesn’t support HTTP/2, HTTP/3, or async. It took them nine months to fix the last security vulnerability. It’s dangerously unmaintained and nobody should use it. Niquests is the lowest effort way to fix this problem.
ProtossLiving@reddit
Basically, since everyone is using requests as a fundamental building block, they don't want to risk/deal with breaking a lot of them by introducing new incompatibilities. They figure it does what it needs to do. I'm glad niquests stepped in to enable developers who want more though.
https://requests.readthedocs.io/en/latest/dev/contributing/#contribution-suitability
Feature Requests Requests is in a perpetual feature freeze, only the BDFL can add or approve of new features. The maintainers believe that Requests is a feature-complete piece of software at this time.
One of the most important skills to have while maintaining a largely-used open source project is learning the ability to say “no” to suggested changes, while keeping an open ear and mind.
If you believe there is a feature missing, feel free to raise a feature request, but please do be aware that the overwhelming likelihood is that your feature request will not be accepted.
xAragon_@reddit
Can you elaborate on what "shanabigans" on httpx you're referring to?
wRAR_@reddit
https://www.reddit.com/r/Python/comments/1rl5kuq/anyone_know_whats_up_with_httpx/
Ousret@reddit (OP)
when reading the comments, they are all very motivating.
regards,
Decency@reddit
What are the most common reasons that cause code that used to work with Requests to no longer work? I haven't run into anything recently, so just curious where you're seeing the most traction.
Ousret@reddit (OP)
As of today, Requests is still stable and working reliably. It's not that it's malfunctioning, it's rather that sync only networking and http/1 only is a bad combination to survive today's need.
now, that being said, I can tell you that when I started working on Niquests I had a few professional acquaintance that saw a phenomena that keeps growing: "The slow death and termination of HTTP/1", some services are explicitly configured to reject HTTP/1 only client and I keep seeing those reports today, but growing steadily. preparing yourself for the modern http is a good investment. My instincts is telling me that http/1 days are counted (at scale). I wouldn't be surprised to see major provider turning it off soon (e.g. cloudflare, fastly, ...) for some of there customers.
regards,
twigboy@reddit
TIL happy eyeballs is a thing
Sounds like a great library, looking forward to type safe requests in the future.
Any plans in the future for something like requests-html
Ousret@reddit (OP)
You can override the base class parent of HTMLSession to niquests.Session It will work as is. Like
HTMLSession.__bases__ = (niquests.Session,).regards,
twigboy@reddit
Awesome thanks!
Gotta love how badly you can abuse the python system
aplarsen@reddit
Is this pronounced nee-quests? Like the Knights Who Say Ni?
Ousret@reddit (OP)
It's "nee-quests". Influenced by my native (French) tongue.
regards,
explodingman031@reddit
Too good to be true! I will use it in my next project
-balon-@reddit
How would I go about replacing httpx with this given that a lot of existing software uses httpx as a dependency?
Ousret@reddit (OP)
Unfortunately we're not a drop in replacement for httpx. So what I can advise, for example, if you relied on "respx", to now rely on "responses" counterpart. You should find everything on the Requests side.
regards,
-balon-@reddit
Hey thank you for your quick reply!
c_is_4_cookie@reddit
This is really interesting. Where I work, we have to use client certs from the Windows Certificate Store for 2 way TLS. I ended up writing an adapter that creates an SSL context from the cert in the cert store and replaced the https adapter in the Session object.
How much is Adapter interface changed from requests? Just wondering if this will be a heavy lift for me to adopt niquests
Ousret@reddit (OP)
I think it would be simpler with Niquests. Once the mTLS cert, key are extracted, we support passing them as-is (str/bytes) instead of a file.
The adapter should work as-is, but this hack is now unnecessary. If not drop-in compatible, we'll investigate.
regards,
c_is_4_cookie@reddit
Oh, that would be amazing. It drove me nuts that the cert arg had to be a file on disk
Wilbo007@reddit
That "real world" benchmark doesn't mean much. When I run it, aiohttp consistently beats niquests, its essentially the same as your table but niquests and aiohttp is swapped around
Ousret@reddit (OP)
No everyone can benefit from this boost. You can follow that issue and participate in it https://github.com/jawah/niquests/issues/356 And a lot of people could reproduce it, as far as I know. We'll improve for sure.
regards,
MeschDog18@reddit
Thank you so much for your contributions to the open-source community. Been looking for an improvement over requests in a lot of my projects, explored httpx for a while, but this looks like a better solution.
amroamroamro@reddit
Small documentation bug
I was reading the quickstart guide, and in the "Redirection and History" section the recorded demo doesn't match the description (http to https redirection), as is shows empty
r.historyregardless ofallow_redirectsbeing true or false:https://niquests.readthedocs.io/en/latest/user/quickstart.html#redirection-and-history
I'm not sure how the demo is run and embedded in the docs (I saw mention of TerminHTML?)
Ousret@reddit (OP)
We'll investigate and fix it as soon as possible. Sharing details about your browser would help, don't hesitate to open an issue.
regards,
amroamroamro@reddit
I took a quick look at the .rst file:
https://raw.githubusercontent.com/jawah/niquests/refs/heads/main/docs/user/quickstart.rst
I could be wrong about sphinx here, but perhaps the
<in the<Response..>line need to be html-escaped as<?which would explain why i am seeing
[]in the rendered output on readthedocs?https://imgur.com/a/Wc0sY38
Ousret@reddit (OP)
Good catch, I will fix it asap.
regards,
forgotpw3@reddit
Fuck it, let's try this 😏 switching from aiohttp
deadwisdom@reddit
This is all great. Seriously great. But the one feature that would advance the whole damned industry is simply built in caching that works with http caching headers. It’s a bedrock of HTTP and desperately unused because libraries don’t address it.
Ousret@reddit (OP)
HTTP cache is a world on it's own, it would be unreasonable for us to tackle this as Requests ecosystem around caching is well furnished and, actually maintained. Try Requests-cache and alike. They work very well.
regards,
deadwisdom@reddit
Sorry, I'm missing something. Is requests-cache compatible with niquests?
Ousret@reddit (OP)
Yes it is working. And actually I am using it. See https://niquests.readthedocs.io/en/latest/community/extensions.html
deadwisdom@reddit
Ah I see, thanks.
Please emphasize it. People need to understand.
HecticJuggler@reddit
Backwards compatibility with requests is a winner.
onyx_and_iris@reddit
I've recently gone through all my repos and replaced httpx with niquests. Love it, great work and thanks for the library!
garar@reddit
This looks great, thank you for your work!
TheCaptain53@reddit
This looks like an awesome piece of software - keen to try it out.
brandonZappy@reddit
This is amazing! I do have a question about what you meant when you said requests is frozen though? Like no new features? They’re still releasing new versions?
wRAR_@reddit
Yes.
mortenb123@reddit
Import niquests as requests
Worked fine for me
Brandhor@reddit
one problem I have with requests is that some websites don't include the full certificate chain so I have to manually add the missing certificate in the cacert.pem file from certifi, I just tried niquests and it seems to have the same issue so I wonder if in the future niquests might be able to handle these cases like browsers do
Ousret@reddit (OP)
If I am not mistaken, it's already supported. If your OS trust store have it, it should be loaded automatically.
If not, you can still register it easily with
wassima.register_ca(open("./myrootca.pem", "r").read())Your experience is valuable, don't hesitate to open an issue at Github.
Regards,
NSE-Imports@reddit
Started using Niquests a couple of years back in a refactor. A few skill issues on my side but soon sorted thanks to good documentation and a quick GitHub issue.
As I refactor older scripts it will replace requests slowly to give me a unified max performance http access client.
GearsAndSuch@reddit
Nice work!
jsabater76@reddit
Amazing and much needed work for the Python community. Cheers! Keep up the good work! 👏
Gubbbo@reddit
An actually helpful post that is useful to the community.
I don't know if I'm more thankful, or more amazed
ThePrimitiveSword@reddit
Been using niquests since shortly after the first release, thanks for making it!
It's genuinely a lot easier to use than the alternatives, especially with it using the OS trust store, which makes things a lot easier when working in a corporate environment that uses self-signed certs.
Performance is much faster in my experience than requests, real-world performance difference of >10x in my experience, even with a single niquests.get and using it as a drop-in replacement for requests.
Really appreciate how you encourage users to raise an issue on GitHub and always help out.
You've delivered on what requests and urllib3 raised tens of thousands of dollars to do but they never followed through on their promises.
Keep up the great work!
mr_claw@reddit
Drop-in replacement for requests is awesome. I'll be looking into adding this to my app. Thanks.
stone_surgeon@reddit
Great work OP!
pablodiegoss@reddit
Started using niquests recently and I'm very happy with it! Thank you for all your work
AutoModerator@reddit
Hi there, from the /r/Python mods.
We want to emphasize that while security-centric programs are fun project spaces to explore we do not recommend that they be treated as a security solution unless they’ve been audited by a third party, security professional and the audit is visible for review.
Security is not easy. And making project to learn how to manage it is a great idea to learn about the complexity of this world. That said, there’s a difference between exploring and learning about a topic space, and trusting that a product is secure for sensitive materials in the face of adversaries.
We hope you enjoy projects like these from a safety conscious perspective.
Warm regards and all the best for your future Pythoneering,
/r/Python moderator team
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.