new python http requests library

Posted by o2mz@reddit | Python | View on Reddit | 33 comments

Today, I finished writing my first Python library, which is an HTTP requests library designed to be easy to use, high-performance, and very lightweight. It utilizes standard libraries and only imports functions from those libraries instead of importing the entire libraries.

The library is built on socket and ssl.

To install the library: pip install fiberhttp

Source code: https://github.com/xsxo/fiberhttp

This post is not intended for promoting the library but rather to gather feedback, improve the library, and exchange programming experiences.

[-]

Python-ModTeam@reddit

Greetings from the r/Python mods!

Your post has been removed as it does not adhere to our Showcase post formatting requirements. To ensure clarity and quality, we require the following sections:

What My Project Does
Target Audience (e.g., Is it meant for production, just a toy project, etc.
Comparison (A brief comparison explaining how it differs from existing alternatives.)

Please update your post to include these elements and feel free to resubmit. Thank you for your understanding and for contributing to r/Python!

[-]

proggob@reddit

What is the status of requests replacement? I was waiting for httpx to reach 1.0 but it’s been years. There’s niquests, which seems a bit of a one person project. aiohttp only supports async.

[-]

o2mz@reddit (OP)

The library is currently not recommended for use. I will update it tomorrow and let you know. I will also add a benchmarks folder that contains a comparison of the library with others like requests and httpx.

[-]

buqr@reddit

Have you run any benchmarks to show that it is actually faster?

[-]

o2mz@reddit (OP)

i will add benchmark folder to Performance comparison with other librarys like

requests, httpx, http-client….

thank you

[-]

yrubooingmeimryte@reddit

This isn't answering the question. Did you do any benchmarking before making the claim that your library is "high-performance"?

[-]

o2mz@reddit (OP)

Since the library sends HTTP requests almost directly, it is quite natural for it to have high performance. However, I will compare it with other libraries and will mention you in this.

[-]

riklaunim@reddit

Note that false claims, misleading marketing can have legal consequences if/when you would do ir on the commercial market. Don't do that no matter the case.

[-]

pokeybill@reddit

If you haven't even taken the first step towards benchmarking it's irresponsible to make the claim it will be faster than industry standard http servers.

[-]

StoryRadiant1919@reddit

You lose even more credibility for making unsubstantiated claims and then hoping people won’t notice. bad stuff.

[-]

bjorneylol@reddit

If I import json and use it directly you would think that would be "high performance", yet msgpack and orjson managed to make their libraries 100x faster

[-]

yrubooingmeimryte@reddit

Other libraries also send the requests "almost directly". So what speed advantage are you assuming you will get that other libraries won't? How much faster would a library need to be for you to think that it is "high-performance" relative to the other standard libraries?

[-]

fiskfisk@reddit

Follow PEP-8
Don't return strings that convey different kinds of errors or states. This means that any caller needs to check the content of the string to know what happened. If you don't intend to handle an exception, don't catch it and let it bubble instead (and you're missing a return on at least one of those strings in close_host).
This will assume that any request that is not against port 80 is an SSL request. This is very, very often not the case.
It seems like your port parsing is broken if the URL contains basic HTTP authentication, although I haven't tried to fully parse what you're doing. You're already using urlparse other places; use it. Don't do it manually. You'll do it wrong unless you're familiar with the standards.
You have the same issue where you try to manually handle urls by simply calling split in other locations as well.
You have `data = data` in your post method.
You have dist files committed to your regular version control repository
You have commented out lines of codes in your init file
status_code on the response returns a string (and lacks typing information, although most other methods in the same class has it)
The headers method parses the response every time the headers need to be return. Parse it when the class gets created and return a multidict. (why multidict? a header can be repeated if the value is defined as a list of comma separate values)
Your header parser assumes that every header key and value is separated by ": " - this is not correct; the space is optional.
Headers are also case insensitive, and can span multiple lines
Any HTTP responses which contain \r\n\r\n will lose anything after the \r\n\r\n in the response.
I stopped reading code

It's perfectly fine as a playground and to learn the HTTP protocol, but no-one should use it for anything useful (and it should not be presented as such) in its current state.

[-]

o2mz@reddit (OP)

This is my first library, and I will correct all these issues, especially regarding reading headers and ports. I will write it in a better way. Thank you for taking the time to provide all these tips.

[-]

andartico@reddit

May I compliment you on how you reacted to this feedback. Well taken. Kudos!

[-]

fiskfisk@reddit

And any library needs to have a suite of unit tests that makes sure that it behaves as it should, and that you don't break backwards compatibility in future releases.

Making a HTTP library is a hard sell as you already have requests and aiohttp, but it's an easy-ish protocol to handle and play around with. There's quite a few details that tend to catch people out when writing clients, though - so you'll probably want to spend some time reading the RFCs as well (and compare how your library is behaving against the existing de facto standard libraries).

[-]

o2mz@reddit (OP)

yes i will add test and benchmark folders in project

this is my first project

It was not a serious library, it was for the purpose of experimentation and practice.

[-]

arwinda@reddit

will add test

Start with writing tests for the result you expect, then write the code and test it against your unit tests.

[-]

andartico@reddit

This ist one of the most valuable feedbacks I read in a long time.

Purely on the technical point.

Well structured.

Without fluff.

Thanks for being an example in this world.

[-]

Stishovite@reddit

Maybe you could name it "requests"

I jest

[-]

magnetichira@reddit

This post belongs on r/learnpython

[-]

TitaniumWhite420@reddit

In this context, your comment comes off to me as condescending and derisive, which makes sincere attempts to direct people to r/learnpython in appropriate cases feel also more condescending than what should be necessary.

[-]

DivineSentry@reddit

Im not sure how only importing the functions directly helps with performance. When you do that it still runs the entire module, the difference is what names will go into your scope

[-]

tacothecat@reddit

Does python provide any introspection mechanism to have dynamic behavior when symbols are bound in locals/globals? In which case, maybe there COULD be a difference but unlikely.

[-]

DivineSentry@reddit

It sounds like you’re talking about lazy loading, which must be explicitly implemented, which they don’t do here

[-]

o2mz@reddit (OP)

I was too quick to share this information before verifying it. I apologize for this mistake and thank you for correcting it.

[-]

cnelsonsic@reddit

You say it's high performance, but include no benchmarks.

How does it compare against `http.client`, `urllib.request`, or `Requests`?

There are also zero tests.

[-]

o2mz@reddit (OP)

i will add benchmark folder to Performance comparison with other librarys like

requests, httpx, http-client….

thank you

[-]

yrubooingmeimryte@reddit

Why did you even make the claim if you haven't actually benchmarked it against any of the competitors?

[-]

yrubooingmeimryte@reddit

LOL, I love how these homemade projects always try to use "lightweight" as a benefit when in reality, in their case, it just means that there has been very little development done relative to the competitors.

[-]

houseofleft@reddit

Hey, looks like a cool project. I think some of the negative comments are because python already has some big request libraries (like requests), but I hope it hasn't got you down! I really enjoyed reading through the code.

Have you though about introducing context handlers? You use con.open() and con.close() to keep a connection open, which would be a great candidate for having something like with con.open() as cxn: so that you don't need to remember to clean up after!

[-]

o2mz@reddit (OP)

I forgot to work on this feature. I will add it in the next update, along with the missing files. This project still has a lot to improve since it’s my first project on this platform.

Thank you for your kind words

[-]

denehoffman@reddit

As mentioned, importing functions from a module will import the entire module but only expose the imported functions. I would also consider a .gitignore file, particularly something like this: https://github.com/github/gitignore/blob/main/Python.gitignore (you can and should just copy this). That way you don’t end up committing pycaches or egg info or your dist directory!