What are your favorite modern libraries or tooling for Python?
Posted by COSMOSCENTER@reddit | Python | View on Reddit | 181 comments
Hello, after a while of having stopped programming in Python, I have come back and I have realized that there are new tools or alternatives to other libraries, such as uv and Polars. Of the modern tools or libraries, which are your favorites and which ones have you implemented into your workflow?
ebmarhar@reddit
DuckDB for analytics and data pipelining. You can create a Pandas or Polars data frame and then "select * from my_df"!
BookFingy@reddit
What's the use case? Can it do anything that polars cannot?
clamorfish@reddit
DuckDB can do transformations on larger-than-memory datasets very easily using simple SQL. I love Polars, but it's not as good at this use case.
HNL2NYC@reddit
Last I checked (about a year ago) non equality joins (eg range joins) were way more efficient/faster in duckdb then polars cross join + filter in lazy mode. No idea if that’s changed recently
that_baddest_dude@reddit
Usecase is that it can treat different types of tables and data frames the same, if for some reason you're working with a mix of arrow tables and pandas data frames for instance.
bunchedupwalrus@reddit
The only benefit I can think of was mentioned above, syntax is SQL, though I’d be interested too
eratis_a@reddit
Seconded. DuckDB is really good one & usually performant too.
denvercococolorado@reddit
Related project: ibis. Defaults to DuckDB, but it’s pluggable and you can use Spark, BigQuery, Flink, all kinds of things as the backend. It uses dataframes natively (if you prefer that syntax over SQL). Exports to Arrow and a few other formats. It’s very nice.
No_Mongoose6172@reddit
Is it able to work with libraries that currently just support pandas? Polars and duckdb are great, but then you find that they are incompatible with most python data analysis libraries. I have ended switching to Numpy memmaps, as they are compatible with anything that supports bumpy arrays
mailed@reddit
I hope the project stays alive. Voltron Data was the driving force behind Ibis and they laid off a ton of people earlier in the year
busybody124@reddit
That's the tip of the iceberg. I love using it to run sql queries on directories or parquet files on S3.
ebmarhar@reddit
"iceberg", are you making a joke? :)
busybody124@reddit
lol I didn't mean to be but it can read iceberg catalogs too of course
yc_hk@reddit
I've tried using DuckDB as a database, but ran into problems with concurrency and just gave up and switched to Postgres.
As for data manipulation within Python, Pandas has served me well enough that I'm not in a hurry to switch. Siuba(https://siuba.org/) is interesting though, as it is inspired by R's dplyr and its pipe syntax.
deepstate_psyop@reddit
Same. Currently using this to allow LLMs to manipulate pandas data frames, since SQL is easier for LLMs to produce.
ebmarhar@reddit
That's a really smart idea! I'm going to try that.
G0muk@reddit
Thanks this one seems useful
psgpyc@reddit
I love Pathlib.
One_Programmer6315@reddit
Meee tooo
gnouf1@reddit
wonderful lib but not new
Jock_A_Mo@reddit
It’s new if you used Python 2 and the os module for file management
rabaraba@reddit
It’s arguably more “modern” than os though. And underused.
gnouf1@reddit
True, as I said it's a wonderful lib Honestly the standard lib is so underestimated
PeanutsUpbeats@reddit
uv for package management ruff for linting commitizen for making conventional commit messages and auto version bumping
DxNovaNT@reddit
FastAPI, SQLModel and Pydantic
HolidayEmphasis4345@reddit
Pandas/polars/narwals FastAPI/Typer Rich/Textual Streamlit
PurepointDog@reddit
NiceGUI over streamlit, depending on what the thing is.
I've build a few cool things in Streamlit. It's very fast to prototype with, bu its lack of deployment options means you have to rebuild the whole thing anyway. NiceGUI is a bit slower to build with, but once it's done, it's unlikely to need a full rebuild
turingincarnate@reddit
Can nicegui be deployed from github? Can you build mobile apps/web apps with it?
Afraid-Jelly2457@reddit
Flet
jon_muselee@reddit
really new, but already great!
PurepointDog@reddit
Yes
kamsen911@reddit
I prefer https://github.com/BrianPugh/cyclopts nowadays. It’s really the best and most intuitive CLI tool imho.
HolidayEmphasis4345@reddit
Just checked it out. Cyclopts looks nice. It is funny, I figured typer out once, and then reused the code as a template since most of my CLI apps are focused on doing one thing with a couple of options. This does look like some of the rough edges are gone. Will try it on next CLI app. Thx
Cod3Blaze@reddit
amen-cli
python web app scaffolding tool with a bunch of other commands
yellowbean123@reddit
toolz / lens / more-itertools
Top-Waltz-4191@reddit
I like pyright and pyupgrade
StandardIntern4169@reddit
uv, ruff, pydantic
echols021@reddit
uv, ruff, pydantic, FastAPI. Poethepoet is alright. Hoping for ty to be usable soon
MackHarington@reddit
What do you guys think about dearpyguiDearPyGUI
unnamed_one1@reddit
Tooling? Everthing Astral
UnicornTooots@reddit
Astral is changing the game for Python in all the right ways. uv and ruff as awesome. Looking forward to ty.
vinnypotsandpans@reddit
Came here to say this.
yc_hk@reddit
I still use pylint since it seems to check for more things -- this ruff issue tracks parity with pylint and there seems to be quite a way to go.
Skewjo@reddit
If you're reading this, don't just blindly
pip install Astral
like I did unless you need a library for calculating the positions of heavenly bodies.unnamed_one1@reddit
duck I might have better referred to uv, ruff, ty - sorry, my bad
Skewjo@reddit
All good, it was my mistake! Just thought it was funny.
PurepointDog@reddit
ty soon I hope
Mustard_Dimension@reddit
ty definitely needs some more time in the kitchen, I've been using the pre-release and it crashes a lot. Although it's good when it's working!
PurepointDog@reddit
Neat! I haven't tried it yet; happy to hear it's nearly usable!
jms_nh@reddit
enaml for easy GUIs
Th3Stryd3r@reddit
I'm still even too new to python to know what the tooling is for >< Can someone ELI5?
mardiros@reddit
The tooling is all the stuff that you are using to build your python projects.
For example, you can use uv or poetry to manage your packaging. You can use ruff or black to format your code. You can use mypy to validate your typing. You can asi use linter, such as flake8, pylint or ruff.
Astral has built game changer in that domain. ruff and uv.
They will release soon a type checker, i's in beta. People here give the name. And Facebook released one recently.
ruff and uv are mostly written in rust and this is part of the reason they are really fast.
You usually declare your tooling as dev dependencies library.
So when you run your program, you don't need them. And end users that install your program do not install them while running the application or library.
Th3Stryd3r@reddit
<3 I truly appreciate all the info.
On that note if I could ask a question. My boss had mention making sure the code was obfuscated from any prying end users wanting to poke around. Obviously we'll likely lock any system they are running on down anyways, but how would one go about doing that?
mardiros@reddit
A code obfuscator is a tool used in the toolchain to rename variables, functions, and more. I never used this kind of tool so I can't say. I am not sure of the result of this approach. Alternative is client server, where the code of the server is never exposed to the client, but it requires an internet connection. The code of the client leak but the algorithm staus on the serve. An other alternative is to code in a compiled language. The code don't leak since it is compiled. pyo3 with maturin for rust for example. The downside here is the complexity of the architecture increase a lot. I don't think that using rust to hide the code is a good argument. You may code in typescript and transpile to javascript. But this is not python anymore.
syklemil@reddit
In addition to
uv
you'll likely also want to try outruff
: It's both a formatter (see alsoblack
) and a linter (see alsopylint
,flake8
). The formatter should help you get fairly "normal-looking" code; the linter has a bunch of lints you can selectively enable to avoid some common mistakes (like doing requests without a timeout).pyright
is the middle-of-the-road-ian option: There are some newer options, there are some older options. E.g. some preferbasedpyright
(not in my distro's default repos), some are still usingmypy
. Likelyty
orpyrefly
will move into the default spot once they're stable.Typechecking in Python can be kinda noisy and depressing depending on the libraries you're using, more so if they're old.
that_baddest_dude@reddit
If you're very new to python you're unlikely to seriously need much of what's talked about here... Except for maybe uv.
I'd look at using uv to manage any python installations or packages. Doing so will also set you up for success if your skills expand.
Th3Stryd3r@reddit
I figured I didn't need most of it now, but good to know. The most advanced thing I've made so far is a script that auto process PDF files and removes blank back pages from PDFs and then if it sees a full blank page, back and front, it breaks the file into multiple parts. Law firm client of ours is using it to mass process mail and give it to their case managers. I think I'm just getting started though :p
that_baddest_dude@reddit
Nice!
uv would help you note all the dependencies and such for that you used for the script, and keep versioning consistent so that it would work if you uninstalled everything and wanted to reinstall and run it again.
And then it could help manage separate new projects that don't require the same dependencies, so you can keep your requirements slim
Th3Stryd3r@reddit
That sounds handy. This is just one automation I put in place, I know this client is likely going to need more so that was my next learning journey. How to run and check on multiple active scripts running all from one machine.
I wish I could just throw them into n8n and save me some time. But they deal with a lot of HIPPA so that's a no go from the security side of things.
Mr_Canard@reddit
Personally for projects like that I like making a webUI to monitor the status of everything in one place (kinda inspired by what wooey tried to do for example).
stibbons_@reddit
Rich, click, pydantic, boltons.
Uv, ruff for tools
mardiros@reddit
pyo3 and maturin.
They are used by pydantic 2, uv and more.
I built few library with it, such as lastuuid and envsub.
Electrical-Split7030@reddit
i love uv
wineblood@reddit
I hate uv
The_Real_Cooper@reddit
Duality of man
j_hermann@reddit
mkdocs over Sphinx by now.
yc_hk@reddit
What do you use as a replacement for Sphinx's autodoc and apidoc extensions?
Rockhopper_Penguin@reddit
mkdocstrings
has worked great for me, although it took a bit of time to get comfortable with it. Here's a sample documentation page for one of my projects.Disclaimer, I've never used sphinx before so idk how mkdocs/mkdocstrings compares.
Good luck!
mardiros@reddit
I heard that a lot. I tried mkdocstring but I had weird results on how it behaves on documenting types and interprets
__all__
. So, I go back to Sphinx, myst_parser, furo and autodoc2.mkdocs is more modern, I will give a other try later. I thought it will be simpler.
j_hermann@reddit
See examples like https://docs.pydantic.dev/dev/api/base_model/
yc_hk@reddit
Looks good, but how was it generated? Personally, I'm looking for a plugin that will automatically generate doc pages from docstrings.
fizix00@reddit
typer over argparse. plotly for interactive notebook visualizations. ruff+precommit
scrdest@reddit
Invoke is a neat, Python-ey replacement for Make - especially handy for 'maintenance toolbox' or CI/CD scripts.
rawrgulmuffins@reddit
I make and maintain a ton of makefiles so I'm instantly interested.
Rockhopper_Penguin@reddit
I'd recommend
just
as an alternative tomake
(I used to use a ton of makefiles too lol).Here's a sample justfile (analogous to a makefile) for one of my Python projects. When you run
just
(I alias toj
), you get the following preview: https://files.catbox.moe/8qo8hi.pngI use this in a ton of other projects as well, it's not just limited to Python.
busybody124@reddit
We use invoke for a project at work. I think it's 90% of the way there but does seem fairly abandoned. I wish you could have a task's arguments optionally passed to its parent tasks.
guhcampos@reddit
I've been almost exclusively using pre-commit for CI/CD lately
CableConfident9280@reddit
I’ve been really liking just for task running
tuukkamustonen@reddit
I liked it too. Unfortunately, it's been unmaintained for a few years.
notkairyssdal@reddit
rich for console output
CoffeeSmoker@reddit
And textual for cli applications
MinuteMeringue6305@reddit
I use it in jupyter
SubstanceSerious8843@reddit
Pydantic all the way
DoingItForEli@reddit
I just found out a few weeks ago that FastAPI can generate documentation for your endpoint automatically if you're using Pydantic. Very cool integration.
frankwiles@reddit
It is cool but also built into other things like Django-ninja which is also Pydantic as an alternative FYI. This also existed for the older Django rest framework in a few forms but having Pydantic in the mix definitely makes the API far more accurate.
DoingItForEli@reddit
thank for the tip!
SubstanceSerious8843@reddit
Tiangolo is a wizard.
SubstanceSerious8843@reddit
I'll second to this. Insanly good stuff.
KimPeek@reddit
Forgot to switch acconts?
SubstanceSerious8843@reddit
Haha :D honest dumb fckery by me. Didn't notice I said that xD
kjerk@reddit
Vastly too often the "Why doesn't this want to even start? It looks fine, it's a fresh venv." clusterf*** has been a pydantic dependency silently-shitting-things-up problem in my experience. Inevitably where the only fix is to
pip install --upgrade pydantic==1.10.MySpecificVersion
which hopefully someone has commented on a github issue because good luck finding which specific dependency trickled down the diarrhea rain.It's been a constant poison pill.
SubstanceSerious8843@reddit
wait.. you don't lock your dependencies with dep == 1.2.3 ? or atleast to major updates. Minors and patches should be backwards compatible.
kjerk@reddit
This is not my projects, this is at least 15 separate github repos over the past 12 months from LLM inference to huggingface demos. Some have plugin systems and the plugins separately have a mutually exclusive pydantic version and nobody bothered to figure it out. Some have rolling releases where a developer bumps a version of a dependency that they singularly updated on their local which now collides with their own requirements.txt.
This spiderwebs out from Gradio quite often where they've had a specific (ish) version of pydantic pinned, a new version comes out and many different downstream repos suddenly break even on a minor version number change.
This is an opinion from depth of experience.
SubstanceSerious8843@reddit
Yeah sry, no I get what you meant.
covmatty1@reddit
Been using Pydantic across at least 10 work projects for well over 18 months now and never run into evening like this.
Are you not pinning dependencies?
covmatty1@reddit
Hugely agree, it's such a key part of my team's workflow and having stronger integration. Making Pydantic models libraries is just a standard part of any project's development process for us now.
guyfrom7up@reddit
If you find pydantic to be too "magical" or simply want an alternative option, I have also had good success with attrs + cattrs. In a nutshell:
attrs
has been around for a long time and was one of the original origins of builtin dataclasses. Builtin dataclasses are basically a stripped down version of attrs.bdaene@reddit
I would argue that attrs is more an alternative (extension) to dataclasses than pydantic. I have application with pydantic for user data validation and attrs for internal dataclasses.
JimDabell@reddit
That’s why they said attrs + cattrs. cattrs does the serialisation / validation part.
CSI_Tech_Dept@reddit
You're correct. In fact dataclasses were inspired by attrs.
SubstanceSerious8843@reddit
Will check out! Thanks!
MeroLegend4@reddit
+1 to attrs/cattrs
daemonoakz@reddit
THIS... I mean, self
These-Bet-6238@reddit
Pandas
TheMcSebi@reddit
plotly as matplotlib replacement
Even_Raisin_6516@reddit
Marimo has been a game changer. Built in data viewer, no more code in JSON, reactive development, AI integration, SQL Integration. Rarely use Jupyter anymore
thearn4@reddit
I kind of hated poetry for awhile but came around to it. Ruff is the preferred linter these days.
bunchedupwalrus@reddit
I avoided poetry just long enough for uv to win my heart lol. Never felt like I got the hang of it
pancomputationalist@reddit
You should try uv, much faster than poetry
Due_Shine_7199@reddit
pydantic uv ruff pyrefly (ty looks promising) fastapi
kamsen911@reddit
Pyrefly‘s LSP is killing my machine (pycharm), had to disable it.
DowntownSinger_@reddit
how did you get it working with Pycharm?
kamsen911@reddit
There is a beta plugin available.
patrickkidger@reddit
If self-promotion is allowed then here are a couple of my big ML ones:
(For the curious, here are the rest of my libraries.)
Airrows@reddit
Pandas
Xchange-maker@reddit
I'm just starting out with coding with python, i want recomendation on some materials or courses that can help me level up ?
MinuteMeringue6305@reddit
I am a classic user. I prefer Django over fastapi, drf serislizers over pydantic (not all the time, tho), I just use venv to manage virtual environments, still use pandas over polars, rely on pycharm on type checking.. Am I doomed?
Meanwhile everyone rewriting their tools on rust
ThatSituation9908@reddit
Replacing pip with uv alone is already big. Not having to think about venv is another great perk.
Mr_Canard@reddit
The modules you cited are still actively maintained so you'll be fine, you should probably try out a few of the ones that were cited though even if you don't end up using them. The hype around Astral's UV isn't undeserved but their type checking module TY isn't producing ready yet.
dhaitz@reddit
Modern data science stack: uv + polars + marimo
For application development: FastAPI, ruff, creosote, pip-audit
Mr_Canard@reddit
Outside of the ones already cited:
Whenever for dealing with timezones Django-Ninja for a FastApi inspired experience in Django Connectorx to load SQL data fast (polars uses it under the hood)
No_Pomegranate7508@reddit
Poetry
GNU Make
oculusshift@reddit
try uv over Poetry, the speed and features outshines Poetry.
You’ll love using it.
No_Pomegranate7508@reddit
The main benefit of uv at the moment is its speed, but Poetry is more mature and has more features and plugins. I understand people like newer things made in Rust, but Poetry works just fine for my projects.
badass87@reddit
Msgspec
ml_adrin@reddit
I don’t know about love but i hate langgraph
bunoso@reddit
Pydantic UV RUFF and Typer
symnn@reddit
Litestar for API development.
p_bzn@reddit
Whats is the difference between it and say FastAPI? Say, why it is needed in the world where we have other solutions like Flask / FastAPI? Curiosity.
lacifuri@reddit
From a quick google search it seems Litestar doesn’t depend on Starlette, which I am not sure is a good thing or not.
p_bzn@reddit
Brief search showed that they have minor differences in ergonomics and features (eg controller like routing, that is routes within class). Standard set of features for this kind of libraries regardless of language. Performance might be a touch better, if anyone would use Python for web server performance that is.
Hence the question, wonder what I’m missing.
bradlucky@reddit
I am so in love with Litestar! I am pretty new to it, but it is so amazing to me.
suedepaid@reddit
I really tried to like Litestar, but found myself going back to FastAPI
ydmatos@reddit
Why?
MeroLegend4@reddit
+10 👆
wunderspud7575@reddit
Litestar feels a bit better structure than FastAPI.
Desperate-Brick-1191@reddit
NiceGUI for easy web based APIs and dashboards
HolidayEmphasis4345@reddit
It depends on your use case. You can tell from my web library that my web stuff sits behind a corporate firewall. That’s why there is no flask, Django. I manage a server that faces 100ish users and streamlit is great for that use case. I don’t know about real web stuff.
I left of databases. I’m forced into sql server, so I use pyodbc, which I’m fine with since writing raw sql is good enough.
That spectrum goes Jupyter, streamlit, flask, Django…. I don’t know about NiceGUI.
greenknight@reddit
I've been replacing my date times with whenever objects. I work in a No DST zone and have had many runtime utc<->tz collisions.
Getting ready to switch my pandas workflow to Polars
LEAVER2000@reddit
I’ve been using a pydantic to handle tzaware user inputs
UTCNaiveDatetime = Annotated[datetime.datetime, AfterValidator(lambda x: x.tz_convert(“UTC”).tz_localize(None))]
greenknight@reddit
If this thread is any indication, I'll be putting pydantic on my list of "to grok".
antediluvium@reddit
I’ve really liked arrow for date times, but haven’t tried whenever. I’d be interested if you’ve tried both
greenknight@reddit
I haven't. Kicked the tires of pendulum a bit but the Instant-ZoneDateTime-PlainDateTime paradigm in whenever really resonated with me so I never made it to evaluating Arrow.
roryhr@reddit
I use black for formatting and that’s the newest thing I can think of.
oculusshift@reddit
Try ruff
justsayno_to_biggovt@reddit
Polars
chazzeromus@reddit
mypy!
SignificantManner197@reddit
tqdm, nltk, spaCy.
utkarssh2604@reddit
fastapi, pydantic, uv, ruff, agno(ai_agent/llm)
Such-Let974@reddit
Since most people are just glazing Astral in this post, I'll suggest something that isn't one their inevitable rug pull products.
I've really been enjoying Marimo. It took some time to get my brain to switch over all my muscle memory from Jupyter Notebooks but it is worth it for situations where you want really nicely designed and presentable notebooks. It's also really nice to be able to manage and even run things as regular python files rather than the sort of custom jupyter style json.
guhcampos@reddit
Came to say Marino, but I don't think they're necessarily going to rug pull.
suedepaid@reddit
how you gonna call Astral a rug pull but stan marimo
Such-Let974@reddit
Easily. I just thought about it and realized it applies to one and not the other.
EarthGoddessDude@reddit
I wouldn’t call it actively hiding their monetization strategy. Charlie Marsh has stated over and over in various podcasts that the aim so create some sort of Artifactory contender that works really well with their FOSS tools. Given how well their FOSS tools work and how well liked they are, people will probably be lining around the corner to pay for their paid product. In other words, uv and ruff (and ty) are a very effective advertising strategy.
I realize that’s a very rosy outlook, and it by no means precludes any sort of rug pull, but I very much hope that that is how it will play out.
marr75@reddit
Ibis project
Dataframe interface that can use swappable compute engines. Pandas, polars, duckdb, just about any popular RDMBS or OLAP engine. Lazy-evaluated. Can get a pandas dataframe, polars dataframe, or sql expression any time you feel like it while debugging.
__s_v_@reddit
!Remindme 1 Week
mloning_@reddit
!Remindme 1 Week
kuzidaheathen@reddit
!Remindme 2 Weeks
RemindMeBot@reddit
I will be messaging you in 7 days on 2025-06-12 17:12:46 UTC to remind you of this link
CLICK THIS LINK to send a PM to also be reminded and to reduce spam.
^(Parent commenter can ) ^(delete this message to hide from others.)
--ps--@reddit
pybind11 for C++ integration
JudgeDefiant4252@reddit
!Remindme 1 week
JabootieeIsGroovy@reddit
hugging face transformers!!
Karose156@reddit
PyQt
78wesley@reddit
FastHTML
Sufficient_Statistic@reddit
If you like polars, replace matplotlib with altair https://altair-viz.github.io
Yo_man_67@reddit
FastAPI, LiveKit, CrewAI and all of ai agents librairies
jollyjackjack@reddit
A few random packages I haven't seen mentioned: * msgspec as a faster version of pydantic * rich for pretty terminal output * deptry for finding issues with project dependencies * repo-review for linting project configuration (very pluggable if you have team specific setups)
Taltalonix@reddit
Poetry, pydantic and anything async
wineblood@reddit
Does precommit count as new? I actually can't think of anything recent that has really impressed me, most of the things I've tried have been disappointing.
ikbennergens@reddit
Despite using `uv` only for short while, I came to the conclusion that `uv` is the fastest. For context, I came from a background of using only `conda` (the slow one), then moved on to using `mamba`, a `conda` drop-in replacement written in C++. For me, I'm more comfortable with the `conda` style of doing things, so I'm sticking with `mamba` a bit longer before moving on to `uv`.
MeroLegend4@reddit
Same here, moved from conda to mamba never looked back
Drewdledoo@reddit
IMO the only benefit Conda/mamba have over uv is that they can install non-python software, so if you are only using python libraries, uv and the like are the way to go.
That said, if you do need non-python stuff, you should give pixi a look!
MeroLegend4@reddit
Litestar Polars Pointblank Msgspec Ruff
turingincarnate@reddit
mlsynth
Bilbottom@reddit
Rust and maturin
/s
Spleeeee@reddit
What have you made?
ebits21@reddit
Uv and ruff. They’re great!
hurhurdedur@reddit
Love Polars and Ibis as a replacement for Pandas. uv is awesome.
ZpSky@reddit
Fastapi, pydantic, ruff, async
DadAndDominant@reddit
I love anything Astral pushes out
Deep_conv@reddit
You're probably looking for sth you overlooked, but the answer for me is uv and polars