Saving Memory with Polars (over Pandas)

Posted by paltman94@reddit | Python | View on Reddit | 37 comments

You can save some memory by moving to Polars from Pandas but watch out for a subtle difference in the quantile's different default interpolation methods.

Are there any other major differences between Polars and Pandas that could sneak up on you like this?

[-]

Jcc_56@reddit

Finance guy here.

I’m working in a PnL transfo environment and I have to do shadow IT, one of my projects being deploying an « explain » for the accrued interests daily pnl with fairly large datasets with streamlit lib.

When I tried doing it with Pandas I quickly faced performance issues and time outs (sorry for the terminology guys, not from IT background :) ).

Then I switched to Polars. Pretty harsh move at the beginning, everything seemed unfamiliar but then it started making more and more sense and now I really like the syntax. Anyway, I moved my project to Polars using lazy evaluation and pipes and the difference is overwhelming. Polars did with ease what I couldn’t do before with Pandas. Ofc it forced me to better cast types and rethink my approach but still, the performance gap is incredible and now we only use Polars in my team.

[-]

spookytomtom@reddit

Already ditched pandas. The polar bear is my new spirit animal

[-]

UltraPoci@reddit

I can't wait to do the same, but I need geopolars first :(

[-]

NostraDavid@reddit

.to_pandas() is your friend.

[-]

UltraPoci@reddit

95% of my use of Geopandas is for operations on geospatial vectors. I'd be using polars just to read and write files, basically

[-]

NostraDavid@reddit

The loading will then get a speedup :P

Especially if you load .parquet files, but even with .csv you can ~10x the loading speed.

[-]

UltraPoci@reddit

That's nice I guess, but I think it won't make much of a difference in my case. I'm interested in polars mainly for the API. I'm also looking into duckdb, it looks nice and supports geospatial applications

[-]

PandaJunk@reddit

You can easily just convert between the two when you need to. They work pretty well together, meaning it is not a binary -- you can use both in your pipelines.

[-]

EarthGoddessDude@reddit

Hell yea brother. Don’t forget the duck as well.

[-]

spookytomtom@reddit

Yeah readin a book on it atm

[-]

Heco1331@reddit

I haven't used polars much yet, bur from what I've seen the largest advantage for those that work with a lot of data (like me) is that you can write your pipeline (add these 2 columns, multiply by 5, etc) and then stream your data through it.

This means that unlike Pandas, which will try to load all the data into a dataframe with its consequent use of memory, Polars will only load the data in batches and present you with the final result.

[-]

roenthomas@reddit

Lazyframes?

[-]

Heco1331@reddit

I don't know what you mean by that, so I think the answer is no :)

[-]

NostraDavid@reddit

When you have a DataFrame, and run .filter(...), it'll immediately return a new DataFrame, whereas if you have a LazyFrame, it'll return an optimized plan (it's just another LazyFrame). If you want your data you must run .collect(). Why? Because you can write your manipulations however you want, and Polars can apply optimizations (maybe remove some duplicate sort, or combine overlapping filters, etc), generating optimized manipulations making your code even faster.

It's eager (run everything one after another, in-order-of-written-code) vs lazy (only run the optimized query once).

[-]

sayhisam1@reddit

This

I processed a terabyte of data in Polaris with little to no issues. Pandas couldn't event load the data into memory.

[-]

sheevum@reddit

that and the API actually makes sense!

[-]

Doomtrain86@reddit

I was baffled when I moved from data.table in R to pandas. Is this really what you use here?! It was like a horror movie. Then I found polars. Now I get it.

[-]

AlpacaDC@reddit

And it’s very very fast

[-]

Optimal-Procedure885@reddit

Very much so. I do a lot of data wrangling where a few million datapoints need to be processed at a time and the speed with which it gets the job done astounds me.

[-]

GriziGOAT@reddit

That depends on two separate features you need to explicitly opt into 1. LazyFrames - you build up a set of transformations by doing e.g. df.with_columns(…).group_by(…).(…).collect(). The transformations will not run until you call .collect(). This allows you to build up these transformations step by step but defer the execution until the full transformation is created. Doing this will allow polars to more cleverly execute the transformations. Oftentimes saving lots of memory and/or CPU. 2. Streaming mode - I haven’t used this very much but is useful to do an even more efficient query plan where it will intelligently only load the data it needs into memory at any point in time, and can process the data frame in chunks. As far as I know you need to do lazy in order to be allowed to do streaming.

[-]

DueAnalysis2@reddit

In addition to that, there's a query solver that tries to optimise your pipeline, so the lazy API has an additional level of efficiency.

[-]

twenty-fourth-time-b@reddit

TL/DR: nearest vs linear

[-]

Hot_Interest_4915@reddit

polars unbeatable

[-]

MolonLabe76@reddit

I want to switch over so bad. But until they make/finish GeoPolars, which is blocked because Polars doesnt/wont support Arrow Extension Types, additionally Polars does not support subclassing of core data types. Long story short, id love to switch, but my main use case is not possible.

[-]

nightcracker@reddit

because Polars doesnt/wont support Arrow Extension Types

Definitely a "doesn't", and "right now". I'm working on adding Arrow extension types.

[-]

UltraPoci@reddit

Can you link a PR or any other source so that I can keep myself updated? I'm also interested in geopolars

[-]

nightcracker@reddit

https://github.com/pola-rs/polars/issues/9112

[-]

zeya07@reddit

I fell in love with polars expressions and super fast import times.I tried using it in scientific computing, but sadly polars does not natively support complex numbers, and a lot of operations would require to_numpy and back. I hope in a while there will be native polars libraries similar to scipy and sklearn.

[-]

True_Bus7501@reddit

I didn't like Polars, DuckDB is better.

[-]

Interesting-Frame190@reddit

I started building PyThermite to compete with pandas in a more OOP way. While benchmarking against pandas, I decided to run against Polars. Its also a Rust backed threaded (rayon) tool, so i thought it would be a fair fight. Polars absolutely obliterated pandas in loading and filtering large datasets. 10M+ rows. Id say querying a dataset couldn't get much more performant unless its indexed.

[-]

klatzicus@reddit

The expression optimization (changing expression order to optimize performance using the lazy api) has given me trouble. Eg. a delete column was moved to occur before an expression manipulating said column). This was a few builds ago though.

Also compressed files are read into memory and not streamed (compressed text file read with the sink_cab or read_csv operation)

[-]

BelottoBR@reddit

I loved from pandas to polars and the performance is amazing. I am used to deal with lazy evaluation (I was using dask to deal with bigger than memory dataframes )

[-]

Secure-Hornet7304@reddit

I don't have much experience using Pandas, but I have already encountered this memory problem when the dataframe is very large. At first I thought that it was my way of implementing the project with Pandas that made it consume so much ram and be slow (I was working on a csv without parquet quet or anything), but it makes sense if pandas loads the entire dataframe into ram and data manipulation becomes an issue of resources rather than strategies.

I'll try to replace everything with Polar and measure the times and resources, see how it goes.

[-]

paltman94@reddit (OP)

Yes. https://pola.rs/