It is optimized for analytic workloads (default in-mem columnar layout), whereas PG and sqlite are OLTP DBs. It's going to be insanely faster for those use cases.
Actual databases pay up to two orders of magnitude of speed for durability. If you can regenerate your dataset in case the DB drops it (or there is a power outage, or whatever), being able to complete a complex, write-heavy query in 10s instead of 15 minutes is actually very useful when doing analysis.
I'm not following the trends closely, but has Polars become a full replacement for Pandas? Are there use cases where one is better suited than the other?
That's a shame, because Pandas has a really quirky/legacy-burdened API and Polars is super clean by comparison. As someone who had Spark and Pandas experience before switching, Polars felt like Pyspark without the added mental overhead of needing you to think about multi-worker-node parallelism
Polars is effectively a full replacement for Pandas for 99.9% of all cases. The only exception I'm really aware of is if you're working with geospatial data, as there isn't yet a "Geopolars" equivalent of the commonly used "Geopandas". However, Geopolars is still in active development and should eventually be production ready.
Imitation is the sincerest form of flattery! GeoPandas — and its underlying libraries of shapely and GEOS — is an incredible production-ready tool.
GeoPolars is nowhere near the functionality or stability of GeoPandas, but competition is good and, due to its pure-Rust core, GeoPolars will be much easier to use in WebAssembly.
This is awesome!! I'd looked at the project only a month or so and it appeared abandoned, but I must have missed the off-main-branch development going on!
There is a geospatial package for duckdb though which is pretty slick. It is actually how I first learned of duckdb. We were dealing with nationwide parcel datasets and need to apply transformations nationwide and save out to more parquet files. It was easier and cheaper to replace all of the pandas workloads with duckdb.
Pandas is better for slight in-place or per-row modifications, for loading from less conventional datasets, for transposition/more nuanced row-based aggregation/multi-axis manipulation, for performance when multiprocessing can be used, for interoperability with other libraries (e.g. plotting and statistics)... When you need to operate on huge datasets, use DuckDB, because its performance is even now on-par with Polars regarding speed, while handling huge or more complex joins is a huge win for DuckDB because it better offloads intermediate results to disk, while Polars just dies on me. I haven't tried such joins in Polars 2 though
Yes and no, its not replacing the reason why pandas was popular ie data scientists, but it a full replacement of its pipeline usage, And I would saw also beating out spark
my understanding is Polars is faster, scales better without using external solutions, better API, +Rust. Pandas wins if you want to use what the vast majority of folks are using and have used in the past. Probably has a more complete set of helpers / recipes for the little things you bump into when using it thoroughly, but in the age of LLMs, I think that's minor.
>Pandas wins if you want to use what the vast majority of folks are using
Vast majority of skilled developers are now using Polars, unless they are constrained by lack of Narwhals support in their third-party library of choice (e.g. Great Expectations, SHAP). That's the more important trend to follow.
Learn SQL and interface with Duck. You will be 100x faster than Pandas/Polaris duo at fraction of memory. Also SQL is supported literally everywhere with a much more capable than Pandas API. Duck outputs to a Pandas Dataframe, but just treat that like a dictionary. Do all your processing, filtering and aggregation in Duck.
Also, try fx.wtf as a replacement for jq. it comes with a in-built tui viewer that supports vi-keybindings. Ecmascript is built into fx.wtf so you can query the JSON with JS notation (where JSON was born). You can use any JS functions including map/reduce/filter or perform any kind of transformation instead of learning jq dsl that you will forget tomorrow.
It kind of reminds me of Dungeons and Dragons, in a sense. Pandas is so popular that people learn/use it because of its popularity, rather than because it's the best for any one use-case. Polars is cleaner, faster, and can easily be scaled up for production use-cases so your little POC script for local analysis can be productionized really easily, but there are holdouts still on Pandas because it was so complicated to learn with so many little extra rules to learn to avoid paper cuts that they feel like learning another data manipulation tool would be really hard.
DnD does the same thing: it's the most popular but its rules are this awkward hybrid of legacy cruft and some modern ideas, so learning it a huge effort, which means most people who play it aren't willing to try any other RPG systems even though most of them are dramatically easier to learn because they were built with a clean design from the ground-up.
It's the sunk-cost fallacy as applied to learning something complex, combined with something like the horn effect (inverse of the halo effect) making any competitors look equally complex even if they're not, causing long-time Pandas users/DnD players to strongly resist even looking at other options. Basically the frustration of learning these older systems seemingly traumatizes some people into never straying.
There are awkward things. For example, if you ingest a nanosecond resolution timestamp, there's no way to re-export that out of the Polars dataframe with nanosecond resolution.
I'm not sure this is true e.g. you can specify schema `pl.Datetime("ns")`. That survives roundtrip to/from parquet in my experience. True though that the default is `us` whereas pandas defaults to `ns`.
From experience I'll say one thing that Polars doesn't have great interfacing with is doing things like string concatenation, like taking multiple columns and combining them in with static string text in complex ways to create new columns.
Pandas has a really simple ability to just define a new column with
`df['col_a'] + "text" + df[col_b']` where "text" can be any string text inbetween your column values from col_a and col_b
If i remember correctly while you can do pl.col("col_a") + pl.("col_b") for plain concatenation, you can't mix in static text strings like you can with pandas and I haven't found really elegant ways to do that personally. Whereas I've found polars doesn't have as simple of a way to do that. You can choose to add one separator and make that anything you want, but only one separator and only inbetween the two values (so no suffixes or prefixes for example).
That being said, I hate everything to do with the pandas API (especially with its indexing system) and really prefer the more polars API for anyone coming from a SQL or database background. Pandas really shows its sort of academia background rather than a data engineering origin.
I'm not sure what you tried. Is this not what you want?
>>> df = pl.DataFrame({"x": ["a", "b", "c"], "y": ["d", "e", "f"]})
>>> df.with_columns(new=pl.col.x + " text " + pl.col.y)
shape: (3, 3)
┌─────┬─────┬──────────┐
│ x ┆ y ┆ new │
│ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str │
╞═════╪═════╪══════════╡
│ a ┆ d ┆ a text d │
│ b ┆ e ┆ b text e │
│ c ┆ f ┆ c text f │
└─────┴─────┴──────────┘
Everything that I build greenfield moving forward I plan to use DuckDB, Polars, or PyArrow. Pandas was a great grandfather of a project (I actually cut my OSS contrib teeth on it, how the time flies)! I'll always appreciate the improvement pandas brought over SAS.
Agreed. For its time, R was a lot of fun to play with data pipelines, visuals, Quarto, and frontier stats. It's a shame it's so hard to make it work for a large swath of production use cases.
Do you have a take on when each of these three choices is the best one? I totally agree that these are the good choices, but I still find myself hesitating about which thing to reach for when!
Depends on need. We started using PyArrow on a reporting microservice when we realized we needed no additional functionality that pandas provided since it has better data type ergonomics. DuckDb is a great go-to for SQL based transformations when working with parquet files outside a managed system like Databricks. I want to actually test duckdb versus polars with a few lower level places like iceberg on S3.
Yeah makes sense. The SQL thing has also been my differentiator (and yeah, straight up arrow if you aren't doing much transformation of the data), but now I'm curious whether polars sql might be just as good. I kind of like that duckdb allows me to work with a database file, like a sqlite db. But maybe persisting parquet (or arrow directly?) is just as good?
This is why I asked someone else who is also figuring this out!
I think the person you're replying to is more so saying that by choosing the right queries, machine, disk setup, caching, settings, thread count, RAM amount, etc, you can get quite different results. There is a reason everyone always wins their own benchmarks, and it doesn't even have to be dishonest - you optimize and iterate for your own benchmark whereas everyone else just gets one shot to do well out of the box.
I tried my best to be as transparent and fair as possible, running everyone with out-of-the-box settings on a third party's queries (DuckDB), a third party's data generator (tpcgen-rs, from the DataFusion guys), on a stock setup available to everyone (AWS machines).
The one exception is that we also ran Polars locked to 32 threads on the large machine (in addition to the out-of-the-box setup), which was to highlight we can do a lot better on small data. We still suffer from a relatively high constant overhead on very high CPU count machines if the data isn't large enough, but I'm working on fixing that. It's possible that DuckDB / DataFusion have similar scaling issues with high thread counts and would do better with 32 threads as well, I didn't test that.
> choosing the right queries, machine, disk setup, caching, settings, thread count, RAM amount, etc, you can get quite different results.
I think it is over-complication. TPC results used to be reported on some standard machines you can order, and now every one uses AWS metal for that. Its Ok if project has configs tuned to popular machine, I think it is representative approach.
It is absolutely without a doubt 100% not diverse enough.
I've spent years of my life running TPC benchmarks. They're useful guidelines, but they're simply not diverse enough to show the whole picture unless your picture is very simple. Certainly no single TPC benchmark in isolation. Maybe you can get a better idea by running all of them.
The largest gaps are around non-relational style data, JSON querying and the like. Nearly every organisation has something in that space now, and TPC-H/TPC-DS don't touch it at all.
> unnesting json into relational data is some trivial op
I mean, everything is representative if you just phrase the bits you don't want to do as "some trivial op". By this logic, Clickbench is a great and totally representative benchmark because joins are also trivial ops.
Billions of dollars are spent processing semi-structured data. It's a critical part of most analytical pipelines. You can't just ignore it and pretend like your benchmark is representative.
A bit surprised about the datafusion results from the post, I have tried it time and time again, but datafusion has always been the leading/trading blowers with polars for our workfloads with duckdb being vastly slower.
Doing the benchmarks for 2.0 on the large AWS metal machines at small data sizes (SF=10) really opened my eyes that we have some low-hanging fruit in Polars when it comes to optimizing our constant overhead for smaller queries.
For example our join currently does a full partition into T partitions, for each of the T threads. Overall we create T^2 partitions, which on a 192-core machine is non-trivial. Great if you have a ton of data to feed that with, but if you 'only' have a few dozen million rows it becomes rather small. This is the primary reason we saw in the benchmarks that Polars pinned to 32 threads beats 192 thread Polars at SF=10.
I'll be working on improving that soon. I expect that to have a big impact on SF=10, and a decent impact on ClickBench, which sits between SF=10 and SF=100 in terms of rows.
Actually yes. We had already been planning to do 2.0 for a long time. We originally said we'd move on from 1.x quickly when released 1.0 but ended up staying at 1.x much longer than intended.
From a quick check our first PRs were merged to the 2.0 branch in June:
2026-06-17T21:27:51Z #27993 chore: Stop coercing `pl.col(...)` to selector ...
2026-06-18T14:19:26Z #27996 chore!: Replace multi-seed hash API with a single seed
2026-06-19T07:05:11Z #27991 chore(python!): Remove `Expr.flatten` function
I would be interested in seeing memory usage differences in these benchmarks. I’ve had issues with excessive memory usage in polars compared to DuckDB.
I’m guessing most of the this disparity should be solved by the steaming engine.
Polars relies on threading heavily, even when streaming files. And it appears that each thread loads quite a bit of memory. I've encountered OOM issues when incrementally reading Arrow IPC files which had very large batches. Fixed it by setting $POLARS_MAX_THREADS to 1, which amusingly also improved the performance on my very narrow task.
Thing I care about most is whether the old eager-vs-lazy footguns got cleaned up. Half my bugs were a stray collect() in a loop killing the query plan.
Although I find pandas a bit aggravating in many ways, for myself and my equally idiotic laboratory scientist pals, seems that it is the default way you might interface with other libraries like SciPy (i.e. they expect things as NumPy arrays or pandas dataframes). Is this a real issue or will most things happily accept a polars dataframe? We don’t work with such large datasets that speed is likely a huge concern tbh.
One nice development in this space is the narwhals library - it is a dataframe agnostic library. It allows you to seamlessly switch between pandas, polars, modlin, or any of the variations coming out.
Narwhal is still fairly new, but I expect its usage to spread since most packages only require rudimentary dataframe manipulation (set a value, math been these two columns, etc) where the limited api surface is not a problem.
Narwhals is also a much cheaper dependency to add than polars/pandas/etc so it is a somewhat easy sell to incorporate.
Not to take away any prop knowledge from you but I've been playing with some very very early ideas of getting some market data, save in duckdb do some analysis, create some kind of portfolio and buy/sell via API and nowhere close implementation. Yours seems rather established platform. Would you be able to share kind of high level architecture of bot?
I can use pandas to clean a dataset, but each cleaning task is usually one line of code. OTOH, With DuckDB with one SQL statement I can replace 40+ lines of polars/pandas. You may reply, SQL isn't as easy to understand! Fair point, it's a declarative language... which is why I use Malloy. Malloy is to TypeScript as Javascript is to SQL. Malloy is much easier to read and write (just as TypeScript is) because it has a built in semantic model -- all the joins, measures, and dimensions are done in one place.
Here is an example [1] of visualizing college football games. Here are all the queries, and semantic model that power all the visualizations [2] Here is the AI generated typescript/react that does the visualizations [3]. The Malloy ecosystem has Malloyyo and Publisher which are replacements for PowerBI and Tableau and Looker. Here is another example for visualizing global trade [4].
I've used Polars before and can only recommend it.
It's like you get a really good 'query planner' like a DB would give you, but for your notebooks/scripts/etc. Much better than pandas imo.
Why use Polars over just postgresql or sqlite?
It is optimized for analytic workloads (default in-mem columnar layout), whereas PG and sqlite are OLTP DBs. It's going to be insanely faster for those use cases.
And with DuckDB?
DuckDB is an in process OLAP, I’ve been using it a lot, and I am keen to use Polars but DuckDB seemings to be flying for me.
Actual databases pay up to two orders of magnitude of speed for durability. If you can regenerate your dataset in case the DB drops it (or there is a power outage, or whatever), being able to complete a complex, write-heavy query in 10s instead of 15 minutes is actually very useful when doing analysis.
Because those are databases? Polars is a data processing engine, not a database. They have different uses.
If you want to use a database instead of Polars, for similar use cases, duckdb is a much better bet than postgressql or sqlite.
I'm not following the trends closely, but has Polars become a full replacement for Pandas? Are there use cases where one is better suited than the other?
From recent Python Bytes podcast (https://pythonbytes.fm/episodes/show/496/a-lake-house-in-sea...)
> 1 Billion Row Challenge benchmark: Pandas took 4m28s vs. Polars 5.04s and DuckDB 5.19s — DuckDB also used 19x less memory
Python Vs Rust : In terms for speed - No comparison
(The above episode transcript has a link to blog post titled "Pandas should go extinct" )
I've nearly entirely switched to DuckDB for anything more than like 500 or 1,000 rows or if there are a tonne of columns.
Polars is great, but I'm just too used to the Pandas API to use it as a replacement for the cases where DuckDB is overkill.
That's a shame, because Pandas has a really quirky/legacy-burdened API and Polars is super clean by comparison. As someone who had Spark and Pandas experience before switching, Polars felt like Pyspark without the added mental overhead of needing you to think about multi-worker-node parallelism
It's just muscle memor, I've been using Pandas daily for over a decade, but I'll likely eventually switch over to Polars.
Part of the problem, as I said, is that I'm just spoiled by DuckDB when performance matters.
Polars is effectively a full replacement for Pandas for 99.9% of all cases. The only exception I'm really aware of is if you're working with geospatial data, as there isn't yet a "Geopolars" equivalent of the commonly used "Geopandas". However, Geopolars is still in active development and should eventually be production ready.
it’s on the correct path. i use rust for geo spatial and the gap with c, c++ closing rapidly or negligible in most cases
from https://github.com/pola-rs/geopolars/tree/main
Comparison with GeoPandas
Imitation is the sincerest form of flattery! GeoPandas — and its underlying libraries of shapely and GEOS — is an incredible production-ready tool.
GeoPolars is nowhere near the functionality or stability of GeoPandas, but competition is good and, due to its pure-Rust core, GeoPolars will be much easier to use in WebAssembly.
Note that (for the time being), we started geopolars developement under: https://github.com/pola-rs/geopolars/tree/dev
> Note that (for the time being), we started geopolars developement under: https://github.com/pola-rs/geopolars/tree/dev
This is awesome!! I'd looked at the project only a month or so and it appeared abandoned, but I must have missed the off-main-branch development going on!
There is a geospatial package for duckdb though which is pretty slick. It is actually how I first learned of duckdb. We were dealing with nationwide parcel datasets and need to apply transformations nationwide and save out to more parquet files. It was easier and cheaper to replace all of the pandas workloads with duckdb.
We were able to do a full port of GFQL from pandas to polars, cypher graph queries on dataframes, including both our CPU + GPU modes, and hit massive speedups: https://www.graphistry.com/blog/cypher-on-polars-cpu-gpu-gra...
It's been impressive!
Pandas is better for slight in-place or per-row modifications, for loading from less conventional datasets, for transposition/more nuanced row-based aggregation/multi-axis manipulation, for performance when multiprocessing can be used, for interoperability with other libraries (e.g. plotting and statistics)... When you need to operate on huge datasets, use DuckDB, because its performance is even now on-par with Polars regarding speed, while handling huge or more complex joins is a huge win for DuckDB because it better offloads intermediate results to disk, while Polars just dies on me. I haven't tried such joins in Polars 2 though
Yes and no, its not replacing the reason why pandas was popular ie data scientists, but it a full replacement of its pipeline usage, And I would saw also beating out spark
my understanding is Polars is faster, scales better without using external solutions, better API, +Rust. Pandas wins if you want to use what the vast majority of folks are using and have used in the past. Probably has a more complete set of helpers / recipes for the little things you bump into when using it thoroughly, but in the age of LLMs, I think that's minor.
>Pandas wins if you want to use what the vast majority of folks are using
Vast majority of skilled developers are now using Polars, unless they are constrained by lack of Narwhals support in their third-party library of choice (e.g. Great Expectations, SHAP). That's the more important trend to follow.
It’s the API that gave me the push to leave Pandas. 10 or more years of occasional Pandas use and I still had to google for any non-trivial queries.
In that regard, I’m still waiting for a credible jq replacement…
Replacement for jq: https://github.com/01mf02/jaq
Learn SQL and interface with Duck. You will be 100x faster than Pandas/Polaris duo at fraction of memory. Also SQL is supported literally everywhere with a much more capable than Pandas API. Duck outputs to a Pandas Dataframe, but just treat that like a dictionary. Do all your processing, filtering and aggregation in Duck.
Also, try fx.wtf as a replacement for jq. it comes with a in-built tui viewer that supports vi-keybindings. Ecmascript is built into fx.wtf so you can query the JSON with JS notation (where JSON was born). You can use any JS functions including map/reduce/filter or perform any kind of transformation instead of learning jq dsl that you will forget tomorrow.
Polars also supports SQL now, per the announcement we're all commenting on.
Avoiding pandas developers is a great reason to use polars imo
It has been for me. I greatly prefer the API, it fits my mental model much better. Give it a try!
See "Pandas should go extinct": https://news.ycombinator.com/item?id=49668198
tl;dr yes
It kind of reminds me of Dungeons and Dragons, in a sense. Pandas is so popular that people learn/use it because of its popularity, rather than because it's the best for any one use-case. Polars is cleaner, faster, and can easily be scaled up for production use-cases so your little POC script for local analysis can be productionized really easily, but there are holdouts still on Pandas because it was so complicated to learn with so many little extra rules to learn to avoid paper cuts that they feel like learning another data manipulation tool would be really hard.
DnD does the same thing: it's the most popular but its rules are this awkward hybrid of legacy cruft and some modern ideas, so learning it a huge effort, which means most people who play it aren't willing to try any other RPG systems even though most of them are dramatically easier to learn because they were built with a clean design from the ground-up.
It's the sunk-cost fallacy as applied to learning something complex, combined with something like the horn effect (inverse of the halo effect) making any competitors look equally complex even if they're not, causing long-time Pandas users/DnD players to strongly resist even looking at other options. Basically the frustration of learning these older systems seemingly traumatizes some people into never straying.
There are awkward things. For example, if you ingest a nanosecond resolution timestamp, there's no way to re-export that out of the Polars dataframe with nanosecond resolution.
I'm not sure this is true e.g. you can specify schema `pl.Datetime("ns")`. That survives roundtrip to/from parquet in my experience. True though that the default is `us` whereas pandas defaults to `ns`.
https://docs.pola.rs/api/python/stable/reference/expressions...
Feels like a feature to me. You could just keep the original timestamp as a string.
From experience I'll say one thing that Polars doesn't have great interfacing with is doing things like string concatenation, like taking multiple columns and combining them in with static string text in complex ways to create new columns.
Pandas has a really simple ability to just define a new column with
`df['col_a'] + "text" + df[col_b']` where "text" can be any string text inbetween your column values from col_a and col_b
If i remember correctly while you can do pl.col("col_a") + pl.("col_b") for plain concatenation, you can't mix in static text strings like you can with pandas and I haven't found really elegant ways to do that personally. Whereas I've found polars doesn't have as simple of a way to do that. You can choose to add one separator and make that anything you want, but only one separator and only inbetween the two values (so no suffixes or prefixes for example).
That being said, I hate everything to do with the pandas API (especially with its indexing system) and really prefer the more polars API for anyone coming from a SQL or database background. Pandas really shows its sort of academia background rather than a data engineering origin.
I'm not sure what you tried. Is this not what you want?
Well done, Polars team!
Everything that I build greenfield moving forward I plan to use DuckDB, Polars, or PyArrow. Pandas was a great grandfather of a project (I actually cut my OSS contrib teeth on it, how the time flies)! I'll always appreciate the improvement pandas brought over SAS.
If Pandas was the great grandfather, R data.frame is the great-great grandfather. R data.frames directly inspired Pandas.
Agreed. For its time, R was a lot of fun to play with data pipelines, visuals, Quarto, and frontier stats. It's a shame it's so hard to make it work for a large swath of production use cases.
Do you have a take on when each of these three choices is the best one? I totally agree that these are the good choices, but I still find myself hesitating about which thing to reach for when!
Depends on need. We started using PyArrow on a reporting microservice when we realized we needed no additional functionality that pandas provided since it has better data type ergonomics. DuckDb is a great go-to for SQL based transformations when working with parquet files outside a managed system like Databricks. I want to actually test duckdb versus polars with a few lower level places like iceberg on S3.
Yeah makes sense. The SQL thing has also been my differentiator (and yeah, straight up arrow if you aren't doing much transformation of the data), but now I'm curious whether polars sql might be just as good. I kind of like that duckdb allows me to work with a database file, like a sqlite db. But maybe persisting parquet (or arrow directly?) is just as good?
This is why I asked someone else who is also figuring this out!
Really comes down to query speed and management cognitive cost. I look forward to trying the new version for polars.
I worked on benchmarking in the past, including TPC benchmarks.
When you see a blog post like this, never interpret it as “database A is X% faster than database B”, there are just too many factors.
It’s more like “we put focused work into performance improvements and we expect certain workloads to perform better than the previous release”.
This seems like a good project, and benchmarking is a good way for a development team to iterate on performance. Just want to get my take out there.
But something like TPC is diverse enough to show whole picture?.. Especially compared to popular clickbench.
I think the person you're replying to is more so saying that by choosing the right queries, machine, disk setup, caching, settings, thread count, RAM amount, etc, you can get quite different results. There is a reason everyone always wins their own benchmarks, and it doesn't even have to be dishonest - you optimize and iterate for your own benchmark whereas everyone else just gets one shot to do well out of the box.
I tried my best to be as transparent and fair as possible, running everyone with out-of-the-box settings on a third party's queries (DuckDB), a third party's data generator (tpcgen-rs, from the DataFusion guys), on a stock setup available to everyone (AWS machines).
The one exception is that we also ran Polars locked to 32 threads on the large machine (in addition to the out-of-the-box setup), which was to highlight we can do a lot better on small data. We still suffer from a relatively high constant overhead on very high CPU count machines if the data isn't large enough, but I'm working on fixing that. It's possible that DuckDB / DataFusion have similar scaling issues with high thread counts and would do better with 32 threads as well, I didn't test that.
> choosing the right queries, machine, disk setup, caching, settings, thread count, RAM amount, etc, you can get quite different results.
I think it is over-complication. TPC results used to be reported on some standard machines you can order, and now every one uses AWS metal for that. Its Ok if project has configs tuned to popular machine, I think it is representative approach.
It is absolutely without a doubt 100% not diverse enough.
I've spent years of my life running TPC benchmarks. They're useful guidelines, but they're simply not diverse enough to show the whole picture unless your picture is very simple. Certainly no single TPC benchmark in isolation. Maybe you can get a better idea by running all of them.
The largest gaps are around non-relational style data, JSON querying and the like. Nearly every organisation has something in that space now, and TPC-H/TPC-DS don't touch it at all.
> It is absolutely without a doubt 100% not diverse enough.
sure, nothing is 100%. But 90% may be good enough.
> JSON querying and the like. Nearly every organisation has something in that space now, and TPC-H/TPC-DS don't touch it at all.
unnesting json into relational data is some trivial op, and then you come back into tpc-h/ds realm.
> unnesting json into relational data is some trivial op
I mean, everything is representative if you just phrase the bits you don't want to do as "some trivial op". By this logic, Clickbench is a great and totally representative benchmark because joins are also trivial ops.
Billions of dollars are spent processing semi-structured data. It's a critical part of most analytical pipelines. You can't just ignore it and pretend like your benchmark is representative.
No. Clickbench:
1. operates on very small dataset (100M rows) with very low cardinality
2. It has very limited ops, trivial group by and aggregation
TPC benchs are way more diverse.
> It's a critical part of most analytical pipelines. You can't just ignore it and pretend like your benchmark is representative.
It is not ignored. All major DB engines provide utilities to work with json integrated into relational paradigm. Not sure what is your point.
I use Polars 2.0(rc) to (pre)calculate billions of weather scores on https://therno.com and it has been a lifesaver
Happy that I can upgrade to 2.0 final tonight.
A bit surprised about the datafusion results from the post, I have tried it time and time again, but datafusion has always been the leading/trading blowers with polars for our workfloads with duckdb being vastly slower.
There are some benchmarks for the previous version https://benchmark.clickhouse.com/#system=+ti%20rud|Dusa,s|PD...
Doing the benchmarks for 2.0 on the large AWS metal machines at small data sizes (SF=10) really opened my eyes that we have some low-hanging fruit in Polars when it comes to optimizing our constant overhead for smaller queries.
For example our join currently does a full partition into T partitions, for each of the T threads. Overall we create T^2 partitions, which on a 192-core machine is non-trivial. Great if you have a ton of data to feed that with, but if you 'only' have a few dozen million rows it becomes rather small. This is the primary reason we saw in the benchmarks that Polars pinned to 32 threads beats 192 thread Polars at SF=10.
I'll be working on improving that soon. I expect that to have a big impact on SF=10, and a decent impact on ClickBench, which sits between SF=10 and SF=100 in terms of rows.
TIL that Polars supports SQL. Amazing.
At what point can we say that Polars is basically an in-memory database? (Genuine question)
I think it always has been? All of these data frame libraries are very much akin to olap databases.
It did since the first releases, but it was limited.
Now it seems they want to go head to head with DuckDB.
Love competition in the local data SQL space, makes everyone better
Does polars have a good equivalent of geopandas yet?
A coincidence with the fact duckdb is supposed to release 2.0.0 very soon? :)
Actually yes. We had already been planning to do 2.0 for a long time. We originally said we'd move on from 1.x quickly when released 1.0 but ended up staying at 1.x much longer than intended.
From a quick check our first PRs were merged to the 2.0 branch in June:
Are they competing for something?
> first class SQL support, which together with the performance improvements has Polars leading DataFusion and DuckDB in TPC-H and TPC-DS1 benchmarks,
Apparently about something ))
I would be interested in seeing memory usage differences in these benchmarks. I’ve had issues with excessive memory usage in polars compared to DuckDB.
I’m guessing most of the this disparity should be solved by the steaming engine.
Polars relies on threading heavily, even when streaming files. And it appears that each thread loads quite a bit of memory. I've encountered OOM issues when incrementally reading Arrow IPC files which had very large batches. Fixed it by setting $POLARS_MAX_THREADS to 1, which amusingly also improved the performance on my very narrow task.
Peak memory usage per query is in the raw data (in `results/`) in the repository: https://github.com/pola-rs/polars-2.0-benchmark/.
Looks like SQL does not support `asof_join()`, not sure what else it doesn't support.
Polars does have an asof join so I guess the mapping hasn't been made yet from SQL. Could you open up an issue at https://github.com/pola-rs/polars/issues ?
I use asof join frequently in polars! Happy to open the issue.
Thing I care about most is whether the old eager-vs-lazy footguns got cleaned up. Half my bugs were a stray collect() in a loop killing the query plan.
Been using polars for over a year now, it is fantastic.
How does Polars relate to DataFusion these days? There's no reason for them not to converge into a single ecosystem, is there?
It does not use datafusion. https://github.com/pola-rs/polars/issues/6197
Yes, but why?
I mean, it's very clear that the projects have a lot in common, even if building one on top of another directly is not a good idea.
In the issue, polars' creator says it is to reduce dependencies:
Looks like the Claude skill hasn't been updated
out of core sounds awesome! biggest thing that forced me to switch from pandas/polars to other solutions back in the day.
My team is all moving over to polars and DuckDB
What was your previous setup / stack?
amazing project !
Pandas is amazing, Polaris is amazinger!
what about daft project ?
Although I find pandas a bit aggravating in many ways, for myself and my equally idiotic laboratory scientist pals, seems that it is the default way you might interface with other libraries like SciPy (i.e. they expect things as NumPy arrays or pandas dataframes). Is this a real issue or will most things happily accept a polars dataframe? We don’t work with such large datasets that speed is likely a huge concern tbh.
One nice development in this space is the narwhals library - it is a dataframe agnostic library. It allows you to seamlessly switch between pandas, polars, modlin, or any of the variations coming out.
Narwhal is still fairly new, but I expect its usage to spread since most packages only require rudimentary dataframe manipulation (set a value, math been these two columns, etc) where the limited api surface is not a problem.
Narwhals is also a much cheaper dependency to add than polars/pandas/etc so it is a somewhat easy sell to incorporate.
You can convert to numpy using .to_numpy().
It's also got much better support for more complex array shapes (e.g. each row storing an array). At least it did last time I used pandas!
finally long time coming
cant wait to upgrade my Quant trading bot
Not to take away any prop knowledge from you but I've been playing with some very very early ideas of getting some market data, save in duckdb do some analysis, create some kind of portfolio and buy/sell via API and nowhere close implementation. Yours seems rather established platform. Would you be able to share kind of high level architecture of bot?
use a higher timeframe to understand the market phase, and a lower timeframe for entries.
when I started, I tried mimicking a human trader. if you want to expand into full quant, be aware of overengineering (past mistake of mine).
you can copy or mimic institutional desk strategies. most of the concepts are fine, but the devil is in the details.
sometimes you open too early or too late. finding that sweet spot, where you don’t want a lot of drawdown, requires a lot of fine-tuning.
I'm also using it for BlockRotate, it's time to upgrade.
I mainly trade xauusd and fx
more predictable
Spelling error in first paragraph. enthousiastic
I can use pandas to clean a dataset, but each cleaning task is usually one line of code. OTOH, With DuckDB with one SQL statement I can replace 40+ lines of polars/pandas. You may reply, SQL isn't as easy to understand! Fair point, it's a declarative language... which is why I use Malloy. Malloy is to TypeScript as Javascript is to SQL. Malloy is much easier to read and write (just as TypeScript is) because it has a built in semantic model -- all the joins, measures, and dimensions are done in one place.
Here is an example [1] of visualizing college football games. Here are all the queries, and semantic model that power all the visualizations [2] Here is the AI generated typescript/react that does the visualizations [3]. The Malloy ecosystem has Malloyyo and Publisher which are replacements for PowerBI and Tableau and Looker. Here is another example for visualizing global trade [4].
[1] - https://mrtimo.github.io/cfb-games/games-2026.html?week=Week... [2] - https://github.com/mrtimo/cfb-games/blob/main/drives.malloy [3] - https://github.com/mrtimo/cfb-games/blob/main/dashboards/gam... [4] - https://tradeexplorer.org/
The linked announcement includes the facts that Polars now supports SQL and the new version scales better than DuckDB