Spark, Trino, Polars, and DuckDB on 157 GB of TPC-H: What SF400 Actually Reveals
July 4, 2026
Most engine benchmarks end with a tidy winner. This one doesn't, and that's the point. I ran the 22 TPC-H queries against the SF400 dataset across four engines - Spark, Trino, Polars, and DuckDB - and two very different kinds of machine, and the result that survived was not "engine X is fastest." It was that hardware, memory, and execution mode are inseparable from the engine at this scale.
The full write-up is also available on Medium: Spark, Trino, Polars, and DuckDB on 157 GB of TPC-H data: what 22 queries actually reveals.
TL;DR
I ran the 22 TPC-H queries against the SF400 dataset (~157 GB of Parquet, with a 102 GB lineitem table, 2.4 B rows) across four engines and two very different kinds of machine - a five-node cluster and a single big box:
- Spark finished all 22 queries in 12.2 minutes on an 80-vCPU cluster.
- Trino, on the same cluster and same data, took 27.8 minutes - Spark was ~2.3× faster.
- On a single Linux node, DuckDB finished all 22 queries in 3.3 minutes on a 32-vCPU box and 6.8 minutes on a 16-vCPU box; Polars-streaming did it in 5.0 minutes. A single node - even one with the same specs as one of the cluster's five machines - beat the entire 80-vCPU cluster.
The headline isn't "engine X wins." It's that hardware, memory, and execution mode are inseparable from the engine at this scale - and that for 150-odd GB of data, one well-fed Linux node is faster and cheaper than a cluster.
The setup
Dataset. TPC-H at scale factor 400 - about 157 GB of Parquet, dominated by lineitem at 102 GB (2.4 B rows). This is the interesting regime: too big to shrug off, small enough that a beefy single node can still hold its own.
The machines:
| Cluster | Linux single node | |
|---|---|---|
| Engines | Spark, Trino | Polars, DuckDB |
| Hardware | 5× r6id.4xlarge (80 vCPU / 640 GiB) | r6id.8xlarge (32 vCPU / 256 GiB) and r6id.4xlarge (16 vCPU / 128 GiB) |
| Storage | Parquet in HDFS on local NVMe | Parquet on local NVMe |
| Platform | EMR 7.13 (Linux) | Ubuntu 24.04 |
Note the second column: the 16-vCPU r6id.4xlarge is exactly one node of the five-node cluster - the cleanest possible "1 node vs 5 nodes, same silicon" test.
Method. Every query is a single, cold, timed pass - no warm-up. That choice matters: warming a query primes the JIT and the page cache and flatters the second run. For a fair cross-engine comparison, cold single-pass is the honest number.
For the cluster engines, the data was copied from S3 into HDFS with s3-dist-cp and replicated to every datanode (setrep) so scans are balanced across the cluster instead of hammering one node. On the single nodes, the SF400 Parquet was generated straight onto local NVMe with tpchgen-cli.
Round 1: Spark vs Trino, same cluster, same data
This is the cleanest comparison in the whole study - identical 80-vCPU cluster, identical HDFS data, identical cold single-pass method. The only variable is the engine.
| Spark | Trino | |
|---|---|---|
| Total, 22 queries | 732.8 s (12.2 min) | 1667.3 s (27.8 min) |
| Relative | 1.0× | 2.3× slower |
Spark won nearly every query, and it won the expensive ones decisively:
| Query | Spark (s) | Trino (s) | Spark advantage |
|---|---|---|---|
| q18 (large-volume customer) | 40.5 | 170.3 | 4.2× |
| q17 (small-quantity order) | 47.7 | 136.4 | 2.9× |
| q6 (forecast revenue) | 20.8 | 61.2 | 2.9× |
q21 (suppliers who kept orders waiting - triple lineitem self-join) | 74.5 | 209.8 | 2.8× |
| q14 (promotion effect) | 23.9 | 61.6 | 2.6× |
| q19 (discounted revenue) | 25.1 | 62.2 | 2.5× |
The only near-ties were the tiny queries - q2, q13, q16, q22 - where fixed startup overhead dominates and the engines land within ~10% of each other.
Why does Spark pull ahead? On these wide r6id nodes with plenty of RAM per core, Spark's whole-stage code generation and codegen'd hash joins turn the big join/aggregate pipelines into tight, vectorized loops. Trino is a superb interactive engine, but for this batch-style, scan-and-join-heavy workload on a small dedicated cluster, Spark's execution model simply did more work per second.
Round 2: a single Linux node beats the cluster
Now the surprise. I ran Polars and DuckDB on single Linux boxes with fast local NVMe: a 32-vCPU / 256 GiB r6id.8xlarge, and a 16-vCPU / 128 GiB r6id.4xlarge (one cluster node's worth of hardware). Same cold single-pass, all 22 queries.
The single node wins - decisively.
| Engine / mode | Hardware | Queries | Total |
|---|---|---|---|
| DuckDB | 32 vCPU / 256 GiB | 22/22 | 3.3 min |
| DuckDB | 16 vCPU / 128 GiB | 22/22 | 6.8 min |
| Polars (streaming) | 32 vCPU / 256 GiB | 22/22 | 5.0 min |
| Polars (streaming) | 16 vCPU / 128 GiB | 20/22 (q5, q18 OOM) | 8.8 min |
| Spark (cluster) | 80 vCPU / 5 nodes | 22/22 | 12.2 min |
| Trino (cluster) | 80 vCPU / 5 nodes | 22/22 | 27.8 min |
Read that again: DuckDB on a single 16-vCPU box (6.8 min) beat the 80-vCPU, five-node Spark cluster (12.2 min) by ~1.8× - on the cluster's own per-node hardware, one machine against five. On the 32-vCPU box, DuckDB was ~3.7× faster than Spark and Polars-streaming ~2.4× faster. At SF400 the whole dataset fits one big box's NVMe and RAM, and a columnar engine skips every ounce of distributed-execution overhead.
Two things the single-node runs pinned down about the ceiling:
- DuckDB is the robust engine. It finishes all 22 at both box sizes, never OOMs (it spills to disk cleanly), and halving the RAM+cores merely doubled the time (199 → 411 s) - textbook linear scaling.
- Polars streaming has a memory floor between 128 and 256 GiB at SF400. It sweeps all 22 with 256 GiB but OOMs two queries (q5, q18) at 128 GiB. Its in-memory mode is a dead end at this scale - it OOMs on q1 and q9 even with 256 GiB (q9 wanted ~257 GB of RAM), and where it survives it's 2-7× slower than streaming. Use DuckDB anywhere, or Polars streaming on ≥256 GiB.
The real lesson: engine, hardware, and memory don't separate
If you take one thing from this, take this table:
| Engine | Hardware | Total (22 q) |
|---|---|---|
| DuckDB | 32 vCPU single node | 3.3 min |
| Polars (streaming) | 32 vCPU single node | 5.0 min |
| DuckDB | 16 vCPU single node | 6.8 min |
| Polars (streaming) | 16 vCPU single node | 8.8 min (20 q) |
| Spark | 80 vCPU cluster | 12.2 min |
| Trino | 80 vCPU cluster | 27.8 min |
The same DuckDB spans 3.3 to 6.8 minutes just by halving the box, and both numbers beat an 80-vCPU cluster. Every number is a product of engine × hardware × execution mode.
What survived contact with 157 GB of Parquet:
- Spark is the batch workhorse among cluster engines - its codegen'd execution beat Trino by 2.3× on identical hardware and data.
- You probably don't need a cluster. A single Linux node - even one the size of a single cluster node - running DuckDB or Polars-streaming beat the whole 80-vCPU cluster. For a lot of "big data" that isn't actually that big, one fat node is the boring, cheap, correct answer.
- Memory and execution mode are destiny at this scale. Polars OOMs without streaming and OOMs two queries at 128 GiB; DuckDB spills and sails through. The engine you pick matters less than whether it fits the machine - and how it behaves when it almost doesn't.
Caveats (read these before quoting a number)
- Cross-machine comparisons mix hardware and engine. The Spark-vs-Trino comparison is apples-to-apples; the single-node-vs-cluster comparison is systems, not engines in isolation - but that's the real-world decision you actually face.
- Cold, single-pass timings throughout. Warm numbers would be lower and less comparable.
- One total is partial: Polars-streaming misses q5+q18 at 128 GiB (OOM). Full-22 totals are labeled as such.
- Storage differs: the cluster used HDFS-on-NVMe; the single nodes used local NVMe. Both are "local fast disk," but not identical.
- TPC-H is a synthetic star-schema workload. Your joins, selectivity, and data layout will move these numbers. Benchmark your own.
Closing
The tidy conclusion "engine X is fastest" didn't survive contact with 157 GB of Parquet. What survived is more useful: Spark dominates cluster batch analytics, a single Linux node beats a small cluster outright, and at these scales your memory budget and execution mode decide more than the logo on the engine.
If you're spinning up a cluster to crunch 150-odd gigabytes, it's worth asking first whether one big Linux box and a columnar engine would just... do it faster. On this workload, it did.