Testing DuckDB 2.0 on our Iceberg datasets

We are excited about DuckDB 2.0, scheduled for the second half of October. The 2.0 preview brings major engine changes—most notably asynchronous I/O across the execution pipeline, far more parallelized remote reads, and partition-aware planning for lakehouse formats. That is a big deal for us because DuckDB is how our users' agents query their data: each agent runs it on its own machine against Apache Iceberg tables in our object storage, through lhbox duckdb, and our own services run it next to the data.

Over the last few days we have been preparing our stack for 2.0 and running performance and correctness checks to catch issues early. Because the preview announcement does not include numbers for remote Iceberg reads, we are sharing our measurements here so others testing 2.0 can compare notes.

Pre-release note: All 2.0 runs used 2.0.0.dev2610011535 (alpha v2.0.0-alpha43763, Iceberg extension 5b9ff899a1), the newest Python build available on 14 October 2026, compared against DuckDB 1.5.6. Behavior and timings may change before 2.0.0 final.

Where we ran the tests

Every test queried the same Apache Iceberg tables stored in Nuremberg, Germany, from two different machines and network paths:

Cold and warm TPC-H reads

In the same data center, every cold query was faster on 2.0, with per-query speedups ranging from 1.05× (q21) to 3.78× (q11)—a 1.93× geometric mean across the 22 queries, and a 1.69× reduction in total suite wall time (105.9 s down to 62.7 s). Cold Iceberg queries spend significant time waiting on sequential round-trips: catalog metadata, manifest list, manifest files, Parquet footers, and column chunks. DuckDB 2.0 reads the same total byte volume (~12.1 GB vs. ~12.3 GB) across 52 % more HTTP GET requests (9,346 vs. 6,126), keeping smaller range requests in flight concurrently via async I/O.

From Madrid, the bottleneck shifts from request latency to link bandwidth. A cold pass transfers ~12 GB, and 1.5.6 already averages 45 MB/s over our residential connection. Selective queries still speed up (q02 is 1.8× faster, q16 is 2.0× faster), large scans break even, and a few queries transfer noticeably more data in 2.0 (q15 pulls 381 MB in 2.0 vs. 132 MB in 1.5.6; q14 pulls 461 MB vs. 292 MB), likely from prefetching ranges the scan ultimately does not need. Over a local data-center link that extra transfer is free; over a home connection it dominates query time.

For warm queries (repeating the query on the same connection), 1.5.6 performs zero network reads across all 22 queries from both locations. The 2.0 preview re-fetches parts of the data on repeat runs (q15 transfers 69 MB on the first repeat and 21 MB on the second). In Nuremberg that extra 30 MB across the suite is invisible; from Madrid it totals 170 MB and makes the warm pass 46 % slower (19.8 s vs. 13.6 s). Separately, the two heaviest joins (q09 and q21) were slower warm on 2.0 even with zero network traffic (q09 went from 3.9 s to 6.3 s on the 2-vCPU server and from 1.3 s to 2.0 s on the M1 laptop).

Iceberg writes: 409 files down to 4

We also timed loading the SF10 dataset into Iceberg with each version (CREATE TABLE … AS SELECT, with lineitem ordered by l_shipdate):

DuckDB 1.5.6 flushes small ~5 MB Parquet files during Iceberg writes, which previously required background compaction (merging lineitem from 409 files down to 17). DuckDB 2.0 writes target-sized ~536 MB files out of the box, finishes the 8-table load 22 % faster, and records total-files-size in the Iceberg snapshot summary.

File count also explains part of the read speedup. When we ran the cold TPC-H suite in Nuremberg against the uncompacted 4-file lineitem layout written by 2.0, DuckDB 1.5.6 sped up by 18 % (105.9 s → 86.4 s) while 2.0 stayed flat (62.7 s → 65.2 s), narrowing 2.0's per-query geomean lead from 1.93× to 1.42×. Much of 2.0's read advantage comes from amortizing the overhead of many small files; writing larger Parquet files helps every reader.

Real queries on public datasets

Interactive sessions and AI agents rarely run 22-query TPC-H sweeps; they typically start a fresh DuckDB process, ATTACH a catalog, and run one selective query. We timed that pattern against our public Iceberg catalog (commons/pubdata), running five fresh processes per query and engine:

For selective queries, 2.0 returns a cold answer 1.4× to 2.9× faster from Madrid, and catalog ATTACH itself drops from 0.28 s to 0.19 s. The Overture pharmacy lookup gains the most (9.39 s → 3.28 s) because it issues 1,146 small HTTP reads that 2.0 overlaps.

The point-in-polygon scan over admin_areas illustrates the tradeoff of splitting large reads over higher latency. The query scans 460 MB across a few large files: 1.5.6 fetches them in 10 HTTP GETs, whereas 2.0 splits the read into 226 smaller GETs. In Nuremberg 2.0 is 1.15× faster (2.26 s → 1.97 s); from Madrid those 226 requests pay the extra round-trip cost and 2.0 is 17 % slower.

One spatial regression

One row in the chart above matters to us more than its size suggests: the Paris rail bounding-box query (geometry && box). Our spatial tables are sorted along a Hilbert curve so that a filter like this skips most files, and on DuckDB 1.5.6 it reads 9.9 MB. The 2.0 preview read the whole 281 MB table. We reported it as duckdb-iceberg#1464; the Iceberg extension's part is merged (#1467), and the remaining part in DuckDB core is a small cast fix we proposed in duckdb#27249. Until it ships, filtering on the plain bbox_xmin, bbox_ymin, bbox_xmax, bbox_ymax columns prunes on both versions (under 1 MB read).

INSTALL iceberg; LOAD iceberg; INSTALL httpfs; LOAD httpfs; INSTALL spatial; LOAD spatial;
CREATE SECRET lhbox_public (TYPE S3, ENDPOINT 'lakehousebox-public-data.b-cdn.net',
                            URL_STYLE 'path', USE_SSL true, KEY_ID '', SECRET '');
EXPLAIN ANALYZE
SELECT class, round(sum(length_m) / 1000, 1) AS km
FROM iceberg_scan('s3://commons--pubdata/transport/rail_segments')
WHERE geometry && ST_MakeEnvelope(2.20, 48.80, 2.45, 48.92)
  AND ST_Intersects(geometry, ST_MakeEnvelope(2.20, 48.80, 2.45, 48.92))
GROUP BY 1 ORDER BY 2 DESC;

What we are doing next

In the same data center, DuckDB 2.0's async I/O cuts cold-query wall time on Iceberg tables by 41 % (1.69× faster in total time, 1.93× per-query geometric mean) and writes 100× fewer Parquet files out of the box. Over a residential internet connection, selective single-table queries also see a clear latency drop, while large scans remain bound by link bandwidth.

We are keeping DuckDB 1.5.6 as our default engine until 2.0.0 final ships and passes our checks, including spatial pruning, and we have exposed the 2.0 build as an opt-in preview engine (lhbox query --engine 2.0) so users can test against it now. Once 2.0.0 is released, we will re-run this benchmark suite and update the numbers here.

Explore how it works →

Related articleWhere does the engine live now?