Polars
| Status | verified |
|---|---|
| Reads | yes |
| Writes | yes |
| Iceberg v3 | not tested |
| Geometry, geography | not tested |
| Last verified | 2026-09-28 (Polars 1.44 + PyIceberg 0.12) |
Polars reads and appends to the Iceberg tables of a LakehouseBox catalog through a PyIceberg table: PyIceberg talks to the catalog and gets the table's short-lived storage credentials, and Polars scans the table into a DataFrame (scan_iceberg) or appends one to it (write_iceberg).
Before you start
pip install polars "pyiceberg[pyarrow]". The run on this page used Polars 1.44.2 with PyIceberg 0.12.0.- Everything on the PyIceberg page about the catalog object applies here.
- Appending needs a format-version 2 table: PyIceberg cannot write version 3, the default for new tables. Create the table with
lhbox table create … --format-version 2.
Connect
lhbox connect --catalog <catalog> --engine polars --show-secrets # the recipe for your access
lhbox connect --catalog <catalog> --engine polars --readonly --show-secrets # the read-only key pair, for a colleague
prints the PyIceberg catalog with your catalog's location filled in, followed by the Polars lines. As in the PyIceberg recipe, the code reads the credential from environment variables, and the export line above it sets them (******** for the secret without --show-secrets). The catalog part is PyIceberg's recipe:
from pyiceberg.catalog.rest import RestCatalog
catalog = RestCatalog(name='lhbox', uri='https://catalog.lakehousebox.com',
warehouse='s3://<handle>--<catalog>/', credential='<client_id>:<client_secret>',
**{'s3.endpoint': 'https://s3.lakehousebox.com', 's3.path-style-access': 'true'})
table = catalog.load_table(('<namespace>', '<table>'))
# writing: create from an Arrow schema, then append
import pyarrow.parquet as pq
arrow = pq.read_table('cities.parquet')
tbl = catalog.create_table(('demo', 'cities'), schema=arrow.schema)
tbl.append(arrow)
and the Polars part (the same two calls, a filtered scan and an append, ran on 2026-09-28 against a table demo.cities):
# pip install polars "pyiceberg[pyarrow]"; the catalog object is the PyIceberg one above
import polars as pl
table = catalog.load_table("demo.cities")
df = pl.scan_iceberg(table, reader_override="pyiceberg").filter(pl.col("country") == "DE").collect()
pl.DataFrame({"id": [8], "name": ["Hamburg"], "country": ["DE"], "pop_k": [1900]}).write_iceberg(table, mode="append")
Keep reader_override="pyiceberg". Polars' own reader does not use the per-table credentials the catalog vends: it looks for credentials at the cloud instance-metadata address instead and fails outside a cloud VM:
OSError: object-store error: Generic S3 error: Error performing PUT http://169.254.169.254/latest/api/token ... after 8 retries
With the override, PyIceberg reads the files and asks the catalog for credentials each time.
Reading
Verified 2026-09-28 with Polars 1.44.2 and PyIceberg 0.12.0, on a format-version 2 table written by PyIceberg: the filtered scan of the recipe returned the expected rows,
shape: (2, 2)
┌───────────┬───────┐
│ name ┆ pop_k │
│ --- ┆ --- │
│ str ┆ i64 │
╞═══════════╪═══════╡
│ Berlin ┆ 3700 │
│ Nuremberg ┆ 520 │
└───────────┴───────┘
and the read-only recipe scanned the table too (7 rows, after the append below).
Polars' native reader also works when you hand it the table's vended credentials yourself, as storage_options taken from table.io.properties (s3.access-key-id, s3.secret-access-key, s3.session-token, the endpoint https://s3.lakehousebox.com, region us-east-1, path-style): it read the same 7 rows. Those credentials expire within the hour; the override asks the catalog each time.
Writing
write_iceberg(table, mode="append") with the writing recipe appended a row: 6 rows before, 7 after, one new snapshot (2026-09-28). With the read-only recipe the same append was refused by the storage when it uploaded the data file:
OSError When initiating multiple part upload for key 'demo/cities/data/00000-0-….parquet' in bucket '…': AWS Error ACCESS_DENIED during CreateMultipartU…
Known limitations
As measured on 2026-09-28 with Polars 1.44.2 and PyIceberg 0.12.0:
- Polars' native reader ignores vended credentials (above): keep
reader_override="pyiceberg". Not reported upstream yet. - Format version 3 and geometry were not tested with Polars. Polars reads and writes through PyIceberg, whose 0.12 release cannot write version 3 nor read geometry values (PyIceberg).