Polars

Statusverified
Readsyes
Writesyes
Iceberg v3not tested
Geometry, geographynot tested
Last verified2026-09-28 (Polars 1.44 + PyIceberg 0.12)

Polars reads and appends to the Iceberg tables of a LakehouseBox catalog through a PyIceberg table: PyIceberg talks to the catalog and gets the table's short-lived storage credentials, and Polars scans the table into a DataFrame (scan_iceberg) or appends one to it (write_iceberg).

Before you start

  • pip install polars "pyiceberg[pyarrow]". The run on this page used Polars 1.44.2 with PyIceberg 0.12.0.
  • Everything on the PyIceberg page about the catalog object applies here.
  • Appending needs a format-version 2 table: PyIceberg cannot write version 3, the default for new tables. Create the table with lhbox table create … --format-version 2.

Connect

lhbox connect --catalog <catalog> --engine polars --show-secrets             # the recipe for your access
lhbox connect --catalog <catalog> --engine polars --readonly --show-secrets  # the read-only key pair, for a colleague

prints the PyIceberg catalog with your catalog's location filled in, followed by the Polars lines. As in the PyIceberg recipe, the code reads the credential from environment variables, and the export line above it sets them (******** for the secret without --show-secrets). The catalog part is PyIceberg's recipe:

from pyiceberg.catalog.rest import RestCatalog
catalog = RestCatalog(name='lhbox', uri='https://catalog.lakehousebox.com',
                      warehouse='s3://<handle>--<catalog>/', credential='<client_id>:<client_secret>',
                      **{'s3.endpoint': 'https://s3.lakehousebox.com', 's3.path-style-access': 'true'})
table = catalog.load_table(('<namespace>', '<table>'))

# writing: create from an Arrow schema, then append
import pyarrow.parquet as pq
arrow = pq.read_table('cities.parquet')
tbl = catalog.create_table(('demo', 'cities'), schema=arrow.schema)
tbl.append(arrow)

and the Polars part (the same two calls, a filtered scan and an append, ran on 2026-09-28 against a table demo.cities):

# pip install polars "pyiceberg[pyarrow]"; the catalog object is the PyIceberg one above
import polars as pl
table = catalog.load_table("demo.cities")
df = pl.scan_iceberg(table, reader_override="pyiceberg").filter(pl.col("country") == "DE").collect()
pl.DataFrame({"id": [8], "name": ["Hamburg"], "country": ["DE"], "pop_k": [1900]}).write_iceberg(table, mode="append")

Keep reader_override="pyiceberg". Polars' own reader does not use the per-table credentials the catalog vends: it looks for credentials at the cloud instance-metadata address instead and fails outside a cloud VM:

OSError: object-store error: Generic S3 error: Error performing PUT http://169.254.169.254/latest/api/token ... after 8 retries

With the override, PyIceberg reads the files and asks the catalog for credentials each time.

Reading

Verified 2026-09-28 with Polars 1.44.2 and PyIceberg 0.12.0, on a format-version 2 table written by PyIceberg: the filtered scan of the recipe returned the expected rows,

shape: (2, 2)
┌───────────┬───────┐
│ name      ┆ pop_k │
│ ---       ┆ ---   │
│ str       ┆ i64   │
╞═══════════╪═══════╡
│ Berlin    ┆ 3700  │
│ Nuremberg ┆ 520   │
└───────────┴───────┘

and the read-only recipe scanned the table too (7 rows, after the append below).

Polars' native reader also works when you hand it the table's vended credentials yourself, as storage_options taken from table.io.properties (s3.access-key-id, s3.secret-access-key, s3.session-token, the endpoint https://s3.lakehousebox.com, region us-east-1, path-style): it read the same 7 rows. Those credentials expire within the hour; the override asks the catalog each time.

Writing

write_iceberg(table, mode="append") with the writing recipe appended a row: 6 rows before, 7 after, one new snapshot (2026-09-28). With the read-only recipe the same append was refused by the storage when it uploaded the data file:

OSError When initiating multiple part upload for key 'demo/cities/data/00000-0-….parquet' in bucket '…': AWS Error ACCESS_DENIED during CreateMultipartU…

Known limitations

As measured on 2026-09-28 with Polars 1.44.2 and PyIceberg 0.12.0:

  • Polars' native reader ignores vended credentials (above): keep reader_override="pyiceberg". Not reported upstream yet.
  • Format version 3 and geometry were not tested with Polars. Polars reads and writes through PyIceberg, whose 0.12 release cannot write version 3 nor read geometry values (PyIceberg).
Using Polars with LakehouseBox and found something missing, wrong or out of date here? Write to hello@lakehousebox.com: what you ran, the version, and what happened. Product names and logos are trademarks of their owners; their use here does not imply endorsement.