PyIceberg
| Status | verified |
|---|---|
| Reads | yes |
| Writes | yes (format version 2) |
| Iceberg v3 | reads; cannot write |
| Geometry, geography | no (reads schema and counts only) |
| Last verified | 2026-10-03 (PyIceberg 0.12) |
PyIceberg is Apache Iceberg's Python library. Its RestCatalog talks to the LakehouseBox catalog and reads and writes the Parquet files with PyArrow, using the storage credentials the catalog vends with each table. It is the Python way to create tables, append Arrow data, register existing Parquet files, look at snapshots and roll a table back; Polars reads and writes through it.
Before you start
- PyIceberg 0.12 with PyArrow:
pip install "pyiceberg[pyarrow]". The runs on this page used PyIceberg 0.12.0 and PyArrow 25.0.1. - The
s3://form of the catalog's location forwarehouse(s3://<handle>--<catalog>/): PyIceberg requires it. DuckDB wants the bare bucket name instead; copying one recipe's value into the other engine fails.
Connect
lhbox connect --catalog <catalog> --engine pyiceberg --show-secrets # the recipe for your access
lhbox connect --catalog <catalog> --engine pyiceberg --readonly --show-secrets # the read-only key pair, for a colleague
prints the Python with your catalog's location filled in. The code reads the credential from two environment variables, LHBOX_<CATALOG>_CLIENT_ID and LHBOX_<CATALOG>_CLIENT_SECRET (LHBOX_<CATALOG>_RO_… for the read-only pair), and the line above it is the export that sets them (******** for the secret without --show-secrets). With the credential written in, it looks like this:
from pyiceberg.catalog.rest import RestCatalog
catalog = RestCatalog(name='lhbox', uri='https://catalog.lakehousebox.com',
warehouse='s3://<handle>--<catalog>/', credential='<client_id>:<client_secret>',
**{'s3.endpoint': 'https://s3.lakehousebox.com', 's3.path-style-access': 'true'})
table = catalog.load_table(('<namespace>', '<table>'))
# writing: create from an Arrow schema, then append
import pyarrow.parquet as pq
arrow = pq.read_table('cities.parquet')
tbl = catalog.create_table(('demo', 'cities'), schema=arrow.schema)
tbl.append(arrow)
credentialis the catalog's key pair; PyIceberg exchanges it for a catalog token at the catalog and renews it.- Vended credentials. PyIceberg asks for storage credentials with every request (
X-Iceberg-Access-Delegation), and they override any statics3.*key you pass. To use your own keys instead, for example to register files from another bucket, pass"header.X-Iceberg-Access-Delegation": "none". The read-only recipe does that already: it reads with the read-only key pair as a static key.
Reading
catalog.load_table(…), then table.scan(…).to_arrow() (or .to_pandas()). Verified against the service:
- Format version 2 (2026-09-27, PyIceberg 0.12.0): the read-only recipe loaded a table and returned its metadata location and storage location.
- Snapshots and time travel (2026-09-29, PyIceberg 0.12):
table.metadata.snapshotslists them andtable.scan(snapshot_id=…)reads an older one (time travel). - Format version 3 (2026-10-03, PyIceberg 0.12.0): six Overture tables of 7.1 to 81.5 million rows opened with their schema,
geometryandgeographycolumns included, and gave their row counts. Their geometry values do not read (below).
Load the table again after an hour. A Table object keeps the storage credentials it was loaded with. On a local copy of the service whose storage sessions lasted 900 s (2026-10-16, PyIceberg 0.12.0), a scan with the same Table after 960 s failed with ACCESS_DENIED, and a fresh load_table read the table at once. On the service a session lasts one hour.
Writing
The recipe ends by creating a table from an Arrow schema and appending to it. On 2026-09-27 PyIceberg 0.12.0 created a format-version 2 table of 5 rows that Trino 483, Polars 1.44 and ClickHouse then read; Snowflake read PyIceberg-written tables the next day. More that has run against the service:
- Rollback (2026-09-29, PyIceberg 0.12):
table.manage_snapshots().rollback_to_snapshot(<snapshot id>).commit()made the older snapshot current again (time travel). - Registering existing Parquet files without rewriting them (2026-10-02, PyIceberg 0.12):
table.add_files([…])registered a 52 MB Parquet file into a format-version 2 table created from its Arrow schema, 407,277 rows in one commit, the stored file byte-identical to the original. - Commit conflicts. Another writer, or the catalog's maintenance, can commit between your read and your commit; PyIceberg then raises
CommitFailedException. Catch it, load the table again and retry (errors).
PyIceberg 0.12 cannot write format version 3: it creates a v3 table with geometry and geography columns, but append and add_files both refused to fill it (2026-10-03), and the newest release, 0.12.0 with pyiceberg-core 0.10.1, still refused append and overwrite into a plain v3 table ("Cannot write manifest for table version: 3", 2026-11-15). New tables are format version 3 by default (since 2026-11-15), so create a table PyIceberg writes as version 2: lhbox table create … --format-version 2, "format_version": 2 on POST /v1/tables, or properties={"format-version": "2"} in PyIceberg's own create_table; a catalog whose tables PyIceberg writes can take lhbox catalog update <catalog> --default-format-version 2. Write v3 tables with lhbox table import, lhbox table add-files, DuckDB or Spark.
Geometry and geography
Measured on 2026-10-03 with PyIceberg 0.12.0, with and without geoarrow-pyarrow, on the Overture tables:
- The schema and the row count read; the values do not:
ResolveError: Cannot promote an binary to geometry(and… to geography). - PyIceberg has no spatial predicate, so a box question cannot be pushed into the scan.
- A geometry column with an explicit coordinate system,
geometry(EPSG:28992)for example, stops the whole table from loading:ValidationError: Could not parse geometry(EPSG:28992)(2026-09-21 and 2026-09-28).
For geometry use DuckDB, and for geography Snowflake or Spark with Apache Sedona.
Known limitations
As measured with PyIceberg 0.12:
- No writes to format-version 3 tables (above).
- No geometry or geography values, and no tables whose geometry names a coordinate system (above).
- Tables Snowflake writes load but cannot be scanned: they record their files as
s3compat://…, and PyIceberg answersValueError: Unrecognized filesystem type in URI: s3compat(2026-10-03). Not reported upstream yet. - A loaded
Tableoutlives its storage credentials after an hour: load it again (above).