Apache Spark
| Status | verified |
|---|---|
| Reads | yes |
| Writes | yes |
| Iceberg v3 | reads and writes |
| Geometry, geography | reads and writes values; spatial functions need Apache Sedona |
| Last verified | 2026-10-03 (Spark 3.5 + Iceberg 1.11; Spark 4.1 + Iceberg 1.12) |
Spark reads and writes the Iceberg tables of a LakehouseBox catalog through Iceberg's own Spark runtime: a SparkCatalog of type rest pointed at our catalog, with Iceberg's S3FileIO for the files. Each LakehouseBox catalog is one Spark catalog, addressed by its name in Spark SQL. The writing recipe asks the catalog for short-lived storage credentials with each table, so it holds only the catalog credential. The read-only recipe turns that off (header.X-Iceberg-Access-Delegation=none) and reads with the read-only key pair as a static storage key.
Before you start
- Two jars of the same Iceberg version, matching your Spark:
iceberg-spark-runtime-<spark>_<scala>andiceberg-aws-bundle. The runs on this page usediceberg-spark-runtime-3.5_2.121.11.0 with Spark 3.5.6, andiceberg-spark-runtime-4.1_2.131.12.0 with Spark 4.1.3 (imageapache/spark:4.1.3-scala2.13-java21-python3-ubuntu). - Java 17 or newer for Iceberg 1.10 and later. On the Java 11 that the
apache/spark:3.5.3images ship, Iceberg 1.11.0 fails every statement withUnsupportedClassVersionError … class file version 61.0(measured 2026-09-19); 1.9.2 is the last Java 11 line. - For
geometryandgeographycolumns: Spark 4.1 with Iceberg 1.12 andspark.sql.geospatial.enabled=true. Spark 3.5 has no such types (CREATE TABLEwith aGEOMETRYcolumn is aParseException). - Every run below also set Iceberg's SQL extensions,
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions; the recipe does not.
Connect
lhbox connect --catalog <catalog> --engine spark --show-secrets # the recipe for your access
lhbox connect --catalog <catalog> --engine spark --readonly --show-secrets # the read-only key pair, for a colleague
prints the configuration with your catalog's name, storage location and credential filled in. Without --show-secrets the secret is printed as ********. It looks like this:
spark.sql.catalog.<catalog_name>=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.<catalog_name>.type=rest
spark.sql.catalog.<catalog_name>.uri=https://catalog.lakehousebox.com
spark.sql.catalog.<catalog_name>.warehouse=s3://<handle>--<catalog>/
spark.sql.catalog.<catalog_name>.credential=<client_id>:<client_secret>
spark.sql.catalog.<catalog_name>.io-impl=org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.<catalog_name>.s3.endpoint=https://s3.lakehousebox.com
spark.sql.catalog.<catalog_name>.s3.path-style-access=true
spark.sql.catalog.<catalog_name>.client.region=us-east-1
spark.sql.catalog.<catalog_name>.header.X-Iceberg-Access-Delegation=vended-credentials
Pass these as --conf flags or in spark-defaults.conf; the catalog is then addressed by its name in Spark SQL (SELECT * FROM acme_corp.demo.cities for the catalog acme_corp). client.region=us-east-1 is the signing region the storage accepts, not a placement (the data stays in Frankfurt, Germany); without it the AWS SDK looks for a region on the JVM and fails before the first request.
Reading
- Format version 3, at scale (2026-10-03, Spark 4.1.3 with Iceberg 1.12.0, from a laptop in Madrid): Spark opened, counted and read the geometry values of all six Overture tables in our public catalog, 7.1 to 81.5 million rows each, written by
lhbox table import(DuckDB 1.5.6) and by Spark itself.SELECT count(*)on the 7,136,355 road segments took 1.4 s; reading one geometry value,hex(st_asbinary(geometry)), about a second. - Tables Snowflake writes, whose file paths are
s3compat://…, read too: count, both geo columns and their SRIDs (2026-10-03, same versions). Trino 483 and PyIceberg 0.12 cannot read those tables yet. - With vended credentials only (2026-10-16, Spark 3.5.6 with Iceberg 1.11.0, on a local copy of the service running the production image, no storage key anywhere): a query on a table answered; the same query in the same Spark session after 960 s, past the 900 s that copy's storage sessions lasted, answered again. Iceberg logs
Invalid credentials endpoint: nullwhen it tries to refresh them, because the catalog names no refresh endpoint; the query still works. On the service a storage session lasts one hour.
The 2026-10-03 runs used one of the catalog's key pairs as a static storage key (header.X-Iceberg-Access-Delegation=none with s3.access-key-id / s3.secret-access-key) rather than the printed recipe's vended credentials.
Writing
On 2026-10-03, Spark 4.1.3 with Iceberg 1.12.0 wrote three format-version 3 tables into the service, each with a geometry and a geography column: 7,136,355 rows in 36 s (23 files, 2.4 GB), 19,155,142 rows in 67 s (36 files, 4.4 GB) and 81,455,423 rows in 319 s (85 files, 7.0 GB). The statements, from the run (the catalog was named lh, the attribute columns are left out):
CREATE TABLE lh.transportation.roads_es_geography (…, geometry GEOMETRY(4326), geography GEOGRAPHY(4326))
USING iceberg TBLPROPERTIES ('format-version'='3', 'write.distribution-mode'='none',
'write.target-file-size-bytes'='134217728', 'write.parquet.row-group-size-bytes'='8388608',
'write.parquet.compression-codec'='zstd');
INSERT INTO lh.transportation.roads_es_geography
SELECT …, st_setsrid(st_geomfromwkb(geometry), 4326), st_setsrid(st_geogfromwkb(geometry), 4326) FROM src;
st_geomfromwkb gives SRID 0, so set the column's SRID with st_setsrid. CREATE NAMESPACE and DROP TABLE … PURGE ran in the same job. UPDATE, DELETE and MERGE INTO have not been run with this recipe against the service yet; they worked against our catalog in its evaluation before launch (2026-09-19, Spark 3.5.3 with Iceberg 1.9.2, MERGE INTO 10,000 updated and 5,000 inserted rows in 6.2 s).
Geometry and geography
Measured 2026-10-03 on the Overture tables, against answers computed separately from the source files:
Spark 4.1 reads the values but cannot compute on them. Its geo functions are five:
st_asbinary,st_geogfromwkb,st_geomfromwkb,st_setsrid,st_srid. There is no spatial predicate.Apache Sedona 1.9.1 adds them. A box question on the road segments of central Valencia counted 14,881, the exact answer, in 31 s (a full scan). On the
geographycolumn, "within 2 km" counted 10,376 in 45 s, the same count as Snowflake, against 10,364 computed separately in a metric projection:SELECT count(*) FROM twin_wkb WHERE ST_Distance(ST_GeogFromWKB(gwkb), ST_GeogFromWKB(ST_AsBinary(ST_Point(-0.3763, 39.4699)))) <= 2000Spark 4.1 and Sedona both define
st_asbinary, and with Sedona registered first Spark's is hidden. The run read the WKB with Spark's function into a view first (SELECT st_asbinary(geography) AS gwkb FROM …), then registered Sedona (SedonaSQLRegistrator.registerAll).
Known limitations
As measured on 2026-10-03 with Spark 4.1.3 and Iceberg 1.12.0; none reported upstream yet:
- No spatial predicate without Sedona, and with it every spatial question is a full scan.
- Spark writes no geometry bounds into the table's metadata, so no engine can skip files of a Spark-written geometry column by location (DuckDB prunes the tables
lhbox table importwrites). - Snowflake fails to read Spark-written geometry columns at scale (a cast error after the first 100,000 rows); the
geographycolumns read (Snowflake). - Tables written by iceberg-go 0.7.0 fail with a
ClassCastException. - Spark 3.5 and Iceberg 1.11 have no geo types: use Spark 4.1 with Iceberg 1.12 for such tables.