Apache Spark

Statusverified
Readsyes
Writesyes
Iceberg v3reads and writes
Geometry, geographyreads and writes values; spatial functions need Apache Sedona
Last verified2026-10-03 (Spark 3.5 + Iceberg 1.11; Spark 4.1 + Iceberg 1.12)

Spark reads and writes the Iceberg tables of a LakehouseBox catalog through Iceberg's own Spark runtime: a SparkCatalog of type rest pointed at our catalog, with Iceberg's S3FileIO for the files. Each LakehouseBox catalog is one Spark catalog, addressed by its name in Spark SQL. The writing recipe asks the catalog for short-lived storage credentials with each table, so it holds only the catalog credential. The read-only recipe turns that off (header.X-Iceberg-Access-Delegation=none) and reads with the read-only key pair as a static storage key.

Before you start

  • Two jars of the same Iceberg version, matching your Spark: iceberg-spark-runtime-<spark>_<scala> and iceberg-aws-bundle. The runs on this page used iceberg-spark-runtime-3.5_2.12 1.11.0 with Spark 3.5.6, and iceberg-spark-runtime-4.1_2.13 1.12.0 with Spark 4.1.3 (image apache/spark:4.1.3-scala2.13-java21-python3-ubuntu).
  • Java 17 or newer for Iceberg 1.10 and later. On the Java 11 that the apache/spark:3.5.3 images ship, Iceberg 1.11.0 fails every statement with UnsupportedClassVersionError … class file version 61.0 (measured 2026-09-19); 1.9.2 is the last Java 11 line.
  • For geometry and geography columns: Spark 4.1 with Iceberg 1.12 and spark.sql.geospatial.enabled=true. Spark 3.5 has no such types (CREATE TABLE with a GEOMETRY column is a ParseException).
  • Every run below also set Iceberg's SQL extensions, spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions; the recipe does not.

Connect

lhbox connect --catalog <catalog> --engine spark --show-secrets             # the recipe for your access
lhbox connect --catalog <catalog> --engine spark --readonly --show-secrets  # the read-only key pair, for a colleague

prints the configuration with your catalog's name, storage location and credential filled in. Without --show-secrets the secret is printed as ********. It looks like this:

spark.sql.catalog.<catalog_name>=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.<catalog_name>.type=rest
spark.sql.catalog.<catalog_name>.uri=https://catalog.lakehousebox.com
spark.sql.catalog.<catalog_name>.warehouse=s3://<handle>--<catalog>/
spark.sql.catalog.<catalog_name>.credential=<client_id>:<client_secret>
spark.sql.catalog.<catalog_name>.io-impl=org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.<catalog_name>.s3.endpoint=https://s3.lakehousebox.com
spark.sql.catalog.<catalog_name>.s3.path-style-access=true
spark.sql.catalog.<catalog_name>.client.region=us-east-1
spark.sql.catalog.<catalog_name>.header.X-Iceberg-Access-Delegation=vended-credentials

Pass these as --conf flags or in spark-defaults.conf; the catalog is then addressed by its name in Spark SQL (SELECT * FROM acme_corp.demo.cities for the catalog acme_corp). client.region=us-east-1 is the signing region the storage accepts, not a placement (the data stays in Frankfurt, Germany); without it the AWS SDK looks for a region on the JVM and fails before the first request.

Reading

  • Format version 3, at scale (2026-10-03, Spark 4.1.3 with Iceberg 1.12.0, from a laptop in Madrid): Spark opened, counted and read the geometry values of all six Overture tables in our public catalog, 7.1 to 81.5 million rows each, written by lhbox table import (DuckDB 1.5.6) and by Spark itself. SELECT count(*) on the 7,136,355 road segments took 1.4 s; reading one geometry value, hex(st_asbinary(geometry)), about a second.
  • Tables Snowflake writes, whose file paths are s3compat://…, read too: count, both geo columns and their SRIDs (2026-10-03, same versions). Trino 483 and PyIceberg 0.12 cannot read those tables yet.
  • With vended credentials only (2026-10-16, Spark 3.5.6 with Iceberg 1.11.0, on a local copy of the service running the production image, no storage key anywhere): a query on a table answered; the same query in the same Spark session after 960 s, past the 900 s that copy's storage sessions lasted, answered again. Iceberg logs Invalid credentials endpoint: null when it tries to refresh them, because the catalog names no refresh endpoint; the query still works. On the service a storage session lasts one hour.

The 2026-10-03 runs used one of the catalog's key pairs as a static storage key (header.X-Iceberg-Access-Delegation=none with s3.access-key-id / s3.secret-access-key) rather than the printed recipe's vended credentials.

Writing

On 2026-10-03, Spark 4.1.3 with Iceberg 1.12.0 wrote three format-version 3 tables into the service, each with a geometry and a geography column: 7,136,355 rows in 36 s (23 files, 2.4 GB), 19,155,142 rows in 67 s (36 files, 4.4 GB) and 81,455,423 rows in 319 s (85 files, 7.0 GB). The statements, from the run (the catalog was named lh, the attribute columns are left out):

CREATE TABLE lh.transportation.roads_es_geography (…, geometry GEOMETRY(4326), geography GEOGRAPHY(4326))
  USING iceberg TBLPROPERTIES ('format-version'='3', 'write.distribution-mode'='none',
    'write.target-file-size-bytes'='134217728', 'write.parquet.row-group-size-bytes'='8388608',
    'write.parquet.compression-codec'='zstd');
INSERT INTO lh.transportation.roads_es_geography
  SELECT …, st_setsrid(st_geomfromwkb(geometry), 4326), st_setsrid(st_geogfromwkb(geometry), 4326) FROM src;

st_geomfromwkb gives SRID 0, so set the column's SRID with st_setsrid. CREATE NAMESPACE and DROP TABLE … PURGE ran in the same job. UPDATE, DELETE and MERGE INTO have not been run with this recipe against the service yet; they worked against our catalog in its evaluation before launch (2026-09-19, Spark 3.5.3 with Iceberg 1.9.2, MERGE INTO 10,000 updated and 5,000 inserted rows in 6.2 s).

Geometry and geography

Measured 2026-10-03 on the Overture tables, against answers computed separately from the source files:

  • Spark 4.1 reads the values but cannot compute on them. Its geo functions are five: st_asbinary, st_geogfromwkb, st_geomfromwkb, st_setsrid, st_srid. There is no spatial predicate.

  • Apache Sedona 1.9.1 adds them. A box question on the road segments of central Valencia counted 14,881, the exact answer, in 31 s (a full scan). On the geography column, "within 2 km" counted 10,376 in 45 s, the same count as Snowflake, against 10,364 computed separately in a metric projection:

    SELECT count(*) FROM twin_wkb
    WHERE ST_Distance(ST_GeogFromWKB(gwkb), ST_GeogFromWKB(ST_AsBinary(ST_Point(-0.3763, 39.4699)))) <= 2000

    Spark 4.1 and Sedona both define st_asbinary, and with Sedona registered first Spark's is hidden. The run read the WKB with Spark's function into a view first (SELECT st_asbinary(geography) AS gwkb FROM …), then registered Sedona (SedonaSQLRegistrator.registerAll).

Known limitations

As measured on 2026-10-03 with Spark 4.1.3 and Iceberg 1.12.0; none reported upstream yet:

  • No spatial predicate without Sedona, and with it every spatial question is a full scan.
  • Spark writes no geometry bounds into the table's metadata, so no engine can skip files of a Spark-written geometry column by location (DuckDB prunes the tables lhbox table import writes).
  • Snowflake fails to read Spark-written geometry columns at scale (a cast error after the first 100,000 rows); the geography columns read (Snowflake).
  • Tables written by iceberg-go 0.7.0 fail with a ClassCastException.
  • Spark 3.5 and Iceberg 1.11 have no geo types: use Spark 4.1 with Iceberg 1.12 for such tables.
Using Spark with LakehouseBox and found something missing, wrong or out of date here? Write to hello@lakehousebox.com: what you ran, the version, and what happened. Product names and logos are trademarks of their owners; their use here does not imply endorsement.