For agents · the instructions your agent follows. The page for people: Bring your files

Importing files

This page turns Parquet or CSV files into an Iceberg table with one command. lhbox table import reads every file you name in one statement, so the whole set lands as one commit, however many files there are. One statement per file costs one commit, and one wait, per file.

The short path

lhbox doctor                                                                  # finds the DuckDB lhbox will use
lhbox table import analytics.sensors.readings 'data/2026/*.parquet'          # a new table, one commit
lhbox table import analytics.sensors.readings 'data/2026-10/*.parquet' --mode append   # more files later, one commit
lhbox table get analytics.sensors.readings                                    # schema, format version, snapshots, rows

The import runs DuckDB on your machine and reads the files there. It writes the table in your catalog with your catalog's credential, which never appears on screen.

A person can do the same from the account page: see From the account page. Both paths write the table the same way: large files, rows sorted by a date column, and a GeoParquet file's metadata kept as table properties.

A first import, end to end

You need DuckDB 1.5.5 or newer on the machine that holds the files. That is the duckdb Python module of the interpreter running lhbox, or the duckdb binary on your PATH. lhbox doctor shows which one it found.

  • Name the table as catalog.namespace.table. Or use namespace.table with --catalog, or with no catalog at all when you have only one. The namespace is created if it does not exist.
  • Quote globs, so that lhbox expands them, not your shell.
  • Add more files later with --mode append. This is again one commit.
  • Check what landed with lhbox table get: the schema, the format version, the snapshots and the row count.

The import prints one summary line for a person. It prints JSON when its output is piped, or with lhbox --json. The report includes:

  • rows, files, bytes and seconds;
  • commits, counted from the table's snapshots before and after, not assumed;
  • the format_version, the order_by column and the path it took.

--full adds the generated SQL statements.

What you can import

Files How they are read
Parquet: .parquet, .parq, .pq DuckDB read_parquet, columns matched by name across files
CSV: .csv, .tsv, also compressed as .gz, .zst, .bz2 DuckDB read_csv, columns matched by name across files
Anything else --format csv or --format parquet says what it is
  • One format per import. A list that mixes CSV and Parquet stops with mixed_formats. Import each format as its own run, and use --mode append for the second one.
  • A file whose extension is not recognised stops with unknown_format, unless you pass --format.
  • Sources can be local files, globs, or URLs that DuckDB reads directly (s3://…, https://…). ** works in a glob: 'data/**/*.parquet'.
  • A pattern that matches nothing is reported. If nothing matches at all, the command stops before anything is written.
  • CSV has no options of its own. DuckDB detects the delimiter, the header and the column types over the whole list. If it guesses wrong, load the file yourself with read_csv(…) and its options (below).

Creating or appending

  • --mode create (the default) makes a new table. If the table already exists, the import stops with table_exists (exit 2) and suggests --mode append.
  • --mode append writes into a table that exists. Columns are matched by name. A column the table has and the files lack is written as NULL.
  • A column the files have and the table lacks stops the import (columns_not_in_table) before anything is written.
  • Importing needs a write credential on the catalog. With a read-only one, the import stops with exit 3.
  • Every column of a table the import creates is optional. For required columns, identifier fields or a hand-written schema, create the table first with lhbox table create (CLI reference). Then import with --mode append.

When the commit's outcome is unclear. If DuckDB reports an error during the commit, the import reads the table back before answering. Rows that did land are reported as landed. If it cannot tell, it answers import_outcome_unknown (exit 5) with the lhbox table get to run. Do not blindly re-run an append: a second run writes every row twice.

Format version

A new table gets the catalog's default format version. That is 3 for a catalog created since 2026-11-15 (2 before), unless it was changed with lhbox catalog update <catalog> --default-format-version 2|3. A table PyIceberg will write needs 2. --format-version 2|3 overrides it for one import; it applies only to --mode create.

version read and written by geometry
2 read by every engine the Engines page has verified; written by DuckDB, PyIceberg, Spark, Trino and Polars stored as WKB in a binary column
3 written by DuckDB 1.5.5+ and by Spark 4.1 with Iceberg 1.12, geometry included; PyIceberg 0.12 reads it but cannot write it typed columns: geometry, or geometry(EPSG:xxxx); Spark 3.5 has no geometry type

See Engines.

Sorted rows and large files

Before its first row, every table the import writes gets two table properties:

  • write.target-file-size-bytes = 134217728 (128 MiB), so DuckDB writes a few large files instead of many small ones;
  • write.parquet.compression-codec = zstd, so files are compressed with ZSTD instead of DuckDB's default SNAPPY. They are smaller, notably for coordinates.

An append into a table that already has its own file size keeps its settings.

The rows are written ORDER BY one column. By default, this is the first DATE or TIMESTAMP column in your files, so a query that filters on that column can skip whole files. Parquet files with a geometry column are written in Hilbert order of their primary geometry instead, so a geom && box filter skips files and row groups.

  • --order-by <column> names another column: choose the one your queries filter on most. A column the files do not have stops the import before anything is written.
  • --no-order keeps the files' own row order.

Why this matters, how much it saves, and how to load very large tables in chunks: How tables are laid out.

GeoParquet and geometry columns

When a Parquet file carries GeoParquet metadata, the import keeps it as table properties:

  • geo.encoding, geo.primary_column, geo.version;
  • geo.columns, with the bounding boxes merged across files;
  • geo.crs: EPSG:xxxx when the file's CRS names an authority code, and OGC:CRS84 when the file names no CRS;
  • geo.crs.projjson, when the file had a PROJJSON CRS.

The properties are set on version-2 and version-3 tables alike. lhbox table get shows them, and lhbox table set-properties corrects one (--property geo.crs=EPSG:28992).

  • Into a version-2 table the geometry is WKB. Read it back with ST_GeomFromWKB(geom) in DuckDB spatial, or any WKB reader.
  • Into a version-3 table it stays a typed geometry column with its CRS:
lhbox table import analytics.geo.parcels 'parcels/*.parquet' --format-version 3

A version-3 import needs a CRS with an authority code, such as EPSG:25830. A file whose CRS has none stops with geometry_crs_without_authority before the table is created. Import it with --format-version 2, which keeps the full PROJJSON in geo.crs.projjson. Or re-tag the file's CRS with its code first.

Files you laid out yourself: lhbox table add-files

table import reads your rows and writes new files. lhbox table add-files instead keeps your Parquet files as they are: row groups, sort order, compression (one exception, for geometry, is below). Use it when you already wrote them the way you want. An example is raster tiles, one per row, prepared with your own tools such as raquet and queried with your own DuckDB.

It works in two steps. Upload the files to the catalog's files with lhbox file put, then add them to a table:

lhbox file put out/ staging/spain/ --catalog analytics
lhbox table add-files analytics.dem.spain staging/spain/ --sorted-by block
  • The files are copied inside LakehouseBox, not uploaded again. add-files reads only their footers, never the rows. It copies them into the table and registers them all in one commit.
  • Then it deletes the uploaded copies; --keep-staged keeps them.
  • What to name: a file (staging/spain/part-0.parquet), a folder (staging/spain/: every .parquet under it), or a pattern in quotes ('staging/spain/*.parquet'; ** also matches subfolders).
  • A file on your machine, or a URL, is refused with the lhbox file put to run first.
  • Every file must have the same columns and types. With --mode append, they must also fit the table's schema. A mismatch stops the command before anything is copied (schema_mismatch, with the file and both schemas).
  • Files written by an Iceberg engine carry field ids and are refused (files_have_field_ids). Add them with that engine.
  • It needs PyIceberg 0.12 or newer with pyarrow, in the interpreter running lhbox (or --python PATH). It needs no DuckDB.
  • --mode create makes a format version 2 table. --full adds each file's footer summary to the JSON.

Into a format version 3 table (geometry)

--mode append also adds files to a format version 3 table, for example one with a geometry column. Create the table first (lhbox table create … --format-version 3, or lhbox table import), then add the files:

lhbox table create analytics.geo.parcels --format-version 3 \
  --column id:long --column name:string --column 'geom:geometry(EPSG:4326)'
lhbox file put parcels/ staging/parcels/ --catalog analytics
lhbox table add-files analytics.geo.parcels staging/parcels/ --mode append
  • GeoParquet works as it is: the geometry column stays WKB in the file, and the table reads it as geometry.
  • The CRS has to match. A GeoParquet file declares a CRS for each geometry column, and no crs there means OGC:CRS84. A file whose CRS is not the table column's is refused before anything is copied (geometry_crs_mismatch): registered, it would make the table unreadable. A column of Parquet's own GEOMETRY type is checked by the crs of that type, and a PROJJSON crs by its id ({"authority": "EPSG", "code": 4326} is EPSG:4326), which is how DuckDB names it: files from DuckDB's COPY … (FORMAT parquet, GEOPARQUET_VERSION 'V2') are accepted as they are. A GEOMETRY column with no crs, a file with no GeoParquet metadata and no GEOMETRY type, or "crs": null, takes the table's CRS. The names are compared exactly, case included, because DuckDB 1.5 does: a plain GEOMETRY column that DuckDB creates is stored as geometry(ogc:crs84) and cannot read OGC:CRS84 files, so create lon/lat columns as geometry(OGC:CRS84).
  • Spatial filters skip files. Each file's bounding box goes into the table's manifest, as DuckDB's own writes do, so geom && box reads only the files the box touches. The box comes from statistics already in the file: Parquet's GEOMETRY type (DuckDB, GDAL 3.12), or a GeoParquet bbox covering column (geopandas write_covering_bbox=True, GDAL WRITE_COVERING_BBOX=YES). A file with neither is added without a box, and the report warns (no_geometry_bounds): every spatial filter reads it.
  • Spatial filters skip row groups too. DuckDB uses the geospatial statistics of a row group only when the column also carries Parquet's ordinary statistics, and files written through Arrow C++ (pyarrow, GeoPandas) have only the geospatial ones, so inside such a file geom && box reads every row group. add-files gives those geometry columns an empty statistics entry: it writes a new footer after the file's data, which is copied unchanged, and the report counts the files (geometry_statistics_added). Measured on 2 files of 400,000 polygons, a small box read 13.1 MB before and 2.6 MB after (DuckDB 1.5.6; 14.1 and 3.1 MB with DuckDB 2.0), with the same rows. A footer it cannot rewrite (an encrypted file, for one) is added as it is, with a warning (geometry_statistics_not_added).
  • The table must be unpartitioned (partitioned_table otherwise).
  • It needs pyarrow and the lakehousebox-iceberg-core package in that interpreter, not PyIceberg. Without it, the command stops and prints the pip install line to run.
  • Each run is one commit. Rows get their row ids in sequence after whatever the table held, like any other write.
  • If anything fails before the commit, the copies are deleted and your uploaded files stay where they are. Nothing is in the table.

The report warns about these, and changes nothing:

  • row groups that make a point read expensive: over 64 MB, or over 4 MB of rows averaging more than 2 KB each;
  • unsigned integer columns, which Iceberg does not have (DuckDB reads them; other engines may not);
  • with --sorted-by <column>, files whose ranges of that column overlap.

From the account page

The person: opens a catalog, goes to Tables and chooses Import files. The dialog has three steps:

  1. Choose files: files on their computer (Parquet, CSV, or JSON: .json, .jsonl, .ndjson), or files in S3-compatible cloud storage. For cloud storage, the key they type stays in the tab and is never sent to LakehouseBox. The bucket's CORS must allow the site to read.
  2. Preview: the columns, their DuckDB and Iceberg types, and the first rows. For CSV and JSON, Ignore errors drops rows that fail to parse.
  3. Destination: namespace, table name, Iceberg format version, and Order rows by. If the table exists, the dialog offers to append.

Import into … writes the table from the tab, with the catalog credential, as one commit. Keep the tab open until it finishes.

The files travel from the disk through the browser, so a large import is bound by the upload speed. The browser engine is single-threaded with a 4 GB address space. Large sets of files are better imported with the CLI.

Under Other ways to run this import, the dialog gives the same import three ways: as an lhbox table import command, as DuckDB statements, and as a prompt for an agent, with no credential in it.

Other formats and your own SQL

For JSON, other formats, or a transformation on the way in, use DuckDB with the catalog attached. lhbox duckdb opens a DuckDB shell with your catalog attached under its name, and never shows the credential. Write the whole file list in one statement, so it is one commit:

-- in `lhbox duckdb --catalog analytics`
CREATE SCHEMA IF NOT EXISTS analytics.events;
CREATE TABLE analytics.events.clicks AS
  SELECT * FROM read_json(['day1.jsonl', 'day2.jsonl']);
INSERT INTO analytics.events.clicks SELECT * FROM read_json('day3.jsonl');
  • A table DuckDB creates itself is format version 2, whatever the catalog's default. It does not get the 128 MiB file size or the sort order. How tables are laid out shows the CALL set_iceberg_table_properties and the ordered INSERT to add.
  • CREATE OR REPLACE TABLE is not supported on an Iceberg catalog. To reload a table, BEGIN; DELETE FROM t; INSERT INTO t SELECT …; COMMIT; is one commit: readers see the old rows until it lands, and a run that fails or is killed before COMMIT leaves them. Only a change of schema needs DROP TABLE, then CREATE TABLE (two commits; readers get "table does not exist" in between).
  • DuckDB's own CREATE TABLE … AS SELECT cannot make a geometry table. Create a version-3 table first, then insert into it with the spatial extension loaded:
lhbox table create analytics.geo.places --format-version 3 --column id:long --column geom:geometry
INSERT INTO analytics.geo.places SELECT id, ST_Point(lon, lat) FROM read_csv('places.csv');

The recipes for PyIceberg, Spark and the other engines are on the Engines page.

When it fails

what you see what it means what to do
table_exists (exit 2) --mode create and the table is there --mode append, or another name
columns_not_in_table the files have a column the table lacks nothing was written; match the files to the table's columns
exit 3 the credential is read-only ask for write on the catalog
mixed_formats CSV and Parquet in one list one run per format, --mode append for the second
unknown_format an extension the import does not know pass --format csv or --format parquet
import_outcome_unknown (exit 5) the commit's outcome is unclear run the lhbox table get it names before any re-run
geometry_crs_without_authority a version-3 import of a CRS with no code --format-version 2, or re-tag the CRS first
schema_mismatch add-files: the files do not share one schema, or do not fit the table fix the files; nothing was copied
files_have_field_ids add-files: the files were written by an Iceberg engine add them with that engine
geometry_crs_mismatch add-files: a file's CRS is not the table column's, case included rewrite the file's crs, or use a table with that CRS (geometry(OGC:CRS84) for lon/lat)
iceberg_core_not_found add-files into a version-3 table: the package is missing run the pip install it prints

Numbers and dates

what figure measured
three Parquet files (3,000 rows) imported as one commit, from a laptop 3.96 s; appended again in 1.21 s 2026-09-21
44 files loaded one statement each 44 commits and 44 waits 2026-09-20
add-files: 66 files (1.11 GB) as one commit 40 s, with a peak of 214 MB on the laptop 2026-10-20
a cold read of one row after that add-files read 1 of the 66 files and 170 KiB 2026-10-20
target data file size 128 MiB current
the browser import's engine single-threaded, 4 GB address space current