How LakehouseBox stores your data

Published from the repository note platform/docs/architecture.md; the numbers come from research/27-storage-tiering-experiment.md (2026-09-21) and are cited by section. Operator switches (the tiering timer, the mirror) are named as such: what they do is in the code, whether they are on is the operator's call.

Written 2026-09-22 from the code as deployed and from the measurements in research/27-storage-tiering-experiment.md (cited by section). One host in Germany today; the pieces below are what an engineer evaluating the service needs to know about where bytes live, how they are protected and how they come back.

One process, two buckets per catalog

The store is SeaweedFS (weed mini, our own build): the Iceberg REST catalog, the S3 gateway, the filer (the metadata index) and the volume store run as one process on one Hetzner server. Every catalog you create is two S3 buckets — a table bucket that enforces the Iceberg layout and a blob bucket for files — and each bucket is its own collection of 64 MB volumes, the append-only files SeaweedFS packs objects into. A catalog's isolation is enforced at the catalog (a bucket policy naming its identity) and every engine gets per-table storage credentials that expire; that model is described in the repository's security note (platform/docs/security.md) and the API reference, and is not repeated here.

  engines (DuckDB, PyIceberg, Spark, ...)  --S3 + Iceberg REST-->  weed mini on al-lake1
                                                                     |  catalog · S3 · filer · volumes
                                                                     |
                             hot set: Hetzner Cloud Volume  <--------+--------> cold set: Hetzner Object Storage
                             (replicated by Hetzner on 3 servers)              (sealed 64 MB volume files, Falkenstein)

Hot and cold: where the bytes are

New writes always land on the host's Hetzner Cloud Volume, a network block device Hetzner replicates across three physical servers. That is the hot set. A catalog whose volumes have had no write for the quiet period (AL_TIER_QUIET_FOR, default 7 days) has its sealed volumes moved, one file each, to Hetzner Object Storage in Falkenstein; the host keeps only the small index of each moved volume. This is the cold set, and it is why storage stays cheap: Object Storage costs about a ninth of a Volume per gigabyte (§2.11).

Three rules follow from how the store works:

  • Cold reads are slower on first access. Nothing is cached: every read of a cold object is a range request to Object Storage, about 0.2 s to the first byte per file; the first scan of a cold table measured 7–8× slower than a local one, later scans in the same session are as fast as local because the engines cache (§2.4, §6.5).
  • New writes never touch a cold volume. A moved volume is read-only; the catalog's next write opens a fresh local volume. Active catalogs therefore stay hot on their own.
  • tier_policy is per catalog. cold (the default) allows tiering; hot keeps a catalog on the Volume for good. The demo catalog and public catalogs are always hot. Set it with lhbox catalog update <name> --tier hot|cold.

The tiering job is a timer on the host that runs after maintenance (snapshot expiry and compaction), because bytes deleted from a volume after it is tiered are never reclaimed (§2.7). It is installed with the platform and enabled by the operator; until it is enabled every byte is hot. cold_bytes per catalog appears in lhbox usage when the tier has moved something.

Encryption at rest

Every chunk is encrypted before it reaches a volume (AL_ENCRYPT_VOLUME_DATA, on by default). The per-chunk keys live in the catalog's metadata, not next to the data, so both the Volume and the cold copies in Object Storage hold ciphertext: a downloaded volume file contained no Parquet at all (§6.2). Measured cost: none on writes, about 3 ms per local read. Chunks written before the flag was on stay plaintext until compaction rewrites them. The metadata copy is therefore the keyring, and it is encrypted before it leaves the host (below).

Recovery

  • Every hour the metadata — the filer index, every volume's index, the store's configuration and its keys — is archived, encrypted to a key the host does not hold, and copied off-host. From that archive alone a fresh host brings back every cold catalog (the bytes are in Object Storage; the drill took 1.5 s to serve, §2.8). Hot catalogs need the next item.
  • Every night the hot set is archived off-host with the store stopped for a few seconds.
  • If the host dies and its Volume survives, the Volume is attached to a new server and the store starts there: measured 25 s from stop to serving, plus provisioning and DNS (§2.9). This is the first move, before any restore.
  • Control-plane state (accounts, keys, grants) is copied off-host every 5 minutes.

Not there yet, labelled roadmap: automatic failover (a host failure is minutes of outage handled by a person or the operations agent), a second location (all of this runs in one Hetzner datacentre; the cold set can be mirrored nightly to another location when that timer is enabled), and a hot standby.

The limits that shaped this

Hetzner Object Storage allows 100 buckets per account, 750 requests per second and 100 TB per bucket, 256 parallel connections per source address, and has no cross-location replication or published durability figure (§6.1). So the cold tier is one bucket per group of roughly 500 catalogs rather than one per catalog, the store speaks HTTP/1.1 to it with a capped connection pool (HTTP/2 there was measured at a tenth of the throughput, §6.5), one server serves about 150 cold requests per second before CPU binds (§6.6), and a nightly mirror of each cold bucket to another Hetzner location is the answer to single-datacentre durability. Growing the hot set is a Volume resize, online (§2.6).