For agents · the instructions your agent follows. The page for people: Your data
How LakehouseBox stores your data
This page says where your bytes live, how they are protected and how they come back. It was written 2026-09-22 from the code as deployed and from our measurements. Updated 2026-10-28: since that day the managed service runs as one container in Frankfurt, Germany, and every catalog's files live in object storage in Frankfurt, Germany, with a full replicated copy in Stockholm, Sweden (The managed service). Updated 2026-10-30: the earlier hosting, its copies and its backups are deleted. The hot and cold sets below are how the store keeps bytes on its own disks: a self-hosted install.
One process, two buckets per catalog
The store is our own build. The Iceberg REST catalog, the S3 gateway, the metadata index and the volume store run as one process (on the managed service, inside one container with the control plane and the MCP server).
Every catalog you create is two S3 buckets:
- a table bucket, which enforces the Iceberg layout;
- a blob bucket, for files.
Each bucket is its own collection of 256 MB volumes: the append-only files the store packs objects into.
A catalog's isolation is enforced at the catalog (a bucket policy naming its identity). Every engine gets per-table storage credentials that expire. Access and permissions and the API reference describe that model, and this page does not repeat it.
engines (DuckDB, PyIceberg, Spark, ...)
|
| S3 + Iceberg REST
v
the store: one container in Frankfurt, Germany
(catalog · S3 · metadata · control plane; its disk is a cache)
|
+-- every write, within seconds --> every catalog's files: object storage, Frankfurt, Germany
| | replicated
| v
| a full copy: Stockholm, Sweden
|
+-- every 5 minutes, encrypted ---> the catalog's state (database, metadata, keys; the newest 144)
separate storage, Frankfurt, Germany
| replicated
v
a copy: Stockholm, SwedenThe managed service
Since 2026-10-28 the store keeps every catalog's buckets in object storage in Frankfurt, Germany, replicated to a full second copy in Stockholm, Sweden. The container's own disk is a cache: a write is accepted there and copied to the storage within seconds, and a read the cache does not hold is fetched from the storage. The replicated copy protects against the loss of a site, not against a mistake: a deletion or an overwrite reaches both copies. A table's recent versions (Time travel) are the protection against a mistaken change.
Hot and cold: where the bytes are (self-hosted)
This section describes the store on its own disks: a self-hosted install. The managed service does not tier: its files are in object storage from the start (The managed service).
New writes always land on the host's disk. It is the hot set.
A catalog's volumes can go quiet: no write for the quiet period (AL_TIER_QUIET_FOR, default 7 days). Its sealed volumes are then moved to an S3-compatible object store, one file each. The host keeps only the small index of each moved volume. This is the cold set, and it is why storage stays cheap: object storage usually costs less per gigabyte than a disk.
Three rules follow from how the store works:
- Cold reads are slower on first access. Nothing is cached. Every read of a cold object is a range request to the object store, about 0.2 s to the first byte per file where we measured it. The first scan of a cold table measured 7–8× slower than a local one. Later scans in the same session are as fast as local, because the engines cache (§2.4, §6.5).
- New writes never touch a cold volume. A moved volume is read-only, and the catalog's next write opens a fresh local volume. Active catalogs therefore stay hot on their own.
tier_policyis per catalog.cold, the default, allows tiering.hotkeeps a catalog on the disk for good. The demo catalog and public catalogs are alwayshot. Set it withlhbox catalog update <name> --tier hot|cold.
The tiering job is a timer on the host. It runs after maintenance (snapshot expiry and compaction), because bytes deleted from a volume after it is tiered are never reclaimed (§2.7). It is installed with the platform and enabled by the operator. Until it is enabled, every byte is hot. Once the tier has moved something, lhbox usage shows cold_bytes per catalog.
Encryption at rest
Every chunk is encrypted before it reaches a volume (AL_ENCRYPT_VOLUME_DATA, on by default). The per-chunk keys live in the catalog's metadata, not next to the data. Both the disk and the cold copies in object storage therefore hold ciphertext: a downloaded volume file contained no Parquet at all (§6.2). Measured cost: none on writes, about 3 ms per local read.
Chunks written before the flag was on stay plaintext until compaction rewrites them. The metadata copy is the keyring, so it is encrypted before it leaves the host (Recovery).
On the managed service this applies to the container's disk. The copy in object storage holds each file as it was written, reached only with that storage's own keys; we do not add our own encryption to it.
Recovery
On the managed service (since 2026-10-28):
- The data is in object storage in Frankfurt, Germany, with its replicated copy in Stockholm, Sweden; there is no separate copy of the data files.
- The catalog's state, every 5 minutes: the control plane's database, the store's metadata, configuration and keys, in one encrypted archive, written to separate storage in Frankfurt, Germany, replicated to Stockholm the same way, and read back after upload. It is taken only once every write has reached the storage. The newest 144 are kept (about 12 hours).
- If the container stops, it is started again. If it loses its disk, it is rebuilt from the latest archive and the storage: the catalogs come back as they were, rehearsed before the move.
The hosting and backups used before 2026-10-28 are deleted (2026-10-30). A self-hosted install keeps the store's own backups: the metadata and the hot set every hour, encrypted, a full archive weekly, to an S3-compatible store of its choice (the self-hosted page).
Not there yet, labelled roadmap: serving from a second location, and a standby.