Lifecycle rules on the file bucket: expire files by age, abort stale uploads

A lifecycle rule deletes the files in a catalog's file bucket after a number of days or on a date. It can also abort uploads that were never finished. Rules apply to the file bucket only: table buckets never carry lifecycle rules.

lhbox catalog lifecycle set   garden --prefix camera/ --expire-days 90   # delete camera/ files 90 days after their last write
lhbox catalog lifecycle set   garden --abort-multipart-days 7            # a second rule; the first is kept
lhbox catalog lifecycle get   garden                                     # the rules as stored
lhbox catalog lifecycle clear garden                                     # remove every rule

Who can set a rule

Rules go on the catalog's file bucket, <handle>--<catalog>--blobs. Older catalogs have a b-<uuid> bucket instead, so read the name from the API's bucket field.

• The catalog credential at write level sets, reads and clears rules. Through the API, write sets and clears, and read is enough to read.
• An uploader key cannot. It may only put files under its prefix: GET, PUT and DELETE of the rules answer 403.
• A read-only credential cannot either, through S3. A reader must not be able to delete by rule.
• A rule applies to every file under its prefix, whoever wrote it. A 90-day rule on camera/ removes what the camera's uploader dropped there.

Which rules work

Every rule below was stored and read back unchanged. The ones marked honoured were traced to the code that runs them.

Not implemented, and staying so:

• x-amz-expiration on HEAD and GET: you cannot ask a file when it will go.
• AWS's refusal of Days: 0, midnight rounding and transitions.

With the API

The API takes a small shape and writes the XML for you. It writes as the tenant, with the catalog's own key pair, not an admin key, and then reads the result back.

PUT    /v1/warehouses/{id}/lifecycle   {"rules": [{"id"?, "prefix"?, "expire_days" | "expire_date",
                                                  "abort_incomplete_multipart_days"?, "enabled"?}, …]}   write level
GET    /v1/warehouses/{id}/lifecycle   [?include_xml=true]                                                read level
DELETE /v1/warehouses/{id}/lifecycle                                                                       write level

• PUT replaces the whole configuration, as S3 does.
• Ids default to <prefix without slashes>-<n>.
• A prefix takes letters, digits and . _ - / = @ :, as an uploader's does. No leading slash, and no * ? $.
• GET parses the stored document back, including rules set with boto3. Tags, size filters and other elements appear on the rule. expressible: false says a PUT here could not re-create it.
• The audit log records lifecycle.set and lifecycle.clear.

PUT refuses these before anything is stored:

With the CLI

The CLI merges. set reads the current rules, replaces the one with the same prefix (or the same --id) and puts them all back. So two set calls for two prefixes leave two rules, even prefixes that differ only in a slash, such as camera/ and camera.

• The default id is the prefix without slashes (camera), or bucket for the whole bucket. When another prefix's rule already has that id, the new rule gets the next free one (camera-2).
• set refuses to rewrite rules it cannot express, such as tag or size filters set with boto3. --replace-all overrides that.
• get --xml prints the stored document. Every command takes --json.

With boto3

Use boto3 for tag and size filters, or for an S3 workflow you already have. It needs the catalog credential (catalog_credential from lhbox connect). This is exactly what was measured:

import boto3
from botocore.config import Config
s3 = boto3.client("s3", endpoint_url="https://s3.lakehousebox.com", region_name="us-east-1",
                  aws_access_key_id=CLIENT_ID, aws_secret_access_key=CLIENT_SECRET,  # catalog_credential from `lhbox connect`
                  config=Config(s3={"addressing_style": "path"}))
s3.put_bucket_lifecycle_configuration(
    Bucket="<handle>--<catalog>--blobs",
    LifecycleConfiguration={"Rules": [
        {"ID": "camera-90d", "Status": "Enabled", "Filter": {"Prefix": "camera/"}, "Expiration": {"Days": 90}},
        {"ID": "abort-stale-uploads", "Status": "Enabled", "Filter": {"Prefix": ""},
         "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7}},
    ]})
print(s3.get_bucket_lifecycle_configuration(Bucket="<handle>--<catalog>--blobs")["Rules"])
# s3.delete_bucket_lifecycle(Bucket=...) removes every rule

When a rule takes effect

The rules run in one pass once a day, and about five minutes after every restart of the store.

• The pass compiles every bucket's rules. It replays the write log for Days and multipart rules, and walks the buckets that have Date rules.
• A file is deleted at the first pass after it is due: up to 24 h after Days elapses.
• A day is 24 h from the file's last write. A file overwritten since the rule saw it is left alone.
• Deletion is permanent. Buckets are not versioned, so there is no undelete.
• A failed delete stalls the pass. It is retried at the next pass.

The storage figure (GET /v1/usage, the catalog's storage) shows the reduction at its next reading, every few minutes. To see it at once, run lhbox catalog measure <catalog> (POST /v1/warehouses/{id}/measure).

Never on the table bucket

The catalog's own maintenance keeps Iceberg data and metadata: snapshot expiry (168 h / 20 snapshots) and orphan cleanup (files no snapshot has referenced for 24 h). Key age plays no part. A lifecycle rule on a table bucket would delete Parquet and metadata.json files the catalog still references, and corrupt every table under it.

The store does not tell table buckets apart: the admin identity could set a rule there, and the daily pass would run it. So the guard is ours, in three layers:

1. No tenant path reaches it. The tenant's user policy carries the five data-path actions. PutUserPolicy refuses s3:PutLifecycleConfiguration and s3:GetLifecycleConfiguration as "not a valid action" (400). The table-bucket policy names s3tables:* and the data path. ?lifecycle on a table bucket answers 403 to the tenant, for GET and PUT.
2. The control plane refuses to grant it. It refuses any policy naming a lifecycle action (s3:*Lifecycle*, s3:*, s3:Put*) on a bucket that is not a file bucket. That means a bucket not shaped like one, or one the database knows as a table bucket: a catalog named blobs has the table bucket <owner>--blobs.
3. The audit. GET /internal/lifecycle-audit (bootstrap secret) reads ?lifecycle and ?policy on every table bucket as the admin. It lists anything but a 404 and no lifecycle grant. The nightly sweep is to call it and alert on ok: false.

Check it end to end

# a QA catalog and its write-level key (lhbox login / an org admin key in LHBOX_API_KEY)
lhbox catalog create qa_lifecycle
lhbox catalog lifecycle get qa_lifecycle                       # None configured: 0 rules
lhbox catalog lifecycle set qa_lifecycle --prefix camera/ --expire-days 90 --abort-multipart-days 7
lhbox catalog lifecycle get qa_lifecycle --xml                 # the stored document
# the same through boto3 with the catalog credential from `lhbox connect --json`: put/get as above; 200/200
# the table bucket stays closed: as the tenant, GET ?lifecycle on <handle>--qa-lifecycle must be 403
lhbox catalog lifecycle clear qa_lifecycle                     # Cleared 1 lifecycle rule
lhbox catalog delete qa_lifecycle --confirm qa_lifecycle

The mechanism, for the record

No user policy can carry the lifecycle actions. ?lifecycle resolves to s3:GetLifecycleConfiguration or s3:PutLifecycleConfiguration, and DELETE uses the Put action. So a tenant could not set a rule at all.

What opens them is a plain bucket policy on the file bucket, naming the identity as an IAM-user ARN:

• Plain-bucket principals are ARNs (arn:aws:iam::<account>:user/<name>), so arn:aws:iam::*:user/t-<uuid> matches. A bare name stays 403.
• A bucket-policy Allow bypasses the IAM check.
• PutBucketPolicy is an S3 call: no IAM write, no propagation deadline.

The control plane writes this one statement at these moments, each time idempotent:

• when a catalog is created;
• when an older catalog gets its file bucket;
• before every lifecycle request (one GET);
• once for every existing file bucket, at the first startup after the deploy. A background migration does it, and retries at the next start if anything failed.

{"Version": "2012-10-17", "Statement": [{
  "Sid": "TenantLifecycle", "Effect": "Allow",
  "Principal": {"AWS": ["arn:aws:iam::*:user/t-<warehouse_uuid>"]},
  "Action": ["s3:GetLifecycleConfiguration", "s3:PutLifecycleConfiguration"],
  "Resource": ["arn:aws:s3:::<handle>--<catalog>--blobs"]}]}

A public catalog's public-read policy lives on the table bucket, so the two documents never share a bucket. The grant is merged by Sid all the same. Deleting the catalog deletes the file bucket with its policy and its rules.

Numbers and dates

Measured 2026-09-21, 05:50–06:10 UTC, against the live service (api.lakehousebox.com, s3.lakehousebox.com) with a fresh QA account.