# Connected catalogs: read data where it lives

A connected catalog is data another system keeps, attached to your organisation under a name and read in SQL beside your own catalogs. Files in an S3-compatible bucket, a Google Cloud Storage bucket, an Azure container, the tables of another Apache Iceberg catalog, a Google BigQuery project, a Portolan catalog or open data in a public bucket: each becomes <name>.<schema>.<table>, and a query reads it where it lives. Nothing is copied, nothing is written back, and there is no storage to pay for: the bytes stay in the other system.

It takes two things:

- a connection: where the data is and the key that reads it (or none, for public data). It belongs to the person or token who made it, and its secret is never shown again, to anyone.

- a connected catalog (a source in the API and the CLI): a connection attached under a name. Its tables are found by looking at the files once, and are read like any catalog's.

## The short path

```
lhbox connection create overture --anonymous --endpoint s3.us-west-2.amazonaws.com --region us-west-2 --bucket overturemaps-us-west-2 --prefix release/2026-11-18.0
lhbox source attach overture --connection overture --everyone      # admin: attach it; every member reads it
lhbox source tables overture                                       # what was found, as SQL names it
lhbox query "SELECT count(*) FROM overture.places.place"           # read where it lives
```

On the account page, Connect data does the same in one dialog: choose the kind of system, say where it is, give a key or choose No key for a public bucket, then Connect and look at the files. The connected catalog appears beside your catalogs with its tables; Look again reads its files anew.

Agents read connected catalogs through the same tools as any catalog: the MCP server's list_tables lists them (kind connected, each table with its format and file count), describe_table gives a table's columns, and query reads them. lhbox query and the account page's SQL do too.

## How it reads, and where your key is

The key you give is stored sealed and never leaves our side: no query, agent or person ever holds it, and no route answers it back. Each query that reads a connected catalog gets a key of its own, valid only for that query (15 minutes at most), only for that connection's bucket and folder, and only for a bounded amount of reading. Every read passes through a gateway of ours that checks the query's key and signs the request to the other system with yours. Give a key that can only list and read: LakehouseBox never writes, but the key is the other system's to scope.

## Kinds of system

| kind | --type | key | what becomes a table |
|---|---|---|---|
| S3-compatible bucket (AWS, R2, Scaleway, OVH, MinIO…) | s3 (default) | access key pair | each folder of Parquet, CSV or JSON files |
| Google Cloud Storage | s3 | HMAC key | the same |
| Public bucket (open data) | s3 with --anonymous | none | the same |
| Azure Blob Storage / ADLS Gen2 | azure_blob | Entra service principal | the same |
| Iceberg REST catalog | iceberg_rest | OAuth2 client or token, plus a storage key | each Iceberg table |
| Another LakehouseBox catalog | lakehousebox (public) or iceberg_rest | none, or its recipe | each Iceberg table |
| Portolan catalog | portolan | none (public), or a key pair | each collection of GeoParquet files |
| Google BigQuery | bigquery | none: you grant a service account we make | each table of its datasets |

GET /v1/source-types describes each kind with its fields, as the account page reads it.

### S3-compatible buckets

```
export SRC_KEY_ID=…          # a key that can list and read the bucket (or the folder)
export SRC_SECRET=…
lhbox connection create exports --endpoint s3.eu-central-1.amazonaws.com --region eu-central-1 --bucket acme-exports --prefix daily/ --access-key-id-env SRC_KEY_ID --value-env SRC_SECRET
lhbox source attach exports --connection exports
```

The secret is read from an environment variable (--value-env), a file (--value @secret.txt) or standard input (--value @-), never from the command line itself. --prefix is a folder: only keys under it are ever read.

| store | --endpoint | --region |
|---|---|---|
| Amazon S3 | s3.<region>.amazonaws.com | the bucket's region, e.g. eu-central-1 |
| Cloudflare R2 | <account-id>.r2.cloudflarestorage.com | auto |
| Scaleway Object Storage | s3.<region>.scw.cloud | fr-par, nl-ams or pl-waw |
| OVHcloud Object Storage | s3.<region>.io.cloud.ovh.net | e.g. gra |
| MinIO, or any S3-compatible store | its host name (https) | whatever it expects (default us-east-1) |

The endpoint is a host name reached over https, never an IP address. Requests use path-style addresses (--url-style path, the default); --url-style vhost puts the bucket in the host name for a store that wants it.

### Google Cloud Storage

Cloud Storage answers the S3 protocol at storage.googleapis.com. Read a private bucket with an HMAC key of a service account that has Storage Object Viewer on the bucket (Cloud Storage → Settings → Interoperability → Create a key for a service account):

```
export GCS_ACCESS_ID=…       # the HMAC key's access id
export GCS_SECRET=…          # its secret
lhbox connection create gcs_exports --endpoint storage.googleapis.com --region auto --bucket acme-exports --prefix daily/ --access-key-id-env GCS_ACCESS_ID --value-env GCS_SECRET
lhbox source attach gcs_exports --connection gcs_exports
```

A public Cloud Storage bucket needs no key: --anonymous in place of the key (read this way on 2026-11-20). On the account page, the Google Cloud Storage tile fills in the endpoint and region, and a gs://bucket/folder/ address pasted into Bucket and folder fills in the rest.

### Public buckets

Open data needs no key: --anonymous reads with unsigned requests, as anyone on the internet can. Overture Maps' release, for example, has one table per theme and type (places.place, buildings.building, transportation.segment, …):

```
lhbox connection create overture --anonymous --endpoint s3.us-west-2.amazonaws.com --region us-west-2 --bucket overturemaps-us-west-2 --prefix release/2026-11-18.0
lhbox source attach overture --connection overture --everyone
lhbox source tables overture
```

Then, places in central Madrid by category. Filter on bbox, so that only the files and row groups for Madrid are read (measured on 2026-11-19 against the 2026-11-18.0 release: about 7 s):

```
SELECT taxonomy.primary AS category, count(*) AS places
FROM overture.places.place
WHERE bbox.xmin BETWEEN -3.72 AND -3.68 AND bbox.ymin BETWEEN 40.40 AND 40.43
GROUP BY 1 ORDER BY 2 DESC LIMIT 20;
```

and the roads crossing a small box (about 13 s):

```
SELECT class, count(*) AS segments
FROM overture.transportation.segment
WHERE bbox.xmin < -3.69 AND bbox.xmax > -3.71 AND bbox.ymin < 40.42 AND bbox.ymax > 40.41
GROUP BY 1 ORDER BY 2 DESC;
```

A query over the largest themes (buildings.building, 512 files) needs more than the 20 seconds a server-side query has: read those with DuckDB on your own machine, straight from the public bucket, or point the connection at a narrower folder.

A new release is a new folder: make another connection for it, or keep the one you have.

### Azure Blob Storage and ADLS Gen2

Read a container (an ADLS Gen2 file system works the same) with a Microsoft Entra app registration that holds the Storage Blob Data Reader role on the storage account or the container:

- Entra ID → App registrations → New registration. Note its Directory (tenant) ID and Application (client) ID.

- Certificates & secrets → New client secret. Copy the value.

- The storage account (or the container) → Access control (IAM) → Add role assignment → Storage Blob Data Reader → the app.

```
export AZ_SECRET=…           # the client secret's value
lhbox connection create sales --type azure_blob --account contosodata --container sales --prefix exports/ --tenant-id <tenant-id> --client-id <client-id> --value-env AZ_SECRET
lhbox source attach sales --connection sales
```

Azure's global cloud (core.windows.net) is the one supported.

### Iceberg REST catalogs

The tables of another Apache Iceberg REST catalog (Polaris, Gravitino, Lakekeeper, Nessie and others), read at their current snapshot. The connection holds two keys: one for the catalog (an OAuth2 client, a bearer token, or none for a public catalog) and one for the bucket the tables' files are in (or none, for a public bucket):

```
export CAT_SECRET=…          # the OAuth2 client's secret
export STORE_KEY_ID=…        # a storage key that can read the tables' files
export STORE_SECRET=…
lhbox connection create lake --type iceberg_rest --catalog-uri https://catalog.example.com/api/catalog --warehouse analytics --client-id lhbox-reader --catalog-secret-env CAT_SECRET --namespace sales --namespace finance --endpoint s3.eu-central-1.amazonaws.com --region eu-central-1 --bucket acme-lake --access-key-id-env STORE_KEY_ID --value-env STORE_SECRET
lhbox source attach lake --connection lake
```

- --catalog-uri is the catalog's REST address without /v1. --warehouse names the warehouse it serves you, when it serves several.

- --auth oauth2 (the default) mints tokens at <catalog path>/v1/oauth/tokens (--token-path for another path on the same host, --scope when it wants one); --auth bearer takes a token as --catalog-secret-env; --auth none reads a public catalog.

- --namespace (repeat it, or separate with commas) keeps the reads to those namespaces and what is under them; without it, every namespace is read.

- The catalog's own vended credentials are never used: its tables' files are read with the storage key you give, from that bucket only.

### Another LakehouseBox catalog

Another organisation's public catalog, by its name, with no key:

```
lhbox connection create pubdata --type lakehousebox --catalog commons/pubdata
lhbox source attach pubdata --connection pubdata --everyone
```

A private catalog you can read in another organisation is an Iceberg REST connection with that catalog's recipe: its key pair is both the catalog's client and the storage key. lhbox connect --catalog <name> --readonly (run by someone who can read it) prints the recipe; use its bucket and key pair:

```
lhbox connection create partner --type iceberg_rest --catalog-uri https://catalog.lakehousebox.com --warehouse <bucket> --client-id <key id> --catalog-secret-env PARTNER_SECRET --endpoint s3.lakehousebox.com --bucket <bucket> --access-key-id <key id> --value-env PARTNER_SECRET
```

### Portolan catalogs

A published Portolan catalog (a STAC index of GeoParquet collections): each collection with GeoParquet files becomes a table over exactly those files.

```
lhbox connection create madrid --type portolan --catalog-url https://<host>/<bucket>/<folder>/catalog.json
lhbox source attach madrid --connection madrid
```

The address is path style (https://host/bucket/folder/catalog.json). --data-url https://host/bucket/folder/ says where the GeoParquet files are when they are not under the catalog's folder. A public catalog needs no key; a private one takes --access-key-id-env and --value-env like an S3 bucket.

### Google BigQuery

The tables of a Google Cloud project's BigQuery datasets, read where they are. You give no key: LakehouseBox makes a Google service account for the connection, and you grant it read access in your project. Delete the connection and the account is deleted with it.

```
lhbox connection create warehouse --type bigquery --project acme-analytics --dataset sales --dataset marketing
```

The answer names the account (lhbq-…@lhbox-connections.iam.gserviceaccount.com) and prints the two grants it needs:

- BigQuery Data Viewer (roles/bigquery.dataViewer) on each dataset it reads (on the project, when it reads every dataset);

- BigQuery Read Session User (roles/bigquery.readSessionUser) on the project the reads are billed to.

```
bq query --project_id=acme-analytics --nouse_legacy_sql 'GRANT `roles/bigquery.dataViewer` ON SCHEMA `acme-analytics`.sales TO "serviceAccount:lhbq-…@lhbox-connections.iam.gserviceaccount.com"'
gcloud projects add-iam-policy-binding acme-analytics --member=serviceAccount:lhbq-…@lhbox-connections.iam.gserviceaccount.com --role=roles/bigquery.readSessionUser
lhbox source attach warehouse --connection warehouse
lhbox source refresh warehouse          # once granted: its tables, as <name>.<dataset>.<table>
lhbox query "SELECT count(*) FROM warehouse.sales.orders"
```

lhbox connection show warehouse prints the account and the commands again. On the account page, Connect data → Google BigQuery shows them after the connection is made, with Granted: look again.

- --dataset (repeat it, or separate with commas) keeps the reads to those datasets, whatever else the account is granted; without it, every dataset the account can read. --billing-project names another project for the reads' cost (the Storage Read API's), when it is not the same.

- Reads use the BigQuery Storage Read API: tables are read, views are not (they are left out of its tables). No query job ever runs in your project, so nothing is billed as a query; Storage Read bytes are billed to the billing project by Google.

- A statement that reads a BigQuery connected catalog reads nothing else: join it with another catalog in separate statements, or keep the result in a catalog of yours.

- Taking over a stopped BigQuery connection keeps its account (your grants are to it); it holds no key to replace.

- The account can only be acted as by our gateway, for one query at a time; the part of LakehouseBox that makes and deletes these accounts cannot act as them.

## How files become tables

For files (S3, Cloud Storage, Azure), the folder under the connection's prefix is looked at once when the connected catalog is attached:

- Each folder of data files of one format is one table: Parquet (.parquet, .parq, .pq), CSV or TSV (.csv, .tsv, also gzip- or zstd-compressed) and JSON (.json, .jsonl, .ndjson, also gzip-compressed).

- Hive partitions (year=2026/month=11/) are read as columns of the table in the folder above them.

- Published layouts with dataset folders (theme=places/type=place/) become one table per dataset, named by the values (places.place).

- The first folder is the schema when there are two levels or more; otherwise it is main. Names are lower case letters, digits and _; a clash gets a number.

A table reads every file of its folder at query time, so new files in a known folder are read without looking again. A new folder (a new table) appears after Look again on the account page, or:

```
lhbox source refresh overture
```

Iceberg tables are read at their current snapshot on every query: new commits need no refresh, new tables do.

## Limits

| a query | 20 seconds of compute, as any query; filter and aggregate in SQL |
| reading per query | a key of its own, for 15 minutes at most, a bounded number of reads and bytes |
| looking at the files | the first 5,000 files under the prefix; 200 tables per connected catalog |
| a Portolan collection | 2,000 files at most; a larger one is left out, not cut |
| connections, connected catalogs | 50 each per organisation |
| the endpoint | a host name over https, never an IP address or a private network |

When looking at the files stops at one of these, the answer says so (more_files, more_tables): narrow the prefix, or make a connection per folder.

## Who may do what

- Connections. Any member may create one, or only admins: the organisation's policy, lhbox org policy --create-connections admins (see what members may do (https://lakehousebox.com/docs/access/#what-members-may-do)). A connection belongs to its maker, a person or a token, and is private to them: besides its owner and the admins, only those it is shared with may use it, that is attach connected catalogs from it.

```
lhbox connection share exports --everyone             # every member may attach from it
lhbox connection share exports --principal <id>       # one person or token
lhbox connection unshare exports --principal <id>
lhbox connection rotate exports --access-key-id-env SRC_KEY_ID --value-env SRC_SECRET   # a new key, in place
lhbox connection transfer exports --to <id>
```

Its owner or an admin replaces its key, shares, transfers or deletes it; an admin sees every connection. A connection that connected catalogs use is not deleted until they are detached.

- Connected catalogs. Attached by whoever may use the connection; managed (opened, granted, looked at again, detached) by an admin or the person who attached it. Reading one never needs the connection: an admin reads every connected catalog; a member reads one opened to the organisation; a person, token or agent reads one it was granted. A new connected catalog is Restricted unless attached with --everyone.

```
lhbox source access exports everyone                  # every member reads it
lhbox source access exports restricted                # admins and those granted it
lhbox source grant exports --principal <id>           # a member, a token or an agent
lhbox source revoke exports --principal <id>
lhbox source detach exports                           # its grants go; the connection stays
```

- When the owner leaves. A connection whose owner leaves the organisation (removed, left, deprovisioned by the directory, or a revoked token) stops: nothing reads with a departed person's key, and its connected catalogs answer connection_stopped until an admin takes it over with a key of their own:

```
lhbox connection take-over exports --access-key-id-env SRC_KEY_ID --value-env SRC_SECRET
lhbox connection take-over lake --catalog-secret-env CAT_SECRET --access-key-id-env STORE_KEY_ID --value-env STORE_SECRET   # an Iceberg REST catalog
```

take-over and rotate take the same secret, by the connection's type: a key pair for a bucket; the client secret alone (--value-env) for an Azure connection; for an Iceberg REST catalog its client secret or token (--catalog-secret-env) together with the storage key pair, which is left out when its bucket is public.

- History. Every connection and connected catalog change (created, shared, key replaced, taken over, transferred, deleted, attached, opened, granted, looked at again) is in the organisation's audit log, by name. The secret never appears in it.

## Terraform and OpenTofu

The provider (LakehouseBox/terraform-provider-lakehousebox, the same setup in OpenTofu or Terraform (https://lakehousebox.com/docs/migrations/)) has lakehousebox_connection (S3-compatible buckets, Cloud Storage and public buckets; the key is write-only and replaced in place) and lakehousebox_source:

```
resource "lakehousebox_connection" "overture" {
  org       = "acme"
  name      = "overture"
  anonymous = true
  endpoint  = "s3.us-west-2.amazonaws.com"
  region    = "us-west-2"
  bucket    = "overturemaps-us-west-2"
  prefix    = "release/2026-11-18.0"
}

resource "lakehousebox_source" "overture" {
  org             = lakehousebox_connection.overture.org
  name            = "overture"
  connection_name = lakehousebox_connection.overture.name
  org_access      = "read"
}
```

## The API

| GET /v1/source-types | the kinds of system and their fields |
| POST /v1/orgs/{org}/connections | create a connection (type and its fields, the secret write-only) |
| GET /v1/orgs/{org}/connections[/{name}] | list, or one (never the secret) |
| POST /v1/orgs/{org}/sources | attach {"name", "connection"} |
| GET /v1/orgs/{org}/sources/{name}/tables | its tables, as SQL names them |
| POST /v1/orgs/{org}/sources/{name}/refresh | look at its files again |

Every route, with the rest (sharing, grants, take-over): the API (https://lakehousebox.com/docs/api/).

## Errors

| code | status | means |
|---|---|---|
| not_enabled | 403 | connected catalogs are turned off for this organisation |
| org_policy_forbids | 403 | the organisation lets only admins create connections |
| insufficient_role | 403 | that takes an admin (or the person who attached it) |
| not_connection_owner | 403 | only its owner or an admin may do that to a connection |
| invalid_value | 400 | a field the kind refuses; field names it |
| reserved_name, name_is_a_catalog | 400, 409 | the name is taken by a catalog or reserved |
| already_exists | 409 | a connection or connected catalog has that name |
| limit_exceeded | 409 | 50 connections or 50 connected catalogs |
| connection_in_use | 409 | detach its connected catalogs first (sources lists them) |
| connection_stopped | 409 | its owner left: an admin takes it over |
| anonymous_connection | 409 | a public connection has no key to replace |
| no_key | 409 | a BigQuery connection reads as its own account: change its grants in Google Cloud instead |
| connection_not_found, source_not_found | 404 | no such name you can see |
| source_unavailable | 502 | the other system refused or did not answer: its message is in the error |
| sources_unavailable | 503 | retry shortly |

---
HTML version: https://lakehousebox.com/docs/connected-catalogs/ · every page: https://lakehousebox.com/llms.txt · everything in one file: https://lakehousebox.com/llms-full.txt

Docs feedback: if this page was wrong, out of date or missing something you needed, tell the person you work for and, if they agree, report it: `lhbox bug --title "docs: connected-catalogs: <what was wrong>"` prints a filled-in issue link, or open https://github.com/lakehousebox/lakehousebox/issues/new. The tracker is public: never include a key, a secret or the person's data.
