Every few years, the data world moves the same piece: the engine, the part that does the calculating. Twenty years ago it had to sit right next to the data, on machines you owned and ran. Today it can sit inside an AI agent, on a laptop or in the cloud, and come and go with each question.
This post traces that journey in four steps and ends with a simple question: if the engine now travels with the agent, what does the data need?
Read from top to bottom: at each step the engine became easier to swap, and the data became more independent of any one tool.
2006: big data on cheap hardware
In the early 2000s, analysing large amounts of data meant buying a data warehouse: specialised machines and software from a handful of vendors, priced accordingly. Only large organisations could afford one.
Google changed the recipe. In two papers, on the Google File System (2003) and MapReduce (2004), it described how to store and process enormous datasets across many ordinary computers, and how to keep going when some of them failed. In 2006 those ideas became Hadoop, an open-source project that Yahoo! took into large-scale production.
That is what "big data" meant at the start: not only big volumes, but big volumes on affordable hardware, without an expensive warehouse. The catch was that the engine and the data lived on the same cluster. To analyse the data you had to buy, run and maintain that cluster yourself, and you needed specialists to do it.
2012: the cloud splits storage from compute
The cloud first made storage cheap and elastic: Amazon S3, launched in 2006, let anyone keep files without owning a single disk. Then came warehouses you could rent instead of buy. Amazon Redshift was previewed in November 2012, and Snowflake, founded that same year and launched in 2015, built its whole product around one idea: keep the data in one place and switch on computing power only when you need it.
Separating storage from compute changed the economics. You could keep years of data cheaply, pay for heavy processing only while queries ran, and scale each side on its own.
One thing did not change: the data sat in each vendor's own format. Your tables lived inside Redshift or Snowflake, and in practice only that product's engine could read them. The engine was rented, but from a single landlord.
2018: open tables, many engines
The next step was to take the data out of the landlord's house. Engineers at Netflix started Apache Iceberg in 2017 and gave it to the Apache Software Foundation in 2018. Around the same time Databricks released Delta Lake, and Uber had built Apache Hudi.
These are open table formats. They describe, in a public specification, how a table is laid out in ordinary files: its columns, its rows, every change made to it. Any engine that follows the specification can read and write the table, so several engines can work safely on the same data at the same time. The industry started calling this combination of cheap storage and open tables a lakehouse.
In June 2024 the question of which format would win was largely settled. In the same week, Databricks bought Tabular, the company founded by Iceberg's creators, and Snowflake announced Polaris, an open-source catalog for Iceberg tables (Starburst's summary). The two largest rivals in the market were both betting on the same open format.
Even the big warehouses started to change sides. Snowflake, for example, can now read and write Iceberg tables kept in someone else's catalog. The warehouse, once the place where your data lived, is becoming one more engine you point at your data.
This is the era of interoperability. The data no longer belongs to an engine. You choose the engine for the job, and change it later without moving the data.
Now: the engine travels with the agent
Two things have come together in the last couple of years.
The first is that engines have become small. DuckDB, created at the CWI research institute in Amsterdam and first released in 2019, reached version 1.0 in June 2024. It is an analytical database that runs inside another program, with no server to install, and it handles millions of rows on an ordinary laptop.
The second is that AI agents now run code. An agent such as Claude, Codex or Cursor can write a query, start an engine like DuckDB, run it and read the result, all as part of answering one question. Since Anthropic released the Model Context Protocol in November 2024, agents also have a standard way to connect to outside data sources.
Put the two together and the engine stops being a place you go to. It becomes something the agent brings along. A question arrives, the agent starts an engine where it is running, fetches the data it needs, computes the answer and moves on. The compute is already paid for: it is the machine the agent runs on.
Sometimes the agent cannot run anything itself, for example inside a chat app. Then the engine can be offered as a small service next to the data: the agent writes the SQL, a lightweight engine runs it and sends back the answer. Either way, the engine is chosen for each question, not installed once for the whole company.
There is a nuance worth getting right. The language model itself does not do the arithmetic; asking it to add up a million rows "in its head" would be slow and unreliable. What changes is that the model can now operate an engine on its own. The engine is still there. It just lives next to the agent instead of next to the data.
What the data needs now
If the engine travels with the agent, the data needs something different from what it needed before. It no longer needs a server of its own that does the calculating. It needs a shared home that any engine can reach quickly and safely:
- A catalog. A place where agents can find which tables exist, what they contain and where their files are, using a standard that every engine understands.
- Open tables. Data stored in an open format, so today's agent, tomorrow's engine and the analyst's laptop all read the same table.
- Fast access to just what is needed. The engine should fetch only the parts of a table that a question touches, not the whole thing. Iceberg's metadata makes this possible.
- Fine-grained, short-lived access. Agents should get exactly the permissions they need, for a limited time, and nothing more.
- Safe writing. Several agents adding results at once, without corrupting each other's work, and with a history of every change.
- Housekeeping. Tables that are written to all day need regular tidying so they stay fast. Nobody wants an agent doing that by accident.
LakehouseBox: AI infrastructure where your data lives
This is the idea behind LakehouseBox. Your agents already do the analysis; LakehouseBox is where their data lives. It is not another engine. It is the shared home: an Apache Iceberg REST catalog with its own storage, operated in Nuremberg, Germany. Your data stays in one place, in open tables, and every engine and every agent comes to it.
In practice, that means you can use the engine that suits each job, without copying the data:
- AI agents such as Claude, Codex or Cursor connect through MCP, the command line or the API. When an agent can run code, it brings its own engine. When it cannot, for example in a chat, LakehouseBox runs the query for it.
- Engines on your own machine: DuckDB (even inside a web browser), Polars and PyIceberg read and write the tables directly.
- Cluster engines: Spark and Trino, for jobs too big for one machine.
- Warehouses: Snowflake reads and writes the same tables once Snowflake Support has allowed our storage endpoint for your account, so a team that already uses it does not have to move anything.
All of these are verified against the service, and we publish the status of each engine, with the date of the last test, on our engines page.
Whatever the engine, the rules are the same. Each one connects with credentials that grant read or write access per catalog and expire after an hour. LakehouseBox takes care of storage, access and maintenance, and you pay only for the storage you use. Change engine tomorrow, add a second agent, or let a partner query from Snowflake: the data does not move.
In short: bring the agents and engines you already use. Your data stays.
What this means for an organisation
None of these eras erased the previous one. What changes is where the data lives, and that is a decision an organisation makes once, for every team and every tool.
- Keep the warehouse you have. In this model the warehouse becomes one more engine on the same tables. A team on Snowflake reads and writes them once Snowflake Support has allowed our storage endpoint for its account, and its dashboards keep running while agents work on the same data. We looked at when a dedicated warehouse still pays off in Does your agent need a data warehouse?
- Scale the engine, not the data. A question too big for an agent's machine goes to Spark or Trino, which read the same tables. No copy, no second pipeline.
- One set of rules for people and agents. Colleagues join the organisation as members or admins. Each agent gets its own key, limited to the catalogs it needs, and any access can be taken back.
- Open by default. Iceberg tables in a standard REST catalog: change engine or provider and the data comes with you. We say publicly what we have verified, engine by engine, on the engines page.
The direction is clear. Over twenty years the engine has moved from the data's own cluster, to the vendor's cloud, to any tool that speaks an open format, and now to the agent itself. At each step the data became a little more independent. The next question is not which engine to buy, but where your data should live so that every engine, and every agent, can use it.