Agents

Always-learning analytics: why agents don't have to start over

Analytics run by people learns slowly and forgets when the person or the project moves on. We gave an agent a real daily job, forecasting Madrid's air, and measured what keeping its knowledge in tables is worth.

8 min read

Madrid's rooftops in 2017 under a brown layer of nitrogen dioxide, with clear blue sky above it.

Every analytics team knows this curve. Someone builds a model, a dashboard or a forecast. Over months they learn where it goes wrong and correct it. Then they move to another project, or leave, and what they learned leaves with them. The next person starts again, close to the bottom.

Two curves over time. People and projects draw a sawtooth: each one learns slowly, and the line falls back when someone moves on. Agents that keep what they learn in tables draw a line that keeps rising.
The idea, not data: knowledge kept by people resets at every handover; knowledge kept in tables builds up.

Agents can draw the other line, under one condition. What they learn has to be kept somewhere the next run, the next agent and the next person can read it. In practice that means tables: every decision, what happened afterwards, and how good the decision was. We call this always-learning analytics. It is the reason LakehouseBox is built the way it is, so we tested it on a real problem, with numbers.

The job: Madrid's air, decided by noon

The photo at the top is Madrid in 2017. The brown layer is nitrogen dioxide (NO2), mostly from traffic. That year the European Commission was pursuing Spain over NO2 levels in Madrid and Barcelona.

Madrid has done a lot since then, and it shows. In 2010, 18 of its stations were over the EU's annual NO2 limit. In 2022, for the first time, none was, and none has been since: 2025 was the fourth year in a row, according to the city's own summary. The city's open data, loaded into our catalog, tells the same story day by day.

Days a year when some Madrid station averaged more than 50 micrograms of NO2: 309 in 2010, 201 in 2019, 101 in 2022 and 54 in 2025. The EU limit from 2030 allows 18.
Measured, from the city's open data. Madrid went from 309 such days in 2010 to 54 in 2025.

The job is not finished, though. From 2030 the EU's rules get much stricter: no station should average more than 50 µg/m³ on more than 18 days a year. Madrid had 54 such days in 2025. And the city's protocol for bad days still stands: when pollution is forecast to be high, it must announce traffic restrictions before noon the day before. Someone has to make that call every day, with what is known at that hour.

Measuring instruments of Madrid's urban monitoring network, in a fenced enclosure between residential buildings.
Instruments of Madrid's urban monitoring network. Photo: Diario de Madrid, CC BY 4.0.

The city publishes what its stations measure, every hour, as open data. So we gave an agent the city's job. Every day at noon it answers one question: will some station go over 50 tomorrow? It gives a probability, and the next day the stations score it.

How it runs

The city's open data flows by scheduled SQL into tables on LakehouseBox. A hosted agent reads them every day at 12:00 and writes its forecast back. The tables also hold how each forecast turned out, and anyone can read them with SQL.
The whole loop is tables: what happened, what the agent said, and how it did.

Everything is in one Apache Iceberg catalog on LakehouseBox:

  • What happened. The city's hourly air quality since 2010 (20.8 million readings), its weather since 2019 and its traffic. Scheduled SQL fetches each source and keeps the tables fresh.
  • What the agent said. A hosted agent runs next to the data at 12:00 Madrid time and appends its forecast to a table.
  • How it did. A scheduled query joins each forecast to what the stations measured the next day. The score is one more table.

Nothing the agent learns lives inside the agent. It is all rows that the agent, another agent or a person can query.

The test: one winter, five kinds of memory

A live agent needs months before its learning shows. So we replayed a past winter, 1 October 2024 to 31 March 2025: 182 days. Each simulated day at noon, the forecasters saw only what was known at that hour. They all used the same model and got the same brief: the last week, this morning's readings, tomorrow's weather and calendar. Only their memory differed.

  1. No memory. Every day starts from zero.
  2. Notes. It keeps up to 12 lessons and rewrites them every day after seeing how it did. This is what "agent memory" usually means today.
  3. Its record, wiped monthly. It queries the table of its own scored forecasts, but the table is emptied on the 1st of every month: the person or the project changes.
  4. Its record, kept. The same, never emptied.
  5. Its record plus the city's tables. The same, and also the days of the last three years most like tomorrow, with what happened on them. The organisation's knowledge, from the first day.

What we found

Without memory, the agent makes the same mistake every day. The model knows the meteorology: calm air and cold clear nights build NO2 up. But it applied that too eagerly. It said 49% on average for something that happened on 35% of the days, and it said so all winter. Nothing told it otherwise.

Notes decay. October 2024 was clean, and the notebook learned that one day at a time. By 18 November, 11 of its 12 lessons capped the chance of an exceedance at 20% or less. That day it said 15%. The model with no memory said 90%, and a station went over. Over the winter, two thirds of all the lessons it wrote said the same thing. Notes are a summary of the last few weeks, written as if they were rules, and they are hard to unlearn. It is the human sawtooth again: knowledge kept as prose.

Forgetting costs a week, every time. In the first week of every month, the forecaster whose record had just been wiped did worse than the one that kept it: in all six months, and on average exactly as badly as no memory. By the third week it had caught up.

Extra error of the forecaster whose record is wiped on the 1st of each month, by week of the month: clearly worse in week 1, level again by week 3.
Measured. After each wipe the forecaster is back to the level of no memory for a week.

What compounds is what the organisation keeps. The more knowledge the forecaster could query, the less it got wrong.

Error removed compared with no memory: notes 8%, record wiped monthly 9%, record kept 13%, record plus the city's tables 18%.
Measured. Each layer of kept knowledge removed more error. The clear step is the city's tables.

The forecaster that read the city's tables did best: 18% less error than no memory, the one difference among these that is clearly larger than the noise of a single winter. It did not need a record of its own to start: in October, its first month, its error was 0.096 against 0.153 with no memory. It was also the only model to match the fixed "almost never" rule on the rarest question we asked, the parking surcharge, which happened twice all winter. Its own record could not teach it something that rare. Three years of the city's history could.

Memory is how an agent calibrates

A forecaster is calibrated when its numbers mean what they say: when it says 70%, it happens about 7 times in 10. A model on its own has no way to know. Ours said "40 to 60%" on 15 days, and it happened on 2.

An agent that reads its record does know. It reads "when you said 60 to 80%, it happened 22 times in 39" and adjusts, the way an experienced analyst learns how far to trust their own numbers. Every forecaster with a memory was better calibrated than the one without.

That is also why the gain from its own record came in weeks and then levelled off. Once the numbers are honest, more record does not make them more so. What kept adding was the organisation's tables. So the compounding is not one forecast getting better forever. It is that nothing starts over.

The Memory tab of the Aire forecaster agent in LakehouseBox: two open decisions, the forecasts for 16 and 17 November 2026, each with the date it must be checked.
The live forecaster's memory in LakehouseBox: the forecasts it has to come back and close. What it learns from them goes into the tables.

Why LakehouseBox is built this way

All five forecasters used the same model. What changed the result was what each could keep, and where it lived. That is the part a platform has to provide, and it is what LakehouseBox is: open Apache Iceberg tables where the data and the agents' work sit side by side, agents that run next to that data with a memory made for work that repeats, and pages that read the same tables in the browser.

Models and compute will keep changing. What an organisation has learned should not change with them.

Your agents bring the compute. We keep what they learn.

See it yourself

The live page shows the forecaster at work: today's air, Madrid under the 2030 rules, every forecast it has made, and how each one turned out. The repository has the queries, the agent's prompt and the replay.

LakehouseBox was born in Madrid, and we have a saying here: de Madrid al cielo. From Madrid to heaven. For fifteen years this city has measured its air, made hard decisions and kept at it, and you can see it when you look up. This post is also a thank-you.

The rooftops, domes and spires of old Madrid under a wide, clear blue sky.
De Madrid al cielo. Photo: Dmitry Dzhus, CC BY 2.0.

Explore how it works

Related articleLooking for a Snowflake Open Catalog alternative?