Auditing what AI adds to science

Formulas help us generalise real world observations by compressing them into something a human can interpret.

But these formulas are not universal. They can fail on observations we never accounted for, and that failure shows up as error in the predictions. Better instruments, new discoveries and other advances keep exposing those gaps, so the equations have to be iterated on and updated.

Artificial Intelligence (AI), and Machine Learning (ML) in particular, has become very good at finding patterns in data. When the error a formula leaves behind has a shape instead of looking like random noise, that shape is a sign of structure the formula is missing, and ML will find it. The catch is that ML can also overfit our data, meaning it memorises the data rather than learning a solution that generalises. The research I worked on takes these proposed corrections and puts them through a pipeline that tests whether a corrected equation still holds for different observations within the same domain, or regime.

First: the residual

Before the core research, one preliminary. Suppose you have a formula that predicts something: how fast an animal burns energy, how a gas behaves when you squeeze it, how long a forecast says it’ll take you to get home. Now compare what it predicted with what actually happened. The gap between the two is the residual, written r = y − f₀(x), where f₀(x) is the formula you already have. It is simply what that formula gets wrong.

If the residual were pure random noise, there’d be nothing to say. But when it has a shape, that shape might be a real piece of physics or biology the formula is missing.

Or it might be nothing of the sort. That’s the problem.

Glossary

Definition	Name
the formula you already believe	reference modelf₀(x)
what it still gets wrong, the leftovers	residualr = y − f₀(x)
a proposed extra term to fix it	candidate correctionh(x)
a meaningful group you could be asked to work in: a fluid, a species order, a decade, a city	regime
does it still work somewhere new?	transport
the axis you chose to test it across	transport boundary
the surface test: did it beat what we knew before?	pooled (prospective) R²1 − E/(W+B)
the real test: did it explain anything inside?	within-regime R²1 − E/W
how much things vary inside one group	within-regime spreadW
how far one group sits from the others	between-regime offsetB
testing without peeking	leakage-safe leave-one-regime-out
keeping only terms that keep showing up	stability selection
reshuffling which groups you happened to get	cluster bootstrap
how many of the groups it has to work in	positive-regime fraction
deciding to add nothing	noise-floor stop

Equation Discovery

Historically, proposing a correction to an established scientific law was hard. Someone had to have an idea, justify it, and do the algebra. The difficulty of coming up with candidates did a lot of quiet quality control for us.

Modern tools such as symbolic regression, sparse modelling, and machine learning can search enormous spaces of possible equations and hand you a plausible-looking corrections surprisingly quickly. Give one the residual of a scientific law and, with enough flexibility, it will usually find something that describes it.

A predictable pattern in a residual might be:

• a real missing piece of science;
• a quirk of the particular species, materials or instruments in your dataset;
• a historical shift that happens to be baked into your data;
• the fact that one group in your data simply sits higher than another;
• or a flexible curve that works beautifully right where you found it and nowhere else.

A short formula, a stable coefficient and a lower error do not tell these apart. All five look the same on the scoreboard. And that is a real problem, because the first one is a discovery and the other four are ways of fooling yourself.

For the curious: what this problem is called

There’s a well-established research area called equation discovery, which asks: can an algorithm propose a useful mathematical description?

The question here is the one that comes next, and it’s a different problem: when has a proposed correction earned enough evidence to actually change how we describe the world? Our work calls that correction acceptance, and treats it as a separate task that deserves its own machinery: an acceptance layer sitting between the thing that generates candidates and the scientist who interprets them.

Try the pipeline demo

This interactive demo runs on real measurements: the daily average temperature in six cities, every day of 2024.

We start from a deliberately terrible reference model f₀: predict the same temperature everywhere on Earth, every day of the year. Just one number. It is obviously wrong, and that is the point. It leaves a large residual, and the interesting question is what you are allowed to bolt on to fix it.

Two candidate corrections are on the table, two different extra terms h(x) to bolt on:

Each of the six cities is a regime, a scientifically meaningful group your claim is supposed to cover. To test a candidate fairly we hold out an entire regime, build everything using only the other five, including recalculating the reference model f₀, then freeze it and turn it loose on the regime it has never seen. Nothing about the held-out city touches any fitted number, and that discipline is called leakage-safe leave-one-regime-out.

The frozen correction is then scored twice, because “did it do better?” has two very different answers. The surface test (pooled R²) compares it with what we knew before we had ever seen the held-out city, which is the average of the cities we trained on. The real test (within-regime R²) compares it with the held-out city’s own annual average, so knowing roughly how warm the city is earns nothing. The first asks whether the correction found the right place. The second asks whether it explained anything once it got there.

The real test asks whether the red line beats the blue dashed line. Sitting on top of the blue line scores exactly zero. That is what “you have told me nothing I could not have got from this city’s own average” looks like as a picture.

The whole thing as one equation

Everything above is one line of bookkeeping. What happened on a given day equals the reference model, plus whatever the correction adds, plus whatever is still unexplained. Drag the slider to walk through the year.

That formula was fitted without the held-out city. Switch between the two corrections and watch the calendar terms appear and disappear. They are the only terms in the whole expression that change from one day to the next.

For the curious: the arithmetic, in full

Both scores are built from the same three quantities. The only difference is what goes on the bottom of the fraction, and that turns out to be the whole story.

E is how badly the frozen correction missed. W is the within-regime spread, how much the residual varies inside the held-out city. B is how far that city sits from the average of the cities we trained on, its offset.

The surface test divides by W + B. The real test divides by W alone. Since B can never be negative, the surface test always has the bigger denominator, so it always gives the friendlier answer. A correction that only knows a regime’s offset gets to keep all of B as credit, and B can be enormous.

You can watch that happen by changing which city you hold out. Some cities sit far from the average of the others and hand the shortcut a mountain of free credit; one or two sit almost exactly in the middle and hand it almost none. The real test barely moves either way, because the shortcut was never explaining anything inside a city to begin with.

Temperature data: Open-Meteo historical archive (ERA5 reanalysis, ECMWF / Copernicus), CC BY 4.0. Frozen into the page so the booth demo never depends on a live connection.

The obvious test can be fooled

If you held out Reykjavík with Correction #1, you saw it sail through the surface test. And the surface test is not a silly test, it is a perfectly sensible one. It asks whether the correction improved on what we knew before we’d ever looked at that regime. Correction #1 genuinely does improve on that, and by a lot.

But look at how. Correction #1 learned that latitude tells you roughly how cold a place is. It looked up Reykjavík’s latitude, slid its prediction down to about the right level, then sat there, flat, for all 366 days while the actual temperature swung past it from winter to summer and back. It has correctly located the city. It has explained nothing that happens in it.

And here’s the uncomfortable bit. If you only ever ran the surface test, which is the standard, respectable thing to do, the two candidates look like the same kind of success. You would write up Correction #1 as a discovery. People do.

Run all six regimes and it gets worse. Averaged across every held-out city, Correction #1’s within-regime score is negative, and it beats the regime’s own mean in none of them. A correction that looks respectable on the standard test is, on the honest one, a clean sweep of failures.

The second gauge is what separates them. It throws away the free credit for knowing the city’s average, and asks the harder question: having arrived in this new place, did you explain anything about it?

That distinction, between explaining which regime you’re in and explaining what happens inside it, is the centre of the whole project. The failure mode has a name: pooled-positive, within-negative, or offset capture.

For the curious: is this a new idea?

The split between within-group and between-group variation is not new at all. It’s classical statistics, and it turns up under names like fixed-effects models and multilevel modelling. Statisticians have understood it for decades, and our work says so plainly rather than dressing it up as an invention.

What’s new is the use. Nobody had turned that old distinction into a forward-looking rule for deciding whether an AI-proposed correction deserves to be treated as science. That is the step: taking a well-understood diagnostic and building an acceptance procedure out of it.

There is also an honest limitation to state. The real test is retrospective: you can’t compute a new group’s own average until you’ve actually observed it. So it is not a forecasting tool and must never be sold as one. It is an audit you run afterwards, to find out what kind of thing you were looking at.

The next problem: “does it generalise?” is an unfinished question

So Correction #2 passed. It works in a regime it has never seen. Tempting to write down: this correction generalises.

But generalises to what? Every city we used sits in the northern temperate band, from Reykjavík to Tokyo. “A city it has never seen” quietly meant “another city like the ones it learned from.” There are other ways for a regime to be new. The axis you hold out is the transport boundary, and it is a choice.

Same correction, built the same way from the same northern cities. All that changes below is where we ask it to work: another one of those six cities, or one of three cities from outside that band altogether.

Which means you have to say, in advance, what your claim actually is. If you claim a correction is a general law of nature, you must test it across the kinds of situations that claim covers, and you must pick that boundary before you look, not afterwards once you’ve found the one that gives you the answer you wanted.

For the curious: why this matters more than it sounds

In the real project this showed up in an uncomfortable way: the same frozen correction, on the same data, produced opposite verdicts depending on which boundary it was tested across. Not a contradiction: the two tests were simply asking different scientific questions, and the correction was a good answer to one of them and a bad answer to the other.

Different boundaries test different claims. Holding out a time period asks whether something survives an evolving scientific record. Holding out a family of materials or a branch of the tree of life asks whether there is one shared law underneath. A single number called “generalisation performance” quietly hides that choice.

Four ways an inspection can end

Here’s the part I like most, and the part that took me longest to appreciate. The goal was never to build something that stamps ACCEPT more often. It was to build something that can tell these apart. The two interactives above stamp a simple accept or reject on one run; these are the verdicts the full audit returns, in the words the paper uses.

Most scientific tools are judged on how often they find something. This one is judged on whether it finds the right kind of something, and on being willing to come back empty-handed when that’s the honest answer.

What I did during my USRI

I was formulating the test at the centre of the paper, building the software that runs it, and writing the first draft of the manuscript and ended up submitting it to a journal and are waiting for their decision.

Things I can do now that I couldn’t in April

The thing that actually changed how I think

I came in believing research was mostly about building something clever. Most of this summer was spent on the opposite: building reasons to distrust the clever thing I’d built. Writing down what I expected before running it. Freezing thresholds so I couldn’t move the goalposts. Deliberately trying to break my own method and reporting it when I succeeded.

It was a slow progress and faced many challenges are obstacles but it was meaningful work and I enjoyed experiencing it.

Process notes

On collaborators

This was collaborative work. My supervisor conceived the study, developed the original diagnostic that motivated all of this, and derived the mathematics; a PhD student produced the final figures and supported the maths. I’m not naming either of them here, because the paper is under anonymous review and naming my co-authors alongside a description of the work would defeat the point of that. They know who they are, and I owe them a lot.

On AI tools

I used them strategically. Since I was an undergrad doing gradute level research, I needed a way to quickly digest the related works and understand the nuance behind what I was researching on and I used a handful of AI tools to help me study, understand and find references for me to get up to speed. In the manuscript itself, we used generative AI tools to help with language revision and formatting, however every statistical analysis, every number, every interpretation and every citation was checked by us against the locked analysis outputs and the original sources. No AI system is an author, and the final decision was verified by us.

In my own work, I mostly used it as an expert I could interrupt at 2 a.m.: “explain wild-cluster bootstrapping to me again”, “what breaks if my groups are this small”. The reading, the comprehension, the design decisions and the responsibility for what went in stayed mine. It sped up my learning; it didn’t do the learning.

On this page

Programmed using HTML, CSS and JavaScript. Both interactives run the real audit arithmetic on a frozen extract of public weather data. Every score you see is computed on the spot from those measurements; nothing is hard-coded, and nothing comes from the manuscript. Using weather here is a deliberate choice, and it’s the same trick the paper uses in one of its own cases: run the method on a problem where the answer is already known, so watching it work tells you something about the method rather than about the weather. Nobody needs an audit to discover that Reykjavík is cold. That’s exactly why it’s a good test.