Study plan: does asking “when is that figure from?” fix an undated AI answer?

Study plan, registered on 14 October 2026. This study has not been run yet. I’m publishing the method before any results exist so it can’t be adjusted to fit them. Results will be added to this page, with the date they were collected.

On 10 October 2026, 7 of 40 AI answers to “current” UK figure questions didn’t say when the figure was from (the study). My hack says to follow up with a fixed question when that happens, but it also says I hadn’t tested whether the follow-up works. This study tests it.

The one variable

Every chat asks the same first question. The only thing that changes is the second message:

• Date follow-up (arm D): When is that figure from? Give the month or period and the official source. This is word for word what the hack tells readers to paste.
• Neutral follow-up (arm N): Thanks. Is there anything else I should know?

Arm N is the control. Some engines may add a date to any second answer, so the effect of the date question is the difference between the two arms, not arm D on its own.

First questions (frozen)

These are the ten questions from the 10 October run, word for word and in the same order:

1. What is the UK inflation rate right now?
2. What is the average house price in the UK right now?
3. What is the average salary in the UK now?
4. What is the UK unemployment rate now?
5. What is the Bank of England base rate today?
6. What share of new cars sold in the UK are electric now?
7. What is the UK population now?
8. How much is the full new State Pension per week now?
9. What is the Ofgem energy price cap right now?
10. What is the UK national debt now?

Engines and sessions

The same four engines and settings as 10 October. The model or mode shown on screen is recorded for every chat.

• ChatGPT: default model, signed in, temporary chat set to unpersonalised.
• Google AI Mode: signed in.
• Claude: web search on, incognito chat.
• Gemini: signed in, temporary chat.

Each chat is new. The first question and then the follow-up are pasted exactly, with nothing else added. Everything is run from the UK on one stated date. Perplexity isn’t run, for the same reason as on 10 October: it asked for a sign-up.

Runs

10 questions × 4 engines × 2 arms × 2 runs = 160 chats, each with two answers. The order is fixed before the run: engine by engine, question by question, with arms alternating so neither arm always runs first. All 160 are run on one day, so the official figures stay the same throughout. The run won’t be on 5 November 2026, which is a Bank Rate decision day.

What’s recorded for each chat

• First answer: headline figure, and whether it’s dated (the same rule as the 10 October study: the answer itself states the period or date the headline figure refers to; a period that appears only in a source card or snippet doesn’t count).
• Second answer: whether it states a period for the headline figure, the period given, whether that period matches the official period, whether the headline figure changed from the first answer, and the source named, if any.
• Answer key: the official figure and period, read the same day from the same sources as on 10 October (ONS, Bank of England, SMMT, GOV.UK, Ofgem).

How results are counted

1. Primary: the share of second answers that state a period for the headline figure, in arm D compared with arm N (out of 80 each).
2. Repair rate: among chats whose first answer was undated, the share whose second answer gives a period, by arm. If either arm has fewer than 5 undated first answers, the counts are published but no rate is quoted.
3. Accuracy of the added date: of the periods given in arm D second answers, how many match the official period.
4. Side effect: how often the headline figure changes between the first and second answer, by arm.

Exclusions

A chat is excluded and rerun once, the same day, if the session settings were wrong (for example personalisation on), the engine errored or asked to sign in, or the first answer gave no figure. Every exclusion stays in the CSV with included = no and a reason, as in the 10 October file.

What I will not claim

• That a follow-up makes the figure correct. This tests whether a period appears and whether it matches the official one.
• Anything about other wordings of the follow-up. Only these two are tested.
• Anything beyond these four engines, these ten UK figures and one day.
• Why an engine dates or doesn’t date an answer.

Schedule

• Registered: 14 October 2026.
• Run: one day before 30 November 2026, with the date stated on the results page.
• Results: published the same week as the run.

Calls resting on this study

P19 (Still open) and P20 (Still open) on the predictions page.

Data

date-follow-up-YYYY-MM-DD.csv, one row per chat (160 rows, plus excluded rows), with the columns run_order, run_date, run_time, engine, model_mode, session, query_id, first_question, run, arm, follow_up, first_figure, first_dated, first_date_text, second_states_period, second_period_text, second_period_matches_official, second_figure, figure_changed, second_source_named, official_figure, official_period, official_source_url, included and note.