The LLM confidence trap: 10+10=20, doesn't it?

There are a lot of people, in a lot of businesses, confidently using LLMs for reporting and calculations. This confidence is misplaced.

If you are passing raw data into an LLM, it is probable your reports will be incorrect some or all of the time. If the LLM itself is doing the sums (no calculator, no code), it is running on luck.

Take the simple case of 10 + 10 = 20. To a calculator or a simple maths program, this is:

[number][operator][number][equals][number]

Very simple, very computable.

To an LLM it is quite different. A naïve interpretation is more like this:

Token 1 [10+]
Token 2 [10=]

What are the likely next tokens? In this case [20 ] (twenty, zero, space) is very likely to be correct in a probabilistic model.

Will it get this right most of the time? Absolutely.

So our confidence grows; however, there is something important to understand about the LLM version:

There were no numbers.

There were no operators.

Just a probability engine.

So how did it get 10 + 10 = 20 correct? Let's run a naïve and simple thought experiement.

There are 100 people in a room. You ask 99 of them what 10 + 10 is, and each answers 20, as expected. You can predict with high confidence that the 100th person will also answer 20.

Now let's do it again with a slight variation. Same question, but this time 99 people answer 21 (obviously wrong, but useful for the illustration). You can predict with confidence that the 100th person will also answer 21.

But as we know, 10 + 10 does not equal 21. Is the prediction wrong?

Not at all. In the second scenario, 21 was the most probable answer, so our human probability engine worked perfectly. It gave you the most likely answer. It just wasn't the correct one.

Modern LLM apps do have some help

If you are using chat apps such as ChatGPT, Claude or Gemini, they have tools built in to help with calculations, which greatly increases the chance of getting them correct. If anything, the bigger risk becomes reasoning issues when you pass a lot of data in. That said, they don't always call those tools automatically. If they do the sums inline, the chance of incorrect values goes back up.

If you are using LLM APIs for the calculations, that is potentially a worse situation, as typically that is only LLM inference. Some APIs offer tooling that might help, but the norm is inference, the model doing the sums in its 'head'.

So the danger isn't that LLMs can't produce results for calculations; they can, and it's probable they'll get the numbers correct, or very close, a lot of the time... Or they won't... there is no way to know for sure which will happen.

Here is the part that matters. A single number is often right, even a small set of numbers can be. But a real report isn't one sum or a small set of numbers, it's hundreds of figures. Even a tiny error rate, spread across a whole report, makes the odds that every number is correct fall away quickly. And because every figure looks equally confident, you can't tell which one slipped.

If you are thinking that your LLM-based reporting is the exception to this reality (you are not alone), this just means your confidence level needs to be significantly adjusted.

Is calculation the only issue?

No. There's another side to passing raw values into an LLM that can sting even more easily.

If you're passing a wide dataset of raw values in for financial reporting, you're relying on the model reasoning correctly every single time it runs. And reasoning fails more quietly than maths. An LLM can answer a slightly different, easier question than the one you actually asked, and still hand you a tidy, confident report where every number adds up. There's no broken sum to catch. The arithmetic is perfect; the judgement isn't.

This is made worse by the fact that an LLM is not a deterministic technology. You can sometimes make it behave more deterministically, through settings like temperature, but in my experience I've yet to see a model that is both genuinely usable and reliably deterministic. So the report that looked right today is not guaranteed to be the report you get tomorrow, from the same data.

None of this means LLMs are useless, far from it. It means you have to give them the right job. Let a calculator do the calculating, and let the model do what it's actually good at: the words around the numbers, not the numbers themselves. Feed it figures that have already been worked out and it becomes genuinely useful. Trust it to work them out for you, and the first time it feels like a shortcut is exactly the moment the trap closes.

Because the confidence you feel when it gets 10 + 10 right may well become a very expensive feeling indeed.

If you have a system that is ingesting raw figures into an LLM for reporting, switch it off.