Why Most AI-FP&A Demos Look Impressive, and Fall Apart in Production

I've sat through a lot of AI-FP&A demos in the last twelve months. The good ones are genuinely impressive. An agent reads a variance, generates a fluent narrative, identifies the driver, and recommends an action, all in about thirty seconds. The CFO in the room nods. Someone says "this is going to change everything." A follow-up meeting gets scheduled.

And then I ask one question that tends to make the room a little quieter: Where did that number come from?

Not the variance. The explanation. The agent said the revenue shortfall was driven by a price-induced volume decline. Where did that conclusion come from? Can the auditor follow the trail? Would the answer be the same if the agent ran it again tomorrow with the same data?

That's usually when the demo starts to feel less impressive.

I'm writing this series because I think the AI-FP&A market has a credibility problem, and finance leaders need a sharper way to evaluate what they're seeing. The technology is real. The opportunity is real. But the gap between what looks good in a demo and what holds up in production is wider than most vendors are admitting, and the cost of getting this wrong is too high to navigate by feel.

This first article in the series is about learning to spot that gap. The next two build directly on it, moving from evaluation to action, covering what finance teams should actually do next.

The demo problem

Here's the uncomfortable truth about generative AI: producing a confident, articulate, well-structured narrative is the easiest thing modern language models do. It is, in fact, the thing they do whether or not the underlying analysis is correct.

That matters in finance more than almost anywhere else. A fluent paragraph that explains a variance feels like analysis. It reads like the kind of thing a senior FP&A analyst would write. But fluency is not the same as truth, and in production, over hundreds of monthly closes, across dozens of departments, in front of audit committees, the gap shows up in three predictable ways.

First, the agent hallucinates explanations that aren't supported by the data. It says revenue is down because of a pricing change when the actual cause was a customer concentration shift. It sounds right. It isn't.

Second, the agent's calculations don't match the numbers in the close. The CFO sees a variance figure in the agent's narrative that's slightly different from the same figure in the management report, because the agent re-derived it from raw data using slightly different logic than the report writer used. Two sources of truth is no source of truth.

Third, the agent gives a different answer when asked the same question twice. Maybe the model temperature is non-zero, maybe it pulled different context, maybe an upstream calculation drifted. Whatever the reason, "ask again and check" is not a process the audit committee will accept.

These are not edge cases. They are what happens when AI agents are deployed against finance work without the right architecture beneath them. And they are exactly the failure modes that don't show up in a thirty-minute demo.

A short story about a breakfast omelette

Let me make this concrete. Imagine an agent is asked to explain a variance: breakfast revenue is down $80,000 against budget for the month, a 16% miss. The agent has access to the general ledger and the point-of-sale system.

Here's what a poorly architected agent does. It looks at the GL, sees the revenue shortfall, and starts pattern-matching. It pulls in some POS data, scans for unusual values, finds that omelette unit sales are down. It writes a confident-sounding paragraph: "Breakfast revenue declined due to lower omelette sales, likely reflecting changing consumer preferences."

That sentence has the shape of an analysis. It has none of the substance. The agent didn't decompose price from volume. It didn't check whether the price changed. It didn't quantify the contribution of any specific driver. It pattern-matched an explanation that sounds plausible and stopped.

Now imagine a properly architected agent. The same agent is reading a structured variance report (built by the finance team, certified for analysis use) that contains a specific row for breakfast revenue. The variance amount is pre-calculated. There's an explicit link key pointing to the slice of POS data that explains it. The POS report itself decomposes the revenue change into volume impact, price impact, and mix impact, each with the underlying calculations visible.

This is the architecture worth insisting on. It is also the architecture behind Solver Copilot, which reasons on the certified reports your finance team already trusts rather than re-deriving numbers from raw data. 

The agent reads: breakfast revenue is down $80,000. The link key points to the breakfast section of the POS report. In that report, omelette unit sales are down 28%. Average price is up from $6.50 to $8.50, a 31% increase. Volume impact is negative $45,500. Price impact is positive $36,000. Mix impact is zero.

The agent writes: "Breakfast revenue underperformed budget by $80,000. The primary driver was a 28% decline in omelette unit sales following a price increase from $6.50 to $8.50. The price increase partially offset the volume loss, but the net effect was a revenue decline. This suggests demand sensitivity to the price change."

Both narratives sound similar at first read. One is reasoning. The other is translation: the agent translating a structured, pre-decomposed analysis into prose. The second one is also auditable, reproducible, and traceable to specific cells in specific reports.

The architectural difference between these two agents is not the model. It's the financial logic the agent reasons on top of.

The principle: AI should reason on financial truth, not re-invent it

The teams that have spent the last two decades building good FP&A organizations have done something quietly important. They have encoded the company's financial logic (chart of accounts, dimension hierarchies, KPI definitions, variance methods, materiality thresholds, time intelligence) into the reports their finance teams actually use. That logic is the company's financial truth.

When an AI agent is asked to analyze a variance by reading raw fact tables and re-deriving the math from scratch, it is implicitly being asked to re-invent that financial truth in real time, every run, with no oversight. That is the source of most of the failure modes I described earlier. Hallucinations, mismatches with the close, non-reproducible answers: they are predictable consequences of asking AI to do work it shouldn't be doing.

The alternative, which I think will turn out to be the only architecture that produces trustworthy AI in finance, is this: agents should reason on the same reports your finance team already trusts. The reports become the calculation engine. The variance is pre-calculated. The drivers are pre-decomposed. The link keys between reports are explicit. The agent's job is to read structured outputs, connect them, and translate them into language a human can act on. The agent does not invent numbers. It interprets them.

This is a principle, not a product feature. Any AI-FP&A capability, built in-house, bought from a vendor, or layered on top of a spreadsheet, can be evaluated against it. And the ones that fail the evaluation are the ones most likely to embarrass you in front of the audit committee.

Five questions every finance leader should ask

Here is the practical version of all of this: a short list of questions you can take into any vendor conversation, any internal AI experiment, any board discussion about AI strategy.

1. Does it reason on the same numbers your finance team trusts? If the AI is reading raw GL data and re-deriving variances, KPIs, and consolidations from scratch, you have two sources of truth, and you will eventually pay for that. If the AI is reading the same certified reports, KPIs, and variance calculations your finance team already uses, you have one source of truth. Insist on the second.

2. Are calculations deterministic where they need to be? A variance is a subtraction. A KPI is a defined formula. Reconciliation matching follows rules. None of these calculations should be performed by a probabilistic model. Ask, specifically, which calculations in the workflow are deterministic and which are AI-generated. The split should be obvious and defensible.

3. Is every output traceable to source data and the rules applied? When the agent says revenue is down 16%, you should be able to click through to the source transactions, the calculation that produced the variance, and the rule that determined materiality. "Show your work" should be one click away. If it isn't, you cannot audit the system, and you cannot trust it.

4. Are humans in the loop at the right decision points? AI should accelerate volume work: gathering data, detecting variances, generating draft narratives, flagging anomalies. Humans should own judgment: approving the flash P&L, validating commentary on material items, signing off on consolidated reports. Ask where the approval gates are, who owns them, and what evidence is captured at each one. If the answer is hand-wavy, the governance isn't real.

5. Can the system improve from human corrections without becoming unpredictable? When an FP&A manager edits an agent's narrative or overrides a recommendation, that correction is gold. It's the highest-quality training signal the system will ever get. Ask how corrections are captured, how they influence future behavior, and how the system avoids drifting in unintended directions over time. Learning without governance is just drift.

These five questions won't make you an AI expert. They will make you a sharper buyer, a more credible internal sponsor, and, most importantly, a finance leader who is unlikely to be the one explaining to the audit committee why a number was wrong.

The principle that ties this together

There is one sentence I want every finance leader to leave this article with, because it cuts through almost every confused conversation about AI in finance:

AI accelerates volume work. Humans own judgment and final accountability.

When AI is deployed against the right work (high-volume, rule-based, evidence-rich) and paired with the right architecture (financial truth, deterministic calculation, traceable outputs, human approval at the right gates), it becomes genuinely valuable. It is not magic. It is not a replacement for finance professionals. It is leverage on the work finance professionals already do, freeing them to spend more time on the things only humans can do: explaining what happened, understanding why, recommending what to do next, and standing behind the answer.

When AI is deployed against the wrong work, or with the wrong architecture beneath it, it produces fluent, confident, articulate, plausible, unverifiable, non-reproducible nonsense. And finance teams pay the price.

The question for every finance leader in 2026 is which of these you're going to end up with. The answer is decided less by which platform you choose than by what you ask it to do, what you build beneath it, and what you insist on being able to verify.

What's next

If this resonated, the next article in this series argues something that may be more provocative than the architectural argument I just made. The real opportunity in agentic AI for FP&A isn't faster reporting. It's something most finance teams will miss entirely, and the ones that miss it will spend the next eighteen months falling further behind their peers in ways that are very hard to catch up to. If this series is useful to you, follow along for the next two articles.