Why we check AI's numbers without AI

Key Takeaways

• Compass 2.0 compares each figure in an AI draft with the books; a mismatch gets one rewrite, then the draft waits for a person with the figure flagged.
• On DualEntry's public accounting benchmark, the top model answered 84.2% of 101 accounting tasks correctly as of 9 October 2026, so about 1 task in 6 still came back wrong.
• In Penrose's AccountingBench (July 2025), models closing a real company's books began inventing or borrowing transactions to pass the checks once discrepancies piled up.
• IRS OPR Alert 2026-19 (June 2026) says due diligence on AI output includes checking its facts, citations and calculations, and recommends documenting how AI was used and checked.
• Checks test arithmetic and sources only: a figure Compass can't trace says Not traced, and whether the advice is right stays with the advisor.

Compass 2.0 compares every figure in an AI draft with the books before the draft reaches you. The comparison runs on plain arithmetic, with no AI involved, and it is the most important choice we made in this release. Here is why.

What goes wrong when AI drafts client work?

The AI usually finds the right data, then picks the wrong number from it, and the sentence around that number still reads perfectly.

Here is an example from our demo firm. A dental practice's owner letter said revenue grew 0.7%. The books said 10.3%. The AI had picked up the growth in operating costs and put it in the revenue sentence. Every word around the figure was fine. You would only catch it by going back to the books.

AI reasons well when the facts are laid out right. It slips when it has to choose: the basis, the source, the formula. And choosing is most of advisory work: cash or accrual, which month is closed, whether Stripe or the ledger is the source of truth for revenue.

So the hours move. You get the draft in a minute and spend the next hour checking it, for every client, because your name is on the letter.

Why not ask the AI to check its own work?

Because the reviewer shares the writer's blind spots.

Popular finance prompt guides suggest asking the model to check its own output before it presents it. That catches dropped steps and typos. It does little when the model chose the wrong source with confidence, because the second pass reads the same context and makes the same choice.

The research points the same way. DualEntry, an accounting software company, runs a public benchmark of 101 accounting tasks graded right or wrong. As of 9 October 2026, the best model answers 84.2% of them correctly. That is real progress, and it still leaves about one task in six wrong.

In July 2025, Penrose asked frontier models to close a year of books for a real SaaS business. The strongest started within 1% of a CPA's results in the first month. Then errors compounded, and once discrepancies piled up, models began inventing transactions or pulling unrelated ones to pass the benchmark's checks. Those were 2025 models, and today's do better. The lesson holds anyway: a check the model can satisfy by changing its story protects nothing.

So we put the check where the model can't reach it.

What does Compass 2.0 check, and how?

Three pieces work together: one definition for every number, each client's rules held as settings, and every figure in a draft compared with the books.

One definition for every number. Runway, gross margin, days to get paid and 31 more standard numbers are each defined once, in plain words, with their inputs and sources. Compass computes them with arithmetic, and the same runway shows on Home, in a report, in a workflow and in Claude.

Each client's rules as settings. Basis, year end and what's worth mentioning are House rules, prefilled from the data and confirmed by you. Months after the client's closed-through month carry a Preliminary mark everywhere, including in Claude. Anything the AI had to assume is labeled Assumed.

Every figure compared with the books. Before a draft reaches you, each figure in it is compared with the number Compass worked out. A mismatch gets one rewrite. If it still doesn't match, a workflow set to send on its own falls back to approval: the draft waits for you with the figure flagged, and one click sends it with the right number.

The same goes for the data underneath. Every time new data lands, Compass compares Stripe, Square payout files and bank files that state a balance with the books, and a month that doesn't agree holds that month's automatic sends until someone fixes it or accepts the difference with a reason. Ten checks come built in, and you can add your own by describing one in a sentence. The Checks docs list every one.

What can't checking do?

Checks test arithmetic and sources, and they say so when they can't.

A figure Compass can't trace says Not traced. A check that can't run says why. Neither one claims more than it knows. A check can confirm that the letter's 10.3% matches the books. Whether 10.3% growth should change the client's hiring plan is still your call, and that's the part of the job we want you spending the hour on.

Why does this matter for the person who signs?

The person who signs carries the responsibility, and regulators now say so about AI.

In June 2026, the IRS Office of Professional Responsibility issued its first guidance on AI under Circular 230. It says due diligence includes verifying the facts, citations and calculations AI produces, and it recommends documenting how AI was used and checked. That guidance covers federal tax practice, and advisory work sits outside it, yet the expectation is easy to read. The Journal of Accountancy's July 2026 professional liability column tells firms the same: record the prompts, how the output was verified and who reviewed it.

In Compass, that record builds itself. Every read and every change, by a person, a skill or Claude, lands in Activity with a name on it, and the Checks page keeps the history of every run.

What's next

We watched advisors paste spreadsheets into chat windows, then trace every figure that came back by hand. 2.0 takes the legwork and the arithmetic, and the judgment stays with you.

Compass 2.0 is in every account today. If you're new, start a 14-day trial with every feature and no card, or watch the 2:19 film first. And tell me what you'd want checked next.

Nikola

Frequently asked questions

Can AI check its own work on financial reports?

AI can catch slips such as dropped steps or typos, and finance prompt guides often suggest asking for that pass. It is weaker on choice errors, like the wrong source or basis, because the second pass reads the same context and tends to make the same choice. A check outside the model, written in arithmetic, has no such context to share.

How accurate is AI at accounting tasks in 2026?

It depends on the model and the task. On DualEntry's public benchmark of 101 accounting tasks, the best model answered 84.2% correctly as of 9 October 2026, so about one task in six came back wrong. A software vendor runs the benchmark and the scores change as models are added, so check the live leaderboard for the current figure.

What does Compass 2.0 check?

Compass 2.0 compares every figure in an AI draft with the number it works out from the books, and compares Stripe, Square payout files and bank files that state a balance with the books every time new data lands. Ten checks come built in, covering the period, sign and scale, whether the math adds up and more. You can add your own agreements and rules by describing them in a sentence, and the Checks docs list them all.

What happens when Compass finds a mismatch?

Compass rewrites the draft once. If the figure still doesn't match, a workflow set to send on its own falls back to approval: the draft waits for you with the figure flagged, and one click sends it with the number from the books. The catch is kept on the Checks page with a link to the exact output.

Does the IRS require CPAs to review AI output?

For federal tax practice, IRS OPR Alert 2026-19 (June 2026) says Circular 230's due diligence duties apply to AI, including verifying the facts, citations and calculations it produces, and it recommends documenting how AI work was checked. The alert covers practice before the IRS, so advisory work falls outside it, and this post isn't legal advice. The Journal of Accountancy's guidance for firms extends thorough human review to advisory recommendations as well.

Do I need Claude or ChatGPT to use Checks?

Compass has an assistant built in, running on Claude, and Checks run on its answers, workflows and skills. If you already work in Claude or ChatGPT, connect it once: its answers carry the company's data checks for the period, and its reads are recorded in Activity.

What does Compass cost?

Every feature is free for 14 days, with no card. After that, plans start at $29 per company per month, and advisory firms use the Portfolio plan at $99 a month for the hub plus $19 per client company (as of October 2026). See pricing for the full list.

Sources

1. DualEntry. The 2026 Accounting AI Benchmark. 2026.
2. Penrose. Can LLMs Do Accounting? (AccountingBench). 2025.
3. Yunyu Lin. Thread introducing AccountingBench. 18 July 2025.
4. Linas Beliūnas. Turn Claude Sonnet 4.6 Into Financial Analyst That Never Sleeps. Linas's Newsletter, 20 February 2026.
5. IRS Office of Professional Responsibility. Introductory Guidelines for Responsible AI Use in Federal Tax Practice (OPR Alert 2026-19). 24 June 2026.
6. David A. Hughes and Tatum Andres. Using AI in Tax Practice: What Practitioners Need to Know Before Filing Season. Kilpatrick Townsend, 29 September 2026.
7. Nicole L. Graham. Drafting an AI policy that actually works. Journal of Accountancy, 1 July 2026.