Paintings Subscribe Tiếng Việt

16 November 2026

Deep Dive/Testing AI features with real data, not mocks

Mocks make AI tests pass while the product stays wrong. How I test an AI data analyst end to end against real databases.

A test that uses mock data tells you the code runs. It does not tell you the answer is right.

For ordinary software the difference is small. For AI features it is everything.

Why mocks lie to AI

While building an AI data analyst (you ask in plain language, it writes SQL, runs it and answers), the scariest bugs were never crashes. They were answers that looked perfectly reasonable and were wrong:

  • SQL that is valid but joins the wrong table, so revenue doubles.
  • A model reading “last month” in a different time zone.
  • Real column names that look nothing like the mock’s column names.
  • Each engine (Snowflake, BigQuery, Databricks) has its own dialect that no mock ever reproduces.

A mock answers what you told it to answer. So the tests stay green while users get wrong numbers.

What I do: test against real data

1. Real databases, small and safe. Trial accounts, public datasets (for example BigQuery’s public data), and always read-only access. Never customer data.

2. Questions with known answers. Every question comes with the correct number, computed by hand-written SQL. The agent must reach exactly that number, not “roughly”.

3. Run end to end like a real user. From the question, through the model, through the database connection, to the answer and chart in the interface. Every run saves screenshots as evidence.

4. Accept that AI is not deterministic. The same question can produce different SQL. I compare results, not SQL. For hard questions I run several times and require most runs to be right.

5. Mocks still have a place. Unit tests for pure logic still use mocks to stay fast. But a feature only counts as done once it works on real data.

The cost

Real tests are slower, cost more in model calls, and sometimes fail for outside reasons (network, account limits). I accept that. A slow test that tells the truth beats a fast test that lies.

What I learned

Most of the serious bugs I found were invisible with mock data.

With AI, the testing question is not “does the code run?” but “is the number right?” And only real data can answer it.