On 23 October 2026 we put the same questions to our Ask and to a leading general AI with web search turned on, about real parcels and real review comments in and around Charleston, South Carolina. Every answer was graded against the public record we hold, and each grade was checked by hand.
| The question | Navius | A general AI with web search |
|---|---|---|
| The rules and maps for a parcel | 16 of 18 checkable answers right, each cited to the code section, map layer or public record it came from | 6 of 18 right: 6 wrong, 6 declined |
| Precedent for a review comment | Real earlier cases, and how they got out, on 5 of 5 comments; all 18 citations checked out | No earlier case's outcome on any of the 5 |
| Odds and time at the board | Both right where both arms answered: 2 of 2 | Both wrong: 0 of 2 |
The rules and maps for a parcel
8 questions about real parcels: how many homes fit by right, a district's limits, how much of a lot lies in a floodway or a wetland. In each arm 21 answers were graded and 3 could not be checked against the record.
Precedent for a review comment
5 real first-round comments from the City of Charleston's Technical Review Committee: "has anyone else been flagged for this, and how did they get out?" The general AI found that the City publishes its agendas but not its comment letters. We hold the letters.
Odds and time at the board
On a second question neither arm gave a base rate; ours missed it through a defect in how it read the board's record.
Where we are no better: predicting comments
Given a project's plans and the code, a general AI caught 28.9% of 629 real first-round comments on 10 City of Charleston cases. With our data added it caught 31.8%. That difference, 2.9 points with a 95% interval of -2.0 to +6.9, is not significant. We do not predict review comments.
How we measured
Run on 2026-10-23. The general AI was a leading model with web search and web fetch on, given no files, asked each question once with no follow-ups. Our arm was the Ask a signed-in customer uses.
One run per arm per question, so run-to-run spread was not measured.
The comment test used 10 cases and 629 real first-round comments, an AI judge and a paired bootstrap interval.
In 2 of the 16, our stored number was wrong, and the answer still came out right because it read the code section itself.