The author has repeated many times that you must verify AI’s answers yourself to ensure they are correct. Just how inaccurate is AI? Staff at the Tow Center for Digital Journalism conducted a comprehensive test of the 8 most popular AI search engines, and the results were quite surprising.
Don’t trust AI search engines too much; research finds they are “confidently wrong” 76% of the time.
First, let’s understand how the Tow Center tested these AI search engines. The 8 chatbots in the study include free and premium models with real-time search capabilities (able to connect to the internet in real time):
- ChatGPT Search
- Perplexity
- Perplexity Pro
- DeepSeek Search
- Microsoft Copilot
- Grok-2 Search
- Grok-3 Search
- Google Gemini
This research is mainly about AI chatbots’ ability to accurately retrieve and cite news content. In addition, the Tow Center also wanted to understand how chatbots behave when they cannot execute requested instructions. To test this, the Tow Center selected 10 articles from 10 different publishers. Then, excerpts were taken from each article and provided to each chatbot. The chatbots were then asked to do simple tasks, such as identifying the article’s title, original publisher, publication date, and URL, as shown in the example below.
Next, the unit divided the chatbot’s responses into 6 categories:
- Correct: All three attributes are correct.
- Not entirely correct: some attributes are correct, but the answer lacks information.
- Partial error: some attributes are correct, while others are incorrect.
- Completely wrong: All 3 attributes are incorrect and/or missing.
- Not provided: No information was provided.
- Crawler blocked: The publisher does not permit chatbot crawlers in its robots.txt.
As you will see below, AI search engines often make mistakes, but arguably the bigger problem is how they make them. Regardless of accuracy, chatbots almost always respond with confidence. Research has found that they rarely use qualifiers or admit that they cannot carry out instructions. The chart below shows the accuracy of responses and the level of confidence given. As you can see, almost all responses fall into the “confident” category, but there is a lot of red.
For example, among the answers returned by Grok-3, up to 76% are “confidently wrong” or “partially wrong.” Remember, Grok-3 is a high-end model that costs $40 per month, and its performance is worse than the free Grok-2 model. The same situation can be seen with Perplexity Pro and Perplexity. High-end paid models ($20 per month in the case of Perplexity Pro) do not necessarily improve accuracy, but they do seem to appear more confident in their wrong answers.
Some AI search engines have signed agreements with publishers to allow access to specific publications. You would think that chatbots would be excellent at accurately identifying information from these publications, but that is not always the case. The chart below shows 8 chatbots and the publishers they have licensing agreements with. As a reminder here, the bots were asked to determine the article’s title, original publisher, publication date, and URL. Most chatbots were able to do this with high accuracy, but some failed. For example, when ChatGPT Search dealt with the San Francisco Chronicle, a publication it has a partnership with, it was wrong 90% of the time.
On the other hand, some publications have blocked AI search engines from accessing their content, yet research shows that in practice, this is not always effective. Some search engines appear to disregard publishers’ decisions in this regard. For example, Perplexity was able to accurately identify all 10 citations from National Geographic that were paywalled and blocked to crawlers. More chatbots not only accessed blocked websites but also provided inaccurate information from them. Grok and DeepSeek were not shown in the chart because they did not disclose their crawlers.
That said, what do all those numbers mean for users? Clearly, relying solely on AI search engines to improve accuracy is a fairly risky approach—even high-end models with licensing agreements can confidently feed users incorrect information. This serves as a clear reminder that critical thinking and cross-verification remain fundamental internet skills in the AI era. If you’re interested, you can check out The full study on Columbia Journalism Review, learn more about discoveries you’ve never imagined.
Source: KOCPC Chinese