Alright, but which one is superior?
These distinctions don’t inherently imply that the AI-generated outputs are “inferior,” of course. The researchers discovered that GPT-driven searches were more prone to reference sources such as corporate entities and encyclopedias for their data, while almost never referencing social media platforms.
A tool for LLM-based analysis indicated that AI-enhanced search results also generally addressed a comparable number of identifiable “concepts” as the conventional top 10 links, implying a similar degree of detail, variety, and novelty in the outcomes. Concurrently, the researchers noted that “generative engines tend to condense information, occasionally leaving out secondary or ambiguous elements that traditional searches maintain.” This was particularly noteworthy for more ambiguous search queries (like names shared by various individuals), for which “organic search results offer superior coverage,” the researchers observed.
The AI search platforms might possess an advantage by integrating pre-trained “internal knowledge” with data sourced from referenced websites. This was particularly evident in GPT-4o with Search Tool, which often refrained from citing any web sources and merely delivered a direct answer based on its training.
Nevertheless, this dependence on pre-trained information can pose a limitation when seeking current data. For search terms derived from Google’s Trending Queries for September 15, the researchers found that GPT-4o with Search Tool frequently replied with prompts like “could you please offer more information” rather than actively searching the web for recent information.
While the researchers did not conclude whether AI-driven search engines were ultimately “better” or “worse” than conventional search engine links, they advocated for further research on “new evaluation methods that collectively assess source diversity, conceptual coverage, and synthesis behavior in generative search systems.”
