In a simulated marketplace experiment conducted by Microsoft using 100 virtual customers and 300 virtual businesses, it was observed that AI agents are prone to manipulation and often struggle with complex choices—much like human shoppers. The study concluded that while advanced models performed better than simple baselines, such as random selection or always choosing the cheapest option, they still have room for improvement before being widely adopted.

The experiment modeled transactions such as food ordering or hiring home improvement services, where each customer had specific requirements to be satisfied. Interestingly, both proprietary and open-source AI models showed better performance than basic methods but did not meet near-optimal results consistently. The top performer among the tested models was GPT-5.

Challenged by Complexity

As scenarios became more complex, the AI agents’ performance dipped. With increased options and search results, these agents tended to settle for the first good enough” option rather than making thorough comparisons. Most models only contacted a small fraction of available businesses, with some repeatedly reaching out to non-applicable service providers.

The experiment also highlighted how vulnerable these AI agents were to website manipulation techniques. Similar marketing tactics that influence human shoppers proved effective on the bots. The conclusion from Microsoft was clear: AI agents should assist, not replace, human decision-making.”

Customers Skeptical

Many have expressed interest in using AI agents for purchasing decisions, according to Javelin Strategy & Research. However, there’s a significant gap between interest and actual usage. Christopher Miller, Lead Analyst at Javelin, noted that while consumers are willing to use chat tools for gathering information, they may not always complete purchases through these agents.

There is substantial evidence of consumer interest in using chat-based tools to gather purchase-related information,” said Miller. However, if people continue to rely on Google and other search engines for information, the potential for AI agents to become a regular part of the buying process may be smaller than some anticipate.”