Three AI Models, Same 20 Products, Three Different Winners

What happened

Slobodan Manic ran a simple test and reported it in Search Engine Journal: give three AI models the same 20 products and the same bare request, and see what they buy. The products came from Shopify's catalog through Shop MCP, the keyless connection Shopify opened to AI agents on Oct. 13. It covers over a billion products.

The models were Claude Haiku 4.5, Sonnet 5 and Opus 5. Each made 50 fresh picks per search, with no budget, no taste, no shopper history and no agent-specific instructions. Across 1,980 picks in total, the models never agreed on the top product in any of the six main product searches.

Each model was consistent with itself. Sonnet's favorite took 94% of its picks on average, Opus's took 89% and Haiku's 67%. But change the model and the winner changed, and on three of the six searches the winner came from a different store.

Different models, different logic

Haiku and Sonnet leaned toward the product with the most reviews. On backpack, coffee beans and water bottle, that was their most-picked item. Opus behaved differently. In 222 of its 300 picks, its reasoning called the choice "safe" or "safest". It built a default shopper and bought the general-purpose option: the medium roast over the dark, the everyday backpack over the roll-top.

On coffee, Opus chose the medium roast 48 times out of 50. Haiku and Sonnet favored the dark roast with the most reviews.

The catalog's own ranking mattered less than you might expect. Haiku picked the catalog's first-ranked product 8% of the time, Sonnet never, and Opus 20%.

The review number problem

The water bottle search shows the risk. The catalog listed nobl's bottle at 5.0 stars from 447,221 reviews. The product page showed 5.0 from 6,537 when the author checked, and the catalog's record for one color showed 4.2 from 1,674. The author does not know what the 447,221 figure counts.

Haiku picked that bottle 45 times out of 50, and Sonnet 39 times. Neither could verify the number. They reasoned from whatever the catalog said. Opus never picked it and chose the Owala FreeSip every time.

Why this matters for merchants

In agentic commerce, a product list is not a sale. Something has to choose, and that something is a model with its own habits. Your visibility is no longer just about ranking in a catalog. It also depends on how your product data reads to different models.

Two caveats apply. This was one author testing Claude models only, with a deliberately bare setup. Real agents add their own private instructions, shopper context and preferences, so results in the wild will differ. The test shows the layer underneath, not the finished experience.

What to do now

• Audit your catalog data. Check ratings and review counts as they appear in feeds and catalog records against what your product pages show. Mismatches can drive picks.
• Keep titles, summaries and the first listed features clear. The models in this test saw only the title, store, price, rating, review count, a one-line summary and three features.
• Keep building genuine reviews. Two of three models favored review volume, and the third's "safe" logic still rewarded broad, general-purpose positioning.
• Decide your positioning. A niche product may win with one model's logic and lose with another. A general-purpose flagship variant may suit "safe" pickers.
• Test it yourself. Run your category's search terms through several assistants and record which products they choose and why. The stated reasons are useful clues.

The bottom line

There is no single AI shopper. There are many, with different biases. Merchants should treat agent-readiness as a data quality and positioning job, not a ranking trick.