We Asked 10 Store Chatbots the Same 12 Shopper Questions

“Do you ship to Singapore, and how much does shipping cost?” One store's chatbot answered: free on orders over 400 SGD, 25 SGD below that. The store's own shipping page lists Singapore as free, with no threshold. We asked that question, and 11 others, to the chat of 10 online stores on November 9, 2026.

Of the 120 answers, 76 passed: they agreed with the store's own pages, or the bot said it could not answer and gave a working route to a person. Nine were wrong. Nine more did not answer the question asked. The rest were partly right, which in practice means the shopper still had to go and look it up.

How we ran the test

We looked for public online stores with a chat that answers typed questions by itself. We tried the chat on 18 stores. Eight dropped out before the first question: four wanted a name or an email before answering anything, two offered only a button menu that ended in an email form, one store was behind a password page and one chat would not open. The ten we tested are nine Shopify stores and one on Magento. The only WooCommerce store we found with a chat widget had no automated answers, so the test has none.

The chats come from Gorgias (three stores on its AI agent, one on article suggestions), Intercom (two), Shopify Inbox, Ada, Zendesk and one store's own branded assistant. Each store got the same 12 questions, in the same order, from a new anonymous visitor in Singapore, between 4:26 and 5:44 pm Singapore time. We typed no personal details and submitted no forms. When a bot moved the chat into a form or a sign-in screen, or a reply never came, we asked the remaining questions in a fresh chat. One store's chat stopped taking questions after its first suggested article, so it got each question in a fresh chat of its own.

Three questions were filled in per store from its own catalogue: a product type and a price cap that some, but not all, of those products fall under, and one product in a size that the catalogue showed as offered or sold out.

1. Where is my order? I placed it last week.
2. How many days do I have to return something?
3. Can I return an item I bought on sale?
4. Are returns free, or do I pay for return shipping?
5. Show me [product type] under $[cap].
6. Is the [product] available in size [size]?
7. Do you ship to Singapore, and how much does shipping cost?
8. How long does standard delivery take to the US?
9. Can I change the delivery address after I place an order?
10. Can I pay in instalments, like with Afterpay or Klarna?
11. Will the [product] be cheaper next month?
12. Can I talk to a real person?

Every answer was graded against the store's own returns, shipping, help centre, contact or product page, and the catalogue saved during the test. Grades: correct, partly, wrong, no answer, or handed off well, which counts as a pass only when the store's pages do not answer the question and the bot routes the shopper to a person. Two answers stayed unverified because no store page gives a figure to check them against. We kept every transcript, a screenshot of every reply and the page quote behind every grade.

The results, store by store

We name only the store with the best result. The others carry a letter, with the platform and the chat product they run.

Store H gave no wrong answers, but six were only partly right: the direction was right and the number, the condition or the route a shopper needed was missing. Store J answered every question with a suggested help article. Four suggestions were on topic, and one excerpt stated the answer in the chat. Eight were about something else: “Can I talk to a real person?” brought back an article on referring a friend.

The three stores on the same Gorgias AI agent scored 10, 7 and 5. The product in the chat window did not decide the result. What the bot could read did: which pages it was given, for which market, and whether it searched the catalogue the shopper was looking at.

What every bot got right

No bot made up an order status. Eight of ten asked for an order number or a sign-in, or pointed to the order tracker, before saying anything about a parcel. No bot promised a cheaper price next month either; eight said plainly they could not know. Nine of ten found a way to a person when asked, though at around 4 am on the US East Coast many of those handoffs asked for an email because the team was offline.

Koala's assistant gave the best set of answers. Asked about a mattress in a size it does not come in, it said so, listed the sizes it does come in and named the three mattresses that are made in that size.

Where the answers broke

By question, the pass counts out of ten stores were: talk to a person 9, order status 8, future price 8, return window 7, instalments 7, sale returns 6, return cost 6, size in stock 6, price cap 5, shipping to Singapore 5, address change 5 and US delivery time 4. Four patterns explain most of the misses.

The answer came from another market. Store I's site showed us its Singapore storefront in Singapore dollars. Its bot quoted prices in another country's currency, told us nothing in the category we asked about was under 120 when the Singapore page listed an item at SGD 85, and gave a Singapore shipping fee its own page does not charge. Store G told us it ships to Singapore for $9.99; its shipping policy lists the US and Canada only. If your store sells in more than one market, your bot needs to know which one the shopper is in.

The search missed a product under the price. Store D's bot said it could not find anything under $100 in the category we asked about. Its category page lists one item from $95, and the page's own price filter shows one item in the $0 to $100 band. A shopper who trusts that reply leaves.

Stock was guessed or dodged. Store G said a product in one colour and size was in stock; the catalogue we saved during the test marked that exact option sold out. Store F was given the product's full name and asked us for the product name.

Delivery times came out close but wrong. Two bots quoted US delivery a day or two shorter than their own shipping pages (2 to 4 days against 2 to 5, 5 to 8 against 5 to 10). For two other stores no page gives a number, so we could not check what their bots said. A shipping page without a number leaves the bot to find one somewhere else.

Our own demo bot

We ran the same 12 questions on our fictional HoverBot demo shop, Tidewalk, a sandal store with three styles. It passed 11. It missed the instalments question: it said it had no details on Afterpay or Klarna, while the shop's page lists the payment methods as card or PayNow. The right reply was “we take card or PayNow, no instalments”. Tidewalk is not counted in the results above. It is a small shop whose pages we wrote ourselves, which makes it an easier test than a real store, and we report it so the miss is on the record next to everyone else's.

Run the 12 questions on your own bot this week

The store chatbot grader walks you through the same questions with our rubric and gives you a scorecard.

1. Open your store in a private window and ask the 12 questions above, filling in a real product, size and price cap from your catalogue.
2. Next to each reply, paste the sentence from your own page that settles it. If you cannot find one, the page needs that sentence before the bot can get it right.
3. If you sell in more than one country, ask the shipping and price questions from a second storefront and check the currency and the fee.
4. Ask for a price cap that you know a product sits under, and a size you know is sold out.
5. Ask for a person outside your support hours and read what the shopper sees.

Want to see how a bot answers from approved store pages? Ask the Tidewalk demo the same questions.