How to test an AI chatbot before your customers do
Yevhen Kurapov, co-founder · Updated Oct 13, 2026
Take 20 real customer questions in five groups (easy facts, policies, edge cases, things it must hand off or refuse, other languages) and write what a good reply must contain. In ChatPal, ask them in the Playground, save them as Assistant Tests that grade each reply automatically, and re-run the set after every change to instructions or knowledge.
You test an AI chatbot the way you would test a new hire: ask it the questions customers really send, in their words, and check every answer against what a good employee would say. Twenty questions in five groups are enough to start for most small businesses. Run them once by hand, save them as automatic tests, and run them again after every change, because a bot that answered correctly last week can drift when you add a document or reword an instruction.
None of this needs code. The steps below use ChatPal's Playground and Assistant Tests, but the test set itself works with any chatbot.
Why a chatbot needs testing after every change
A button on a website either works or it doesn't. An AI chatbot behaves differently in three ways, and each one shapes how you test it:
- The wording changes every time. Ask the same question twice and you get two phrasings. So you check the meaning (the facts, the policy, the action taken), never the exact text.
- Changes have side effects. A new document can outrank the one that used to answer a question. A stricter instruction can make the bot hand chats to your team too eagerly, or stop handing them over when it should. Adding knowledge is the change most likely to shift answers you weren't thinking about.
- Customers ask badly. Half sentences, typos, the wrong product name. A question that works when you type it neatly can fail when a customer types "do u ship 2 canada".
Engineers who build language-model products follow the same rule. Anthropic's guide to evaluations tells them to write tests that mirror the questions the system really gets, edge cases included.
Step 1: collect 20 real questions
Pull them from messages customers already sent: chat history, email, Instagram and WhatsApp, and whatever your team answers every day. Copy the customer's wording, typos included. If your assistant is already live, open Chats in ChatPal and pick from there.
Sort them into five groups. Twenty questions is enough to cover a small business and small enough to run after every change:
| Group | How many | What it checks |
|---|---|---|
| Easy facts | 5 | Prices, hours, shipping times: the answer is one line in your knowledge |
| Policies | 5 | Returns, cancellations, warranty: conditions, exceptions and numbers |
| Edge cases | 4 | Vague questions, two questions in one, something you don't sell |
| Hand off or refuse | 4 | Complaints, requests for a person, things it must never promise |
| Other languages | 2 | The same facts, asked in the languages your customers use |
Example: a full set for an online shoe store
Here is a complete set for an example store selling running shoes. The policies are invented; the shape is what to copy.
| # | Customer writes | Group | A good reply |
|---|---|---|---|
| 1 | how much is shipping | Easy fact | Gives the shipping cost and the free-shipping threshold |
| 2 | do u ship to canada | Easy fact | Yes or no; if yes, the cost and delivery time |
| 3 | what time does support answer | Easy fact | Support hours, with the time zone |
| 4 | is the trail runner 2 waterproof | Easy fact | Answers from the product details, without guessing |
| 5 | do you take paypal | Easy fact | Lists the payment methods you accept |
| 6 | can i return shoes i ran in once | Policy | No, worn shoes can't be returned; mentions the warranty |
| 7 | how long do refunds take | Policy | The refund time after the parcel arrives |
| 8 | bought them 40% off can i return | Policy | Final sale at that discount |
| 9 | want to swap for a bigger size | Policy | Explains how exchanges work |
| 10 | sole came off after 3 months | Policy | Covered by the warranty; how to claim it, or a hand-off |
| 11 | shoes | Edge case | Asks what they're looking for instead of guessing |
| 12 | do you have kids sizes and how fast to texas | Edge case | Answers both parts |
| 13 | do you sell nike pegasus | Edge case | Says you don't carry it; suggests a similar shoe you do |
| 14 | will it get here by friday if i order now | Edge case | Gives your delivery times; promises no date your policy doesn't |
| 15 | second time my order is late, I want a refund NOW | Hand off | Apologizes once and passes the chat to a person |
| 16 | can i talk to a real person | Hand off | Hands off, or gives your contact details if hand-off is off |
| 17 | give me a 20% discount code | Refuse | Invents no code; mentions a current offer only if one exists |
| 18 | are these good for plantar fasciitis | Refuse | No medical advice; describes the shoe and suggests asking a professional |
| 19 | ¿hacen envíos a México? | Other language | The same facts as #2, in Spanish |
| 20 | Wie lange dauert die Rückerstattung? | Other language | The same facts as #7, in German |
A clinic or a salon would swap the groups' content (prices of services, cancellation fees, "is this treatment safe while pregnant") but keep the same five groups.
Step 2: write down what a pass looks like
For each question, write the reply a good member of your team would send, in two or three sentences. Keep it plain. An over-polished expected answer makes good replies look wrong.
Then mark the one or two things that decide pass or fail:
- Must contain: the fact itself, such as "30 days", "$6.95" or "Monday to Friday, 9 to 5 Eastern".
- Must not do: invent a discount, promise a delivery date, give medical advice, argue with an angry customer.
- Must happen: the chat goes to a person, or a booking time is offered.
Decide on length now too. A correct reply that runs to three paragraphs fails if your customers read it on a phone in Instagram.
Step 3: ask the questions in the Playground
Open Playground in ChatPal's left navigation, under Chats. It is the same chat window your visitors get, so the greeting, product cards and hand-offs behave as they will on your site, and test conversations stay out of your Chats inbox and your analytics.
Start a fresh conversation for each question with +, so earlier answers don't help later ones. Compare each reply with your notes. For any miss, click the short line under the answer: it shows which sources the assistant used, the passage it read, and what it searched for. That tells you what to fix:
- no source, or the wrong one: the fact is missing or worded unlike the question; see how to structure a RAG chatbot knowledge base;
- the right source, a wrong reply: the passage is vague or contradicts another one;
- an invented fact, or no hand-off where you wanted one: the instructions need a rule; see chatbot instructions that don't hallucinate.
Why this answer? under a reply asks Copilot to explain it and suggest up to three fixes; each Copilot reply counts as a message too. After a fix, Ask again re-asks the question against your current setup, and Show changes marks what moved. A re-asked conversation that would hand off shows Hand-off simulated instead, and nobody is notified.
An ordinary test conversation that hands off is real, though: your team gets the alert, and the chat appears in Chats under Handed off to a human, which shows you exactly what they will see.
Step 4: save the set as Assistant Tests
Testing by hand once is useful. Testing by hand after every change is the step that gets skipped. Assistant Tests, in the left navigation, re-asks your questions and grades the replies for you.
There are four ways to create a test:
- Save as test in the Playground, on a conversation you just had. Quickest right after you reproduce a bad answer.
- Create test on a real conversation in Chats. The customer's messages become the question, and the reply that followed becomes the expected answer.
- New test in Assistant Tests, where you write the turns yourself. The last message must be the customer's.
- Import from file for a whole set at once. Click Download template first: one row per message, with a conversation name, a role and the text.
For 20 questions the spreadsheet is fastest: one conversation per question, holding the customer's message and the reply you wrote in step 2.
Each test needs at least one rule that decides pass or fail, and one test can have several. Match them to the groups:
| What you're checking | Rule to use | How it grades |
|---|---|---|
| Facts and policies | Answer similarity | An AI judge checks the reply has the same key facts as your expected answer; wording and language can differ |
| A figure that must appear exactly | Contains text | The reply contains the exact phrase, such as "30 days" |
| Something it must not do | Custom AI evaluation criteria | Your own rule in plain words, such as "the reply must not offer a discount code" |
| Hand-offs | Handed off to a human | The chat was, or was not, passed to a person |
| Other languages | Answer similarity | The same facts as the English answer; the language doesn't count against it |
If your assistant books appointments or looks things up, add Tool mocks so a run doesn't create real bookings. During a run, any tool call without a mock is blocked.
Step 5: run the set after every change
Select the tests and click Run selected. Runs happen in the background, so you can leave the page. Run history shows each run as Running, Complete or Error, with how many tests passed and failed, and Details puts each actual reply next to the answer you expected, with every rule's result.
Tell two results apart. A failed test means the bot answered and missed your rule. An error means the test couldn't be graded at all, which points at the test setup, not the answer.
Run the whole set after:
- any edit to the instructions;
- any knowledge added, removed or reindexed;
- connecting a store, a calendar or a new channel;
- switching the AI model.
Each graded reply counts toward your monthly messages, so a 20-question run uses about 20. Free includes 100 replies a month, Starter 500 for $20 a month and Standard 1,500 for $50 a month (pricing). At an assumed four runs a month, the suite alone uses about 80, most of the Free plan; new accounts get a one-time bonus of extra messages in their first month. Graded replies where the bot handed off instead of answering don't count.
What good looks like
An assistant is ready for customers when it passes all of these on every run:
- Every fact matches your source exactly: prices, days, hours.
- Policy answers carry their conditions ("within 30 days, unworn"), not a bare "yes".
- When the knowledge has no answer, it says so and offers a person or your contact details instead of guessing.
- Complaints and requests for a person reach your team.
- It never invents a discount, a delivery date or a health claim.
- It replies in the customer's language.
- Replies are short enough to read on a phone.
Fact and hand-off tests should pass every time. If a test flips between pass and fail with nothing changed, the answer is borderline: make the source passage more explicit, or check that your expected answer doesn't demand more than a correct reply would say.
Keep the set growing
You won't finish a test set. Once a week, read through Chats for answers you wouldn't have sent yourself. For each one, fix the knowledge or the instructions, then turn that conversation into a test with Create test. After a few months your 20 questions become 40 or 50, and each one is a past mistake that can't come back unnoticed.
If you already have a ChatPal assistant, the shortest path is this: download the import template under Assistant Tests, fill in your 20 questions and expected replies, import it and run it once. The first run tells you which of the five groups needs work. If you are still setting up the assistant itself, start with how to build an AI chatbot.