How to test two versions of a call script: A/B testing step by step
Sado AI team · · 5 min read
In short
To A/B test a call script, change only one thing — for example, the opening line or how you make the offer. Split the list at random into two equal halves, call both on the same days and hours, and compare one number chosen in advance: for example, the share of answered calls that ended in an order. Don’t decide until each version has about 100 conversations. Beyond the numbers, listen to recordings to hear where people stop. Make the winner your main script and test the next change.
What an A/B test is and why test a script
An A/B test runs two versions of a script at the same time, and the numbers show which works better. Managers often argue “I think this line is better”. A test ends the argument: same customers, same time, one difference. Sometimes a small change matters: for example, opening with the customer’s own request instead of the company name.
What you can test in a script
- The opening line: “I’m calling from Charm Poyabzal” or “Yesterday you asked about boots on our website”.
- The first question: open (“Which model are you looking for?”) or a choice (“For yourself or as a gift?”).
- The offer: mention the discount at the start, or after the customer picks a model.
- The next step: “come to the shop” or “we’ll deliver by courier”.
- The time of the call: daytime or evening. Not a script, but tested the same way.
Change only one thing per test. Change two at once and you won’t know which one helped.
A/B testing step by step
- Pick one target number: for example, the share of answered calls that ended in an order. Call length or “the customer liked it” is not a target.
- Write two versions. A is the current script, B has one change. Everything else stays the same.
- Split the list at random: for example, even and odd rows in Excel. Don’t split by alphabet or district, or the groups will be different kinds of people.
- Call both halves on the same days and hours, with the same voice.
- Collect about 100 answered calls per version. With few conversations, a difference may be chance.
- Count, then listen to 10–15 recordings of each version: the number shows what happened, the recordings show why.
Example: two different opening lines
For example, a shoe shop calls people who left a request on its website. The list is split into even and odd rows.
| Measure | A: with the company name | B: with the customer’s request |
|---|---|---|
| Opening line | “I’m calling from Charm Poyabzal, our new collection is in” | “Yesterday you asked about winter boots on our website, so I’m calling about that” |
| Answered | 120 | 118 |
| Ordered | 14 | 23 |
| Share | 14 ÷ 120 ≈ 12% | 23 ÷ 118 ≈ 19% |
These numbers are only an example. With a gap like this, B becomes the main script and the next test starts: for example, when to mention the discount.
What version B sounds like
Sado: Assalomu alaykum, is that Dilshod? This is Shohruh from Charm Poyabzal. Yesterday you asked about winter boots on our website, so I’m calling about that.
Caller: Yes, I did. Do you have size 43?
Sado: We do, in black and in brown. Which do you prefer?
Caller: Black. How much are they?
Sado: The black ones are 680,000 so‘m. Shall we deliver tomorrow, or will you come to the shop?
Caller: Deliver them, I’m home in the evening.
Sado: All right, booked for tomorrow evening. The courier will call before setting off.
Running the test with auto-calls
With human operators a test breaks quickly: one reads the script word for word, another improvises, and moods change daily. An AI operator follows the same instructions on every call, so the difference comes more from the script. With Sado auto-calls you split the list into two Excel files and create two auto-calls in the Auto-calls section: same days and hours, only the “what to say” text differs. Put only people you may call on the list: those who left a request, or your customers.
Each call’s recording, transcript and collected answers are in the portal, and you can download the results to Excel to compare. You pay only for talked minutes: pricing. Try it live or call 78 778 01 23.
Questions
How many calls does an A/B test need?
There’s no exact rule, but about 100 answered calls per version is a good minimum. A small difference needs more. A conclusion drawn from a few calls may be chance.
Can we test three versions at once?
Yes, but each needs enough conversations, so the test takes longer. With a small list, start with two.
What should the share be counted from?
From answered calls, not from numbers dialled. Someone who didn’t pick up never heard the script and shouldn’t affect the test.