A/B testing

A test splits your shoppers into groups and measures each group. The numbers tell you which setup earns more — before you commit to it.

A/B testing needs the Growth plan or higher. Each shopper stays in the same group for the whole test, so the comparison stays clean. The screen is A/B Testing in the PersonalizerAI dashboard. On the Free and Starter plans the screen shows an upgrade card, and the sidebar link carries a small Growth badge so you can see the gate before you click.

The life of a test

The filter tabs on the screen match these states: All, Running, Paused, Completed, and Draft. A paused test holds its groups — shoppers stay assigned. Completed is final.

What you can test

The wizard offers three test types. Each one sets what makes every group different:

You can run one test of each type at a time. A paused test still holds its type. Complete it to free the type for a new test.

Bundle widgets (Bought together) are never part of a test. They always show bought-together pairs, because the bundle discount applies only to those pairs. Their views, clicks and sales do not count toward any group.

Create a test

The wizard has five steps. The dialog shows Step 1 of 5 through Step 5 of 5.

1. Click New A/B test.
2. Select a test type under What do you want to test?
3. Click Next.
4. Name each group under Define variants.
5. Click Next.
6. Select a metric under Choose your primary metric.
7. Click Next.
8. Set the split under Traffic allocation (must sum to 100%).
9. Select the Target audience.
10. Click Next.
11. Check the summary, and name the test. Leave the name empty to use the suggested one.
12. Check the Safety guardrails. There is one by default: the app pauses the test if revenue per impression falls more than 20% below Control, once it is 95% sure. Change its percentage (5 to 50), switch it to another metric, click Add guardrail to guard more metrics (one guardrail per metric), or remove them all so the test never pauses on its own.
13. Click Launch test.

Click Save as draft instead to keep the test without starting it. Start a draft later with its Start button.

If the test is saved but fails to start, the wizard says so and keeps it as a draft. Click Launch test again to retry the start; this never creates a second copy.

The wizard, step 1: pick what you want to test.

Groups and split

Every test starts with Control — your store as it is — and Variant B. Click Add another variant for a third or fourth group. The split percentages must sum to 100. A 50/50 split reaches an answer fastest.

Control always serves your current configuration, unchanged — the other groups are measured against it. Every other group needs its own setting, matching the test type you picked:

• Model comparison — pick the Model this variant serves. It applies to every recommendation widget except bundle widgets for that group's shoppers while the test runs, and it takes priority over auto-pick: on a spot where auto-pick is on, the group's shoppers see the test's model, not auto-pick's choice. Control shoppers, and shoppers the test leaves out, still get auto-pick. If auto-pick is on anywhere, the review step names those spots before you launch.
• Discovery dial — set the group's Discovery rate (%), from 0 to 100. The control keeps your store's current rate.
• Personalized search ranking — set Personalized search ranking to on or off for the group. This is how you measure whether personalized ranking helps your store before you turn it on for everyone.

The wizard, step 2: name each group and set what it changes.

Audience

The Target audience options are:

• All visitors
• New visitors only
• Returning visitors only
• Mobile users only
• Desktop users only

Two details to know. A shopper counts as returning when the app knows their customer ID — usually a signed-in customer. Tablets count as mobile.

An add to cart pressed straight on a recommendation card counts toward the test three ways — as a click, as an add to cart, and, if the shopper goes on to buy, as a conversion. The shopper does not have to open the product page first. This matters if you are comparing setups where one makes buying quicker: before, those shoppers registered only a view, which quietly counted against the very variant that was working, and Add-to-cart rate read near zero for widgets whose whole point was the shortcut.

The metric

The metric step comes before the split, so the time estimate can use the metric you pick. The app tracks all four metrics on every test. The primary metric decides the winner:

• Revenue per impression — best for overall business impact.
• Click-through rate — best for comparing recommendation models.
• Add-to-cart rate — best for product relevance.
• Conversion rate — best for end-to-end funnel impact.

How long to run

The wizard shows an Estimated time to significance card for the metric you picked. For Conversion rate it uses your daily sessions and your store's conversion rate. For Click-through rate it uses your daily recommendation views and your store's click-through rate. Set Smallest lift to detect (%) to the smallest improvement you care about, measured against your current rate: 5% means 2.0% becomes 2.1%. Small stores and small lifts need more days. The app cannot estimate Add-to-cart rate or Revenue per impression tests, and says so.

The estimate is a guide only. A test ends on its own within 60 days: see When a test ends.

Read the results

Each test card shows a table with one row per group:

The Confidence column uses three labels:

• Significant — 95% or more. There is enough data to trust the difference.
• Approaching — 90% or more. Promising, but keep the test running.
• Not yet — below 90%. The difference can still be luck.

Control shows Baseline instead — the other groups are measured against it. After about 50 sessions, two charts appear: the confidence ranges per group, and the certainty over time. When a winner is called, the card shows a badge such as Winner: Variant B (+4.2%).

A running test: the per-group table, then the two charts.

Pause or end a test

A running test card has up to three buttons:

• Pause — freeze the test. Click Resume later; the groups hold.
• Call winner — declare the leading group the winner. This ends the test.
• End test — end the test with no winner. You cannot restart it.

Call winner and End test both ask you to confirm first. The Call winner box names the group, how sure the app is that it beats Control, and its lift. If the app is less than 95% sure, it says plainly that the difference could be chance.

When a test ends, every shopper goes back to your current settings. To keep the winning setup, make that change yourself, in the screen where the setting lives.

When a test ends

The app checks every running test once an hour, on the primary metric you chose:

• It calls a group the winner and ends the test only when all three are true: the test has run for at least 7 days, Control and every group each have at least 100 clicks (click-through rate), add-to-carts (add-to-cart rate) or orders (conversion rate), and the app is at least 95% sure the group beats Control.
• If no group gets there within 60 days, the test ends with No clear winner. Every shopper sees your current settings, and you get a notification and an email.
• For Revenue per impression tests the app cannot compute that certainty, so it never calls a winner. Check the numbers and call the winner yourself; if you haven't after 60 days, the test ends with no clear winner.
• Resuming a paused test starts both clocks again.

Safety guardrails

Before it checks for a winner, the app checks the test's safety guardrails. A guardrail pauses the test when a group does worse than Control on its metric by more than your percentage (20% unless you changed it) and the app is at least 95% sure the drop is real.

You can guard any of these: Revenue per impression, Click-through rate, Add-to-cart rate (add-to-carts out of clicks), Conversion rate, and Add-to-carts per impression. A guardrail on the metric the test is judged on is allowed: it pauses a group that is clearly losing instead of waiting for you to end the test.

It never pauses on a few unlucky hours. Control and each group need at least 200 impressions (widget views, or searches in a search test) before the app judges them, 200 clicks for add-to-cart rate, or 200 sessions for conversion rate. Revenue per impression and conversion rate also need 10 orders per group.

A test paused this way says why, for example: Paused automatically: revenue per impression is ~23% below control (96% likely). Shoppers see your current settings while it is paused. You can resume it, but if the drop holds, the app pauses it again.

You hear about it right away: a notification in Notifications with a link to the test, and an email. The email follows your Product updates email setting, so if you turned that off you still get the in-app notification.

Tests started before the guardrail existed run without one.

The wizard and each running test card show this rule for the test's metric.

On a completed test with a winner, a Mark winner applied button appears. It records that you adopted the winning setup. It does not change your settings.

What happens on downgrade

When you move to a plan without A/B testing, the app pauses your running tests automatically. Those cards say Paused automatically when your plan dropped below Growth. Your tests stay visible under Your past tests, below the upgrade card, with all their results. You can still end a test there; starting, resuming and calling a winner need Growth. Upgrade again to resume the tests the app paused. See Plans, billing & limits.