A/B testing

In an A/B test, version A is the current page (the control) and version B is the change (the variant). Visitors are assigned to one version at random and keep seeing it. Because both groups arrive at the same time from the same sources, a difference in results can be put down to the change rather than to a campaign, a holiday or a news cycle. Google Optimize, Google’s free testing tool, shut down on 30 September 2023, so teams now run tests in dedicated experimentation platforms or in the experiments feature of their analytics stack.

Two ideas decide whether a result means anything. Sample size is how many visitors each version needs. You estimate it before starting from three numbers: the current conversion rate, the smallest lift worth detecting, and the confidence level you want. Statistical significance tells you how unlikely the observed gap would be if the two versions actually performed the same. Many teams use 95% confidence as their threshold. Small effects and low conversion rates need large samples.

Why it matters for a SaaS website

A test replaces opinion with evidence, which matters most on pages that already make money. Redesigning a high-traffic homepage or pricing page on taste alone puts pipeline at risk. A test limits that risk to part of the traffic for a few weeks. The catch is traffic: many B2B sites don’t have enough visitors on a page to reach a reliable answer, and there research and usability testing are the better tools.

How DesignMe handles it

We design variants around one clear hypothesis and a metric agreed up front. On Seamless.AI we redesigned the homepage first and pushed it live head to head against the existing one, and the new version roughly doubled conversion. Seamless then rolled out the rest of the redesign in batches, every change A/B tested against what was live. Winners along the way included a headline that beat the original by 12.5% and a “Try It Free” CTA that outperformed the alternatives.

Example

Say a pricing page converts 3% of visitors to a trial and the team wants to detect a lift to 3.6%. At 95% confidence and 80% power, a standard calculator asks for roughly 14,000 visitors per version. With 4,000 pricing visits a week, that is a seven-week test, so the team tests one meaningful change, not a button colour.

Common mistakes

• Peeking and stopping early. Decide the sample size first and wait for it.
• Testing on thin traffic. A test that can’t reach its sample size produces noise, not answers.
• Changing ten things at once. You learn that the bundle won, not which part worked.
• Watching only the top of the funnel. More sign-ups with fewer qualified demos is not a win.
• Leaving finished tests running. Google may treat a long-running experiment as an attempt to deceive search engines.