A/B Testing
A/B Testing
What is A/B testing?
- Randomly show version A or B and compare performance
- Choose the better version with data, not gut feeling
- Change one thing at a time, with a large enough sample
What A/B testing changes is not the screen but the mood in the meeting room. Meetings where the loudest voice won with “I think the blue button is better” become meetings where everyone reconvenes two weeks later with numbers. The bigger benefit is that marketers earn the right to be wrong. Even when version B loses, you gain hard evidence that “this direction is out,” so even losing ideas become groundwork for the next experiment. This is where the gap opens between a team that runs 40 tests a year and one that does a full redesign once a quarter.
In the design, decide the sample size and duration first. To detect a lift in conversion from 2% to 3% at a 95% significance level and 80% power, you need roughly 2,000–3,000 people per group. For a B2B site with 5,000 monthly visitors, that means a single test takes a month. That is why in practice teams use micro-conversions such as resource downloads or reaching the pricing page as the primary metric instead of final inquiries, and run the test for at least two uninterrupted weeks to absorb day-of-week swings. Multivariate tests, which change several elements at once, need more sample for every added combination, so they don’t suit low-traffic companies.
A frequently confused practice is the “before-and-after redesign comparison.” Saying conversion rose this month versus last month mixes in seasonality, ad spend, and competitor news, so you can’t isolate the cause. The essence of A/B is randomly splitting traffic from the same period, which is why it is more reliable than a before-and-after comparison. By contrast, a bandit approach that gradually sends more traffic to the winner suits short-term campaign banners, but it doesn’t record why something won, which works against building up learning.
The most common mistake in practice is peeking midway and stopping early. If you end the test on day 3 because version B is ahead, you declare a win on a gap that might have reversed with more data. Statistically significant doesn’t mean profitable. Even if click-through rises 12%, if that traffic doesn’t turn into contracts, all you add is wasted vetting time for the sales team. Teams also keep tweaking button colors and copy and go six months without seeing the real problem: the offer itself is weak.
John KimInstructor John Kim, author of B2B Marketing Design and CEO of Nanum Consulting
Euijung KimBrand strategist with 30 years in advertising, formerly a division head at Daehong CommunicationsIt applies the principle of controlled experiments (A/B) from statistics to the web and marketing, and became a standard method as digital experimentation tools matured.
Let’s look at a hypothetical case: Company A, an industrial sensor manufacturer with 90 employees and KRW 13 billion (≈ US$9,726,200) in annual revenue. Its product page gets 4,200 visits a month, but only 28 quote inquiries (0.67%). The marketing team was split between “the headline just lists specs” and “there’s no price,” and had been stuck for two months.
- Team Lead Kim said, “Meetings won’t settle this. Let’s run it for four weeks,” and built two versions that differed in just one headline. Version A read “3-axis high-precision sensor system”; version B read “30% less equipment downtime.”
- Data analyst Lee set a firm rule not to stop until each group had 2,100 visitors and 40 conversions, and locked the end date on the calendar. “Don’t ask for results in week 2” also became a team rule.
- With only 28 quote inquiries a month, the team couldn’t detect a difference, so they switched the primary metric to catalog downloads. When Sales Manager Park said, “If only downloads go up, we’re the ones who suffer,” they added inquiry volume and the share of consultations that qualify as secondary metrics.
- On day 10, B pulled ahead and the CEO said, “Let’s just go with B.” But Team Lead Kim pushed back—“If we stop now, it could flip”—and held out until the planned end date. Sure enough, by day 15 the gap had shrunk by half.
- After four weeks, download conversion rose from 3.1% to 4.4%, quote inquiries grew from 28 to 41 a month, and the lead qualification rate held steady, moving from 61% to 64%. The team picked whether to show pricing as the topic of its second test.
- When you have plenty of visits and samples
- You need to validate the effect of a single element
- Your conversion rate has room to improve
- When traffic is too low to produce statistics
- When you need to change several elements at once
- Early stages where fast decisions matter more