Key obstacles SMBs face with A/B testing
We have interviewed dozens of SMBs to understand their challenges with A/B testing and have run over 100 A/B tests in our own careers at companies of all sizes, from startups to big tech. From this, we've identified four main obstacles that SMBs face when running A/B tests with in-house teams: they don't have enough traffic, they lack the required expertise, they can't invest enough time, and considering all factors, the return on investment is likely negative.
Problem #1: Not enough traffic
The single most important barrier to A/B testing is a lack of traffic or sample size. A/B testing requires testing a sample to determine a statistically significant difference in performance between two or more variants. The required sample size is determined by the effect size, baseline conversion rate, number of variants tested, and required significance level.
For example, an SMB eCommerce store with 400k monthly visitors and a 1.5% conversion rate wanting to detect a 5% uplift would need a sample size of 830k visitors — about two months of data. But this creates problems: iterations are too slow (80–90% of A/B tests fail, per HBR research), and tracking becomes unreliable as ~30% of cookie data is lost within a month.
Problem #2: Lack of expertise
The success of each test ultimately depends on the quality of the hypotheses being tested. The real limitation lies in experience with formulating hypotheses specific to the conversion problem at hand.
In big tech companies, ownership is highly departmentalized — one PM owns registration, another owns onboarding. These PMs are experts in their specific domains. However, building this kind of knowledge internally is much harder for smaller organizations. They can't compete with big tech for hiring, and they can't afford to dedicate a PM solely to growth optimization.
Problem #3: Lack of time
A/B testing is a manual and time-consuming process: analyzing data to craft a hypothesis, designing variants, implementing them in the codebase, setting up the split test, monitoring assignment, and analyzing results. This is particularly challenging for small companies without a dedicated testing team, where there are always more pressing core business activities.
Problem #4: Negative ROI
For most SMBs, the expected rewards of A/B testing often do not outweigh the cost. Consider an SMB SaaS company with €10M ARR and a dedicated team: the team costs at least €500k per year, plus ~€100k for tooling. Assuming a 5% incremental uplift, that's €500k in revenue vs. €600k in costs. Newly formed teams rarely deliver any uplift in the first 12–18 months, making it an investment only high-growth companies should consider.
Which options do SMBs have?
There are three paths forward:
- Not running any A/B tests: Focus on building core product value and delay testing until the company has more traffic and can afford a dedicated team.
- Working with an agency: Agencies bring experience from optimizing many products and can quickly identify critical areas for optimization with high-quality hypotheses.
- Using AI & ML optimization: A new category of AI-powered tools, like Levered, leave the traditional path of A/B testing and use machine learning algorithms instead. This radically reduces cost, time, expertise, and traffic requirements.
The future of optimization: How AI is changing the game
While classic A/B testing has remained largely unchanged for decades, machine learning and AI are now transforming product optimization at an unprecedented pace. This is especially good news for SMBs, as it makes continuous product growth optimization accessible to companies that can't rely on traditional A/B tests.
ML algorithms require significantly fewer user interactions to predict which designs will convert best, because the statistical approach behind machine learning is far more efficient at computing probabilities and updating predictions. Additionally, AI dramatically reduces the cost per test by automating nearly all manual steps — from identifying what to test to designing changes and implementing them in the codebase.
In the coming years, we can expect a wave of AI-powered product optimization tools that will make a large part of A/B testing obsolete. These innovations will empower SMBs to improve their products, lower customer acquisition costs, and reinvest those savings into building products their customers love.