Meaningful test results require a sample size determined by four factors: the baseline conversion rate, the minimum detectable effect (the smallest improvement worth identifying), the desired confidence level (typically 95%), and statistical power (typically 80%). Lower baseline conversion rates and smaller detectable effects both require larger samples. Use a sample size calculator before launching any test, not after.
The relationship between these variables is mathematical, not intuitive. A site with a 2% baseline conversion rate testing for a 0.5 percentage point improvement needs a much larger sample than a site with a 10% baseline testing for a 2 percentage point lift. Online sample size calculators (Evan Miller's, Statsig, Optimizely, AB Tasty) compute the required visitors per variant based on these inputs. The critical discipline is calculating sample size before the test starts and running the test until that threshold is reached, regardless of what intermediate results show. Stopping a test early because it looks like a winner (or loser) violates the statistical assumptions and produces unreliable conclusions. Higher traffic volumes allow faster data collection with the same sample requirements, but they do not reduce the minimum sample needed for a given confidence level and detectable effect.