We’ve advised over 140 companies—including Amazon, Dropbox, and WIRED—and run thousands of split tests.
Those tests have made tens of millions of dollars.
Today we’ll show you:
- How to discover what’s working on your site (and what isn’t)
- How to prioritize test ideas
- How to implement the ideas
- Tools you should use
- How to react to the results
Let’s get started.
What is split testing?
Split testing, also known as A/B testing or conversion testing, involves comparing multiple versions of a webpage or element to determine which performs better.
For example, you can “split” up traffic to your homepage.
- The original—also called the “control”—gets half your traffic.
- The variation—also called the “treatment”—gets the other half.
You then look at the results. Did the new homepage increase sales? Great! Promote it so that is your new homepage headline. If it didn’t increase sales, keep the original. Either way, you repeat the process.
Over time this process ensures your pages become more effective, user-friendly, and profitable.
Here’s how to get started:
Find out what’s working—and what isn’t
Here’s a common mistake marketers make when split testing: they jump right into a list of things they want to change.
This is a bad idea.
Instead, gather feedback about what’s working with your website (and what isn’t).
1. Use Hotjar surveys to ask users “If you didn’t buy today, could you explain why? Thanks!” (You can fire these on-page surveys when users are about to leave the page.)
2. Use Hotjar heatmaps to see what elements people click on.
3. Use email surveys to ask your current customers (or prospects) where they first heard about you, what convinced them to buy, and what almost stopped them from buying. (You can do this easily with a Google Form or Typeform.)
4. Run usability tests. Either with a company like UserTesting.com or by asking someone you know to use the site and “speak their thoughts out loud.”
5. Look at your analytics. What are the top pages? Which ones generate the most revenue? Start with those.
If you do just these five things, you’ll be light years ahead of most marketers and designers.
Generate your test ideas
As you gather your feedback—using the five approaches we reviewed—ideas will naturally present themselves.
For example, your email survey reveals people love feature X of your product. You could then test a headline that calls out feature X specifically.
Or your user tests show that people love the testimonials you have at the bottom of your homepage. You could then test moving those testimonials above the fold (or right next to the “Buy” button; this is a nice place for reassurance).
Chances are, you’ll have lots of ideas to test. But don’t pick one yet! Focus on building out your list of ideas first.
You can put your ideas into Notion, a Trello board, or Google sheet.
Prioritize your test ideas
Now that you’ve got a list of test ideas, it’s time to prioritize them.
Grade each of these on a scale of 1–5:
- How likely is this idea going to win? Does it directly address an issue you discovered in the research? (1 = low probability; 5 = high)
- How easy is it to implement? Changing text is easy; changing your business model is hard. (1 = hard; 5 = easy)
Now multiply those two numbers together and, voila, you’ve got a list of prioritized ideas.
Next, you’ll want to…
Set up your tests
Split testing software is a must.
The best tools are easy to use (especially for non-technical people) and offer great support.
Four that we’ve used and recommend are:
- Convert
- VWO
- AB Tasty
- HubSpot
In addition to the above, many third-party tools offer split testing. For example, OptinMonster and ConvertKit let you split test your opt-in forms without using a standalone tool like Convert or VWO.
How long should you run a test?
There are six factors which affect how long a test should run:
- Traffic
- Conversions
- Current conversion rate
- Number of variations in test
- Level of confidence in the results
- Difference in conversion rate between the original and variation
Let’s look at each.
- Traffic: More visitors make it faster to get reliable results.
- Conversions: The more people convert, the quicker you’ll get meaningful results.
- Current conversion rate: Higher-converting websites see faster outcomes, needing less test time.
- Number of variations: Testing more options takes longer as you gather enough data for each one.
- Confidence levels: Higher confidence requires bigger samples and more testing to avoid wrong conclusions. Finding a balance is important.
- Difference in conversion rates: If the expected change is big, you’ll reach reliable results faster.
To get a feel for how these factors affect your testing time, use an A/B test duration calculator (from VWO) to estimate how long your test will need to run.
At a minimum, run the test for at least one week (and preferably longer) to smooth out any daily fluctuations. (For example, an e-commerce site that sells office equipment may see a lower conversion rate during the weekend.)
The short answer: until your split testing software declares the test is statistically significant.
How to react to the results
Split tests end in one of three ways.
They either:
- Win
- Lose
- Tie (are inconclusive)
If a test wins:
- Promote the winner immediately. You can do this by setting 100% of the traffic to the winning variation. (Note: this approach works in the short-term. Eventually you’ll want to hardcode the winner to your site.)
- Repurpose the winning element elsewhere. Did your new homepage headline win? Use it on your landing page, ad copy, email subject lines—wherever it makes sense. Rest assured that you’re building your marketing materials on proven winners (instead of just guessing).
If a test loses:
- Celebrate. Seriously. Losing a split test often leads to big future wins. Why? Because you’ve found something that strikes a nerve with your audience. And once you’ve found that “nerve”, you can…
- Test the opposite approach. For example, let’s say you currently offer a 14-day free trial. You test a 7-day free trial and it loses. Since you’ve learned your audience is sensitive to the free trial period, your very next test should be a 30-day free trial.
If a test is inconclusive:
- Persist or pivot. Ask yourself: was this test inconclusive because of the idea, or the execution? If it’s the idea, pivot to another idea. If it’s the execution, try another variation.
For example, let’s say your headline test was inconclusive. The idea was good (since headlines are powerful levers for conversion), but the execution wasn’t (since the test was inconclusive). Try more headlines. - If you persist, try radically different variations. Most inconclusive tests are too timid; make your tests bolder: use stronger guarantees, longer trial periods, different testimonials (which appeal to different values).
Summary
Split testing is a powerful technique to grow your business.
You can get great results in just 5 simple steps:
- Gather feedback from surveys, heatmaps, and analytics.
- Brainstorm ideas to address what you’ve learned from your feedback.
- Prioritize these ideas based on their potential for success and ease of implementation.
- Set up your tests using reliable split testing tools.
- Double down on what wins.
By following these steps, you can optimize your website, enhance user experience, and improve your conversion rates.
Remember, though, that split testing is an ongoing process—so keep experimenting, learning, and optimizing.