While it’s a staple of marketing departments around the globe, split testing is still relatively new to the SEO world. That means many of us are still wrapping our heads around using them effectively to get the best data. 

Below, we share insights from 10 experts to see how they run conclusive SEO A/B tests.

What Is SEO Split Testing?

If you haven’t tried it yet, a split test is an experiment where you make an update to one group of pages and compare the results to another unchanged group. You might have heard these referred to as A/B tests. 

In the SEO world, you’d typically use a split test to try a proposed site update on a smaller scale and verify the results before launching it across the entire site. If you’re new to split testing and want to learn more about how to set one up, check out our Split Testing 101 article. 

Meet the Experts

We interviewed 10 SEO experts about their experience running SEO split tests: 

• Jean-Christophe Chouinard, Senior SEO Specialist at SEEK
• Koen Leemans, Senior SEO Consultant and Product Owner SEO at OrangeValley agency 
• Oleg Zabugin, Head of SEO at Scyscanner
• Brandon Duchon, Staff SEO Strategist at Course Hero
• Christian Højbo Møller, Co-founder of Zoios People Analytics
• Althea Reilly, Search Engine Optimization Manager at Sally Beauty
• Nina McKinster, SEO & Local Marketing Specialist at Batteries Plus Bulbs
• Roy Hinkis, SEO & Growth Consultant at Robinhood
• Ayoub Chouchou, Search Engine Optimization Specialist at Upwork
• Roxana Stingu, Head of Search & SEO at Alamy

How Do You Know What to Split Test?

If you want to run an impactful split test, knowing what to test is half the battle. In A/B testing, your test subject is your hypothesis—the change you want to make and the result you expect that change to have. Choosing the right one, along with the right group of pages to test it on, could have a huge impact on your SEO strategy moving forward, regardless of the result.

But how do you know which hypothesis is “right” to test?

Koen Leemans recommends that you choose one that will make a large impact, while also being easy to test. For example, you could test whether changing your H3s to H2s increases your organic traffic. To figure out your hypothesis, Leemans says you need to know your business objectives, then analyze your site and traffic with that in mind. 

Once you have a hypothesis lined up, you also need to know which pages you’re running your test on. Depending on the size of your site, this may be time consuming. For example, Althea Reilly says that, because her organization has such a large number of product SKUs, she tackles this on a quarterly basis.

If you have a large quantity of pages on your site, it may be helpful to use a similar strategy. By analyzing, grouping, and testing pages several times a year, you can break your site down into more manageable chunks. Then you can focus on a different set each time. With a strategy like this, you can schedule tests across your site without the risk of neglecting any particular section.

Ultimately, as Jean-Christophe Chouinard says, you need to make sure you have enough time to properly investigate and plan your split tests if you want to succeed. He notes: 

“We have to adapt and spend a few weeks thinking of the best way to execute experiments in a way that will yield results not to waste [developers’] efforts. It can take weeks to properly plan an experiment.”

Need to spark more ideas for your split test? Find inspiration with 20+ SEO Experiments for Increasing Organic Traffic.

How Do You Split Your Pages into Test Groups?

Once you’ve established your hypothesis, the next step is to establish your variant and control groups. Your variant group is the set of pages you’re modifying to test your hypothesis. The control group is a similar set of pages that have not been modified. Later, you’ll measure the results by comparing the performance of each group.

For your test to have significance, your control and variant groups should be similar. That means the pages have the same purpose, are diversified in similar ways, and attract enough traffic to draw a reliable conclusion.

A note on our research: 70% of those we surveyed performed SEO experiments on domains earning between 1 and 5 million organic sessions per month. The remaining 30% earned more than 10 million organic sessions per month.

As such, if your site earns significantly less organic traffic on average each month, you may need to modify your testing strategy accordingly. 

When it comes to selecting pages for your variant and control groups, Christian Højbo Møller starts with analyzing the site for weak points and then uses his split tests to target those. This can take some time up front, but that effort can pay off later:

“The first time you run a big analysis to identify weak points that require experimentation takes maybe a couple of days for a big site, but then that data typically doesn’t change much over the course of 12 months.”

Leemans suggests choosing a template and going from there:

“Once you’ve decided what to test and for which templates, select pages that are indexable, return a 200 status code, and have sufficient historical data (minimum 100 days).”

Testing pages that use the same template and similar on-page elements can be advantageous, because it helps to avoid outliers. This could help you collect more consistent data.

However, as Chouinard points out, how you split your pages ultimately depends on: 

• Your site structure 
• How much traffic you earn 
• What you want to learn 

However you approach it, he recommends a few practical considerations: “You want to have pages that have a lot of volume, are crawled often, and are likely to stay indexed for the duration of the experiment.”

As for the actual process for selecting pages for your variant and control group, you have two basic options: select them manually or use a tool to help you.

Splitting your pages manually is possible, but it can have drawbacks. Nina McKinster spoke about her own experience manually splitting pages. Not only was this process “extremely tedious,” but it also reduced the number of pages her team was able to fully test. 

On the other hand, plenty of teams use a tool to make the process easier. For example, Oleg Zabugin says that his team relies on an automated system that uses performance data to split pages. Then, they use an A/A test to check the resulting groups.

Leemans agrees that the best approach for you depends on your site’s size and traffic. He suggests:

“In most cases, stratified sampling of Google Analytics or Search Console data should suffice. You can then plot the groups against each other to visually check whether the control group is representative.”

Whatever method you use to create your variant and control groups, make sure you check them before moving forward with your test. This way, you can make sure that your test groups are consistent, diversified, have enough data, and align well with your goals.

What Tools or Resources Do You Use to Create Your Test Groups?

You have a variety of tools at your disposal to help you split your pages more efficiently. When we surveyed the experts, seven out of 10 said that they prefer to use Google Search Console. Four out of 10 said that they used Google Analytics or their own tool. Two respondents also used Semrush and Google Sheets.

How To Split Pages Using SplitSignal

When you run your test in SplitSignal, it can help you create your test groups seamlessly. Here’s how it works. 

First, select the section of your website that you want to test.

Then, use the filters to narrow down your options.

Finally, double check your completed list of URLs to make sure it’s correct. 

It’s that simple! Want to try it for yourself? Find out more and request access to SplitSignal here. 

How Do You Make Sure Other Teams Prioritize Your Split Test?

When you’re running a split test, you’ll most likely need to collaborate with other teams. Unfortunately, the reality is that those teams have other things to focus on, and your test might not be their top priority. 

So how do you move your SEO split test to the top of the heap?

As Leemans notes, you need to help people understand why you want to perform a test in the first place. He says: 

“Search engines are a major source of traffic for almost all businesses and organizations. With SEO A/B testing, you can be confident that all changes that are implemented (and time spent) will also yield positive results.”

To help teams understand the importance of your test, you’ll typically need to demonstrate two things: 

• That the amount of work or resources you’re asking for is justified 
• How they (or the organization as a whole) are likely to benefit from a successful test

This is especially true when it comes to the teams that will actually make the updates to your variant pages, i.e., your development team. When we surveyed the experts, only 20% said they had dedicated development or engineering resources at their disposal.

If you’re sharing development resources, you’ll need to ensure your split test is a priority if you want to get it done in a timely manner. 

Roy Hinkis notes that, in his experience, engineers are “incentivized by seeing impact and clear measurements,” making split test results uniquely suitable for convincing them.

Oleg Zabugin agrees: “Engineering resources are precious. Proving that the change is going to move the needle is almost the only tactic that works.”

At first glance, this presents a bit of a challenge. After all, the whole point of a split test is to gather evidence and prove that your changes will make an impact. How can you use that same information—which you haven’t tested for yet—to convince a team to help you run the experiment in the first place?

Ayoub Chouchou suggests using data you already have. Results from previous tests, along with specific areas of improvement, can be especially helpful.

But data isn’t the only important factor. As Brandon Duchon notes, you should also make sure your test is relevant. He says, “our strategy is to ensure projects solve meaningful problems or provide useful insights. Our experiments should help us make progress towards specific business goals or objectives, such as growing active users or reducing time to first crawl.”

This can work for more than just development teams. Tying your work to company goals or KPIs, such as ROI, can help convince influential stakeholders to agree to run your test. 

According to Roxana Stingu, “being able to correlate the test’s success with potential revenue increase can help prioritize SEO testing over other business projects.” She also recommends providing worst- and best-case predictions to help stakeholders get a more holistic understanding. 

Reilly agrees. By tying her proposed changes to a monetary value, she says she can convince teams to prioritize her work. However, she notes that this might be challenging for some at first: 

“We SEOs tend to under promise (in hopes of overdelivering) so that we level-set expectations. However, I think it’s important to go out on a limb sometimes and give a sizeable projection to what this split testing can do for the bottom line, in order to get your changes prioritized.”

How Do You Track and Measure Your Test Results?

All the hours your team spends designing and implementing your split test are wasted if you don’t track and analyze your results effectively. 

Half of our surveyed experts said that they prefer to track their control and variant pages by setting up a dashboard in Google Data Studio. Often, this dashboard would visualize data pulled from Google Search Console, Google Analytics, or Google Tag Manager. 

By contrast, three respondents said that they rely on third-party tools to get the job done. One said that they primarily use their own internal tools to track their progress.

As for measuring the results, here’s how it broke down:

• 7 out of 10 said they use pre- and post-analysis
• 6 out of 10 said they use a statistical model (like CausalImpact) in addition to pre- and post-analysis
• 2 out of 10 said they only use statistical models

Using SplitSignal To Analyze Your SEO Split Test

If you use SplitSignal to run your SEO experiments, you can track and analyze your results without leaving the tool. SplitSignal provides a snapshot overview of your test results, telling you whether the results were positive or negative, and how statistically significant those results were.

The tool will also visualize your results in more detail. For example, at a glance, you can see not only your test group’s performance for the duration of the experiment, but also how accurate your predictions were.

You can learn more about how to create, run, and analyze tests in SplitSignal on our Knowledge Base.

Start Split Testing Your SEO

The SEO game grows more competitive with each Google algorithm update. SEO split testing can help you make data-driven decisions that drive your site to the top of the SERPs. The right tools can help you set them up more efficiently and draw more confident conclusions.

When it comes to setting up the perfect SEO experiment, SplitSignal was designed to do the heavy lifting. Try it for yourself by requesting a demo today.