Before you start: what do you know about SEO split-testing? If you’re unfamiliar with the principles of statistical SEO split-testing and how SplitSignal works, we're suggesting you start here or request a demo of SplitSignal. 

First, we asked our Twitter followers to vote:

The test was POSITIVE. Our followers guessed it right (follow us on Twitter to vote next time).

The Case

Since we started split testing, one of the biggest eye-openers is the effect that user behavior can have on the outcome of a test. Google isn’t the only ‘user’ you optimize for; real users can make or break the success of optimization, especially when it comes to SERP testing, as this case will show.

For an international fishing charter company, we wanted to validate whether changing non-keyword-related words in the page title would make the page title more appealing, positively impacting CTR.

The Hypothesis

Our client hypothesized that by reformatting the page titles with “The 10 Best Fishing Charters in destination” of their destination fishing charter listings, we would see an increase in clicks (CTR).

The control and variant looked like this:

The Test

So, we used SplitSignal to set up and analyze the test. 2,279 fishing charter lister pages were selected as either variant or control. We kicked off the test and ran it for 30 days. We were able to determine that Googlebot visited 87% of the tested pages.

The Results

The above image shows the development/progress of the variant (orange line) compared to the modeled control group (blue line). We saw that the traffic to the variant pages performed better than predicted, which means the test is positive. 

Note that we are not comparing the actual control group pages to our variant pages but rather a forecast based on historical data. We compare this with the actual data. We use a set of control pages to give the model context for trends and external influences. If something else changes during our test (e.g., seasonality), the model will detect and take it into account. By filtering these external factors, we gain insight into what the impact of an SEO change really is.

The cumulative view shows additional or lost organic traffic to the tested pages and shows that the test performed is significant. When all three curves perform below (negative) or above (positive) the y=0 (cumulative gradient) axis, the test is statistically significant. That means we can be sure that the increase we are seeing is due to the change we made and not due to other (external) factors. 

After running the test for 30 days, we saw a 12% increase in organic clicks to the tested pages, with a confidence level of 99%. Not bad, right? 

Why?

SEO A/B testing is not only about validating your SEO changes for Google but also real users. Google will always be a factor, but it's not the only one that can tip the scales.

How Can SEO Split Testing Help With Google’s Page Title & Meta Description Rewrites?

Analysis of the data shows that this test impacted the variant pages’ click-through rate (CTR). Compared to our modeled control group, impressions increased slightly but not significantly. There were also no significant changes in terms of ranking. The increase in clicks seems to be purely due to the behavior of Google users. Within search, you always compete with other search results. If your answer is not what a user wants or expects, a user can simply click on a different search result.

Optimizing page titles is not just about including your main keyword(s). There is much more to writing appealing and effective page titles. This test shows that you need to think about and experiment with different factors that make up a page title as an SEO. Knowing what users expect and find appealing can make a big difference in how users interact with your search result snippets. Finding ways to stand out and be the most relevant answer is now more important than ever. If not, you might be leaving money on the table.

We know from experience that the results of a comparable test can differ between websites. Given the impact it can have, it’s a good idea to test it before rolling it out on your website.

Do you love reading about these test result? Join our email list and receive one email a month with the most exciting tests.

Have your next SEO split-test analyzed by OrangeValley Agency.