Before you start: what do you know about SEO split-testing? If you’re unfamiliar with the principles of statistical SEO split-testing and how SplitSignal works, we suggest you start here or request a demo. 

First, we asked our Twitter followers to vote:

Here’s what other SEO professionals have to share about this test:

Vladimir Gertner, Senior Project Manager at Soft Road Apps:

I can’t see how this would help the ranking, or the clickthrough rate. So i would have to go with a negative impact. Or, at best, a neutral one.

Follow us on LinkedIn to share your thoughts on the next test.

Introduction

Content on an ecommerce product site is a challenging topic. 

It is difficult to produce content that differentiates you from your competition. 

It’s even more difficult to quantify if it is valuable.

This SplitSignal test looked at the value of producing quality content for an ecommerce site.

The site in question here had a detailed expanded content section on some of their product category pages.

The challenge is that it takes a lot of time and effort to produce high-quality content.

So, this test looked to see if this content section was valuable and whether or not they should commit more resources to build it out across all of their product categories.

The Hypothesis

The test was to validate if the additional content section was valuable or not.

The reasons why it was considered valuable and introduced are discussed below.

NOTE: This content section is well below the fold and occurs after the full product listings. It requires a lot of scrolling to get to.

Helpful Content

This test is quite timely as it seeks to address if additional helpful content is useful. Note that this test occurred just prior to the helpful content update. However, this embodies many SEO best practice elements for ecommerce sites.

Above all else, the content section that was added in some sections of this site is helpful for users. It serves as a product guide and gives useful information that users might want when selecting the product in question.
 

Relevance

Relevance is at the core of search engine optimization, so it stands to reason that adding additional content that is relevant to the product ought to be net beneficial.

Adding this helpful content will help extend the potential organic searches that these product pages will rank for.

One of the bigger challenges with ecommerce content is a variation from the product name and very brief descriptions that usually accompanies an ecommerce product listing.
 

Schema Mark Up

The additional content section had detailed FAQs, which were also accompanied by the appropriate schema markup.

Providing the schema FAQ markup with the additional content section is useful and should be a net positive.

This is an SEO best practice and should be used where ever possible.

The Test

The test looked at removing this additional content section. 

It was well below the fold, and after all of the different products offered, significant scrolling was required to get to it.

The test conducted used 116 Control URLs and 116 variant URLs, testing in a total of 240 pages. The 8 excluded URLs were outliers.

The Results

This test resulted in a “No significant change” classification. However, there is usually more to unpack than the top-line result, so let’s dive in.

Below you can see the traffic trend over time prior to and during the test.

Although this test is classified as having no significant change, it can be seen that the trend is a slightly negative effect. 

There is s 91% confidence level, and the outer bound does cross the Y=0 line, which is the primary reason this test is considered as having no significant change.

If you do linear regression through the mean values, you would show a negative trend and also note that in some areas, the test was qualifying as a negative result that was significant.

An interesting observation to make is that there was a trend (as shown in the chart below) in the traffic before the test starting, which continued through the test. It is important to make such observations so that any confounding factors can be considered.

It should also be noted that this type of testing should not be susceptible to any trends or changes during the test as the effect on the control and variants should be the same.

When such trends and events occur, it is important to look at the pointwise data to see if the variants or control groups deviated at all. The cumulative clicks data gives you the overall result and the big picture, but the pointwise data gives you the detail.

Why

So, there was a hypothesis that this type of helpful content section was valuable.

The result was not a clear signal that it was, but it does elude to a negative trend by removing this helpful content section. 

Therefore the theory that the helpful content section is beneficial is valid, even though this specific test did not give full validation of that.

So, let’s explore some of the reasons why this test didn’t show a stronger negative effect and what it might mean.
 

Proximity

Proximity and above-the-fold content has long been considered important. It makes sense you would mention something important early and at the beginning of your content. 

This may mean that although the content was relevant and helpful, with it being pushed right to the bottom of the page, it may lose some of its value.

Now, it may also be that if you had it at the top, it would have better organic search performance but would likely result in fewer sales and revenue from the products since they would be pushed below the fold.

This SEJ article on above-the-fold content discusses many important aspects of proximity and highlights how, since the hummingbird update, Google is able to better understand the words on the page and needs to rely less on keywords in headings and proximity.

FAQ Schema will have been retained even with the element being removed

The helpful content section was indeed helpful, and the addition of FAQs and associated schema make it even more so. 

However, deleting the content section and leaving the FAQ schema markup in place may have had some negative consequences.

One thing that search engines really don’t like is saying you have something that you don’t show on the page.

This may be a confounding factor, and it is hard to quantify exactly how much effect this would have.

It would be possible to remove this section as well in a similar follow-up test.

Consider this within the context of Google’s Helpful Content Update

This test was very timely, or perhaps just a little bit before its time.

Google’s helpful content update was indeed intended to reward sites that have helpful content. 

If this test was done post helpful content update, it might well show a different effect, and this would be a logical follow-on.