Before you start: if you’re unfamiliar with the principles of statistical SEO split-testing and how SplitSignal works, we’re suggesting you start here or request a demo of SplitSignal. 

First, we asked our Twitter followers to vote:

The majority of our followers couldn’t decide for sure, but the great news is, we have all the answers ready. Read the case study to find out the result.

The Case Study

Over the years, there have been many concerns and discussions about “boilerplate content” within the SEO community. SEOs have discussed how Google might treat different page elements differently based on surrounding HTML tags or their location on the page. While Google says it’s generally pretty good at recognizing boilerplate text and says it won’t penalize you for this type of content, Google’s general guidelines state that you should minimize boilerplate content.

Almost every website has some form of boilerplate content. Boilerplate content in terms of SEO is text or links that are reused across multiple web pages of a website without significantly altering the original content. Text and links in headers, footers, or sidebars are well-known examples.

As an SEO, you want to give search engines the right signals and help them understand which content and topics are important on the page. This is why we wanted to test the impact of heading tags in boilerplate content across multiple templates of an ecommerce website. 

The Hypothesis

To understand what a web page’s content is about, Google looks at different things, such as headings, to figure out what is actually being emphasized. Proper use of HTML heading tags can make content more meaningful and help search engines understand the page and its purpose a little better. 

This led to the hypothesis that unnecessary or unrelated headings in boilerplate content can make your content less meaningful and relevant, potentially negatively affecting SEO performance.

The website in question used <h2> HTML tags in the sidebar (on category pages) and footer (examples translated):

product pages additionally had boilerplate <h1>, <h2> HTML tags in main content elements (examples translated):

We decided to change all boilerplate heading tags to <div> tags. Note that this did not change the visual presentation of the content as it is styled with CSS class selectors.

Only heading tags that emphasize primary content, such as <h2>product description</h2>, were left unchanged.

Control

Variant

The Test

We used SplitSignal to set up the test. As mentioned, we chose to run two tests, each on a different page template: category page (PLP) and product page (PDP). Approximately 500 category pages and 1500 product pages were selected as variant or control via stratified sampling. We started the tests and ran them for 21 days. We were able to determine that Googlebot visited 99% of the tested pages.

The Results

After 21 days of testing, we reviewed the results of the two tests. We saw that for both tests, the traffic to the variant pages outperformed the modeled control group, meaning the tests were positive. Removing heading tags in boilerplate content resulted in an estimated 3.9% increase in organic traffic for the tested category pages and an 8% increase for the tested product pages.

When the blue shaded area in the cumulative view performs below or above the x=0 axis, the test is statistically significant at the 95% level. Looking at the cumulative representation of the two tests, we can conclude that the effect became significant very quickly. This means that we can be confident that the increase we are seeing is due to the change we have made and not to other (external) factors. 

Note that we are not comparing the actual control group pages to our tested pages. but rather a forecast based on historical data. The model predicts the counterfactual response that would have occurred had no intervention taken place. We compare this with the actual data. We use a set of control pages to give the model context for trends and external influences. If something else changes during our test (e.g., seasonality), the model will detect and take it into account. By filtering these external factors, we gain insight into the true impact of an SEO change.

Why

The test shows that Google certainly gives some weight to certain HTML elements (such as heading tags) in boilerplate content. As mentioned, headings can make content more meaningful and relevant to users and search engines. We can conclude that the opposite is also true; Unnecessary or unrelated headings in boilerplate content can make content less meaningful and relevant, resulting in sub-optimal organic performance.

Analysis of the data shows that the pages tested started to perform better in search results. Not only did we see an increase in clicks, but we also saw an increase in impressions, suggesting that the pages tested became more relevant to the search queries for which the pages were ranked.

Headings give value to the content within the tag element. So it makes sense that emphasizing your primary content and putting less emphasis on so-called peripheral matters can improve relevance. 

While Google says it’s generally pretty good at recognizing and evaluating boilerplate text, the results suggest it’s better to take control and ensure that the headings used on the page help to emphasize primary topics and page content.

The outcome of this test further piqued our interest, so we decided to roll out variants of this test across multiple websites. You can expect more on this topic in future case studies.

Keep in mind that something that works for one website may not work for another. The only way to know for sure is to test what works for you!