Before you start: if you’re unfamiliar with the principles of statistical SEO split-testing and how SplitSignal works, we’re suggesting you start here or request a demo of SplitSignal. 

First, we asked our Twitter followers to vote:

This time the majority of our followers were right, read further to find out why.

But first, let’s see what SEO professionals had to say about the result of this test:

Victor Valentine Romo, SEO Manager at Grey Matter:

Removing the FAQ Schema does give you the opportunity for Sitelinks to take its place. I can make a case for users who are not quite convinced to purchase may spend more time reading those answers, rather than a user who’d click on a category / product sitelink if the FAQ were not taking up SERP real estate.
If Google didn’t automatically assign Sitelinks I’d say this result is a negative.

Cedric Chambers, Demand Generation Manager at Research Square Company:

CTR increased because more users clicked through to the site rather than get distracted by the FAQs. For this query users weren’t even asking a question. But, organic impressions went down because the questions didn’t populate for the additional queries that the FAQs would have previously populated for.

Do you agree with these statements? Follow us on LinkedIn to share your opinion on the next test.

The Case Study

FAQPage structured data or “FAQ schema” is one of the most sought after rich results for many SEOs. Google released this feature back in 2019. It allows you to display questions and answers directly in Google’s search results:

With this SERP, real estate searchers will probably notice you better. The rich result nearly doubles the amount of vertical space your snippet takes up in search results, pushing competitors down. This can lead to a higher click-through rate (CTR). In addition, by providing users with additional information about the entity (for example, a product category), users can also make better judgments directly from the search results.

However, there can also be situations where this rich result can lead to a lower CTR. For example, if you fulfill a user’s query in the FAQ snippet, there may be no reason for a user to click through to the website because they already have the answer they were looking for.

So a major ecommerce player wanted to put this to the test by actually removing their FAQ schema they had already implemented. Because they never statistically measured the effect of this rich result on their SEO performance.

The Hypothesis

The website in question marked up all their FAQs on category pages with structured data for FAQs. 

They hypothesized that FAQ schema has a positive effect on organic traffic. So they wanted to validate what would happen if they completely removed the FAQpage structured data from the page. Which meant they lost the rich result for the tested pages

The Test

We used SplitSignal to set up and analyze the test. All category pages with FAQs were selected as either variant or control. We started the test and ran it for 30 days. We were able to determine that Googlebot visited 94% of the tested pages.

The Result

How to read SplitSignal test result

Removing the FAQ schema (and the nested objects and values) resulted in a negative effect of −5.3% on organic clicks to the tested pages.

Within a week, we were able to determine that the effect we saw was significant. When the blue shaded area performs below or above the x=0 axis, the test is statistically significant at the 95% level. This means that we can be confident that the effect we see is due to the change we made and not to other (external) factors.

Note that we are not comparing the actual control group pages to our tested pages. but rather a forecast based on historical data. The model predicts the counterfactual response that would have occurred had no intervention taken place. We compare this with the actual data. We use a set of control pages to give the model context for trends and external influences. If something else changes during our test (e.g., seasonality), the model will detect and take it into account. By filtering these external factors, we gain insight into the true impact of an SEO change.

Why?

While there are other important use cases for structured data, rich results are one of the biggest drivers for adopting structured data. This test shows that rich results can certainly have a big impact on SEO performance. 

As assumed, the CTR for the tested category pages decreased. As mentioned, the rich result provides a much more extensive search result. Within the SERPs, the tested website competed; they stood out (because competitors usually didn’t use FAQ schema), and they pushed competitors down, especially on mobile. By removing the FAQ schema, they lost that edge. Concerns that the snippet could negatively affect CTR by fulfilling a user’s search were dismissed.

With split testing, you want to make sure that the changes you make or, in this case, made will yield positive results. So as an SEO, you need to think about and experiment with different elements that make up your SERP snippet. Within search, you always compete with other search results. If your answer to a search doesn’t stand out or doesn’t match expectations, a user can simply click on a different search result.

Something that works for one website may not work for another. So if you feel inspired to implement and test FAQ schema, check out this step-by-step guide to test and analyze FAQ schema (yourself). The only way to know for sure is to test what works for you!