What does SplitSignal measure? Sessions, click-through rate, or rankings?

• SplitSignal measures Organic Clicks from websites connected to Google Search Console. For paid pilot users, we can also pull CTR data manually from Google Search Console. 
• We are exploring other metrics in the future. Have a suggestion for some other metrics? Feel free to contact splitsignal@semrush.com.

Why are there minimums in order to run a statistical SEO split-test? What are those minimums?

In order to run a test, you need three things: 

In General:

• The website must have a strong ‘concept’ of product and category pages.
• The website must have over 500K monthly organic sessions.

Per Test:

• Organic Traffic — 100K organic sessions per month to the test group.
• Templatized Pages — Over 300+ templatized product/category pages per test group. Examples of templatized pages include product pages, category pages, blog/article pages, etc.

Why? 

• These thresholds significantly increase the probability of a statistically significant result. 
• The more traffic and pages a website has, the better the chances are for a statistically significant test. We may revisit these thresholds when more data becomes available.

​How are the pages identified for the Variant and Control groups?

SplitSignal uses cohort analysis to split pages in the test group without the use of a data-science team. We utilize 100+ days of historical traffic data to come up with two statistically similar groups of pages. Such a split increases the accuracy of the test and controls for external factors such as seasonality, Google Algorithm updates and more. 

How does SplitSignal know when the Variant group URLs are visited by Googlebot (Chrome Headless Browser)?

While we can’t be sure when Google will index the changes, SplitSignal provides an indicator of when each page was visited by Googlebot based on the URL information from Google Search Console. In our experience, Googlebot visits about 60% of the Variant pages in week 1, and the remaining 40% of Variant pages in week 2. It may take a few days before changes appear on the Google SERPs after the page is visited. This is all totally dependent on how much organic traffic your site receives monthly. 

The example below shows that all 460 pages in a variant group have been visited (100%): 

Variant Group Change Management: Client-Side vs. Server-Side

There are two types of statistical SEO split-testing. 

Client-side testing happens when the Javascript code makes the change to a Variant group after the page has loaded on a browser (“client”). The code is stored in a CDN and is triggered every time a visitor loads the page. SplitSignal uses client-side change management.

Server-side SEO testing works by making changes to the Variant pages directly on the web server. Instead of loading the original version and updating after a few milliseconds, server-side changes are hardwired. They trigger before the page is delivered to the client (i.e., the browser.) 

Does Flickering occur?

What is Flickering? This is when users see the Control version of the page in a web browser before it quickly changes to the Variant version.

We have not noticed any substantial flickering in the over 100+ tests run thus far. SplitSignal allows for previewing of changes made before launching a test so the UX/UI impact can be seen by the user before launch. Even though our main stakeholder for these tests is Googlebot, the user experience is still very important to us. 

What types of elements can SplitSignal test?

• Title tag tests
• Meta tag tests 
• Header test (example: Change the header from h1 to h2) 
• Move Elements (e.g. moving <h1> before <img> or moving content)
• Delete Elements (e.g. removing an <img> or piece of content)
• Add Elements (e.g. add new <p> to page).

We are exploring options for additional test types all the time.

Can we run multiple tests at once?

Users can run as many tests as your website allows, as long as each test meets the minimum thresholds for traffic & pages. It is important to note that once a page is selected for a test, it cannot be selected for another test while it is still running in the first test. You would have to wait until the first test is complete or stopped in order to use that page again; We do recommend waiting around 7 to 10 days to run on those same pages again. We call this the ‘cool-down period.’ 

How does SplitSignal affect Core Web Vitals?

We take Core Web Vitials very seriously. The benefit of client-side split-testing is that there is almost no impact on Core Web Vitals. We’ve run extensive performance tests with our script.

It’s important to consider things like the impact of third-party code and JavaScript execution time because this can affect site performance. We have yet to see any noticeable impact the SplitSignal JS snippet has had on any performance metrics. We have tested websites from sites of 300,000 monthly organic sessions to 50M monthly organic sessions. Before running a test, we encourage your dev/engineering teams to run a performance test in the form of a 2-day proof of concept on either a staging dev/test or live/prod site.

For more information, contact the SplitSignal team at splitsignal@semrush.com

Can we preview what a test would look like before it goes live?

Yes, SplitSignal preview functionality allows the user to see how the change will look to ensure that not only Googlebot sees the change, but the user does as well. Our main stakeholder is Googlebot, but we also highly value user experience and do not want that to be negatively affected.

What kind of information/data does SplitSignal have access to?

SplitSignal analyzes access logs of requests by SplitSignal’s JS snippet to improve SplitSignal’s quality and for use in certain tool features (i.e. tracking Googlebot visits). The lifetime for log data is 45 days from an initial request. After 45 days, the logs are removed, and SplitSignal no longer has any access to them.

• SplitSignal or Semrush doesn't share any data related to your SEO split-testing with 3rd parties or other Semrush tools.
• SplitSignal does not place any cookies onto the customer's browsers.
• SplitSignal uses GSC data only for splitting pages before an SEO split-test.

NOTE: Users can revoke access/connection to GSC at any time.

More information is available in the official privacy policy statement: Privacy Policy | Semrush

Can I try SplitSignal? 

We have a pilot option:

• Done For You — white-glove service where the SplitSignal team will draft, launch, and analyze your first ten tests (cost: $5,000/one-time)

Learn more about SplitSignal and apply for pilot/proof of concept here.