Google’s John Mueller recently answered a question about A/B testing web pages for long durations, warning that an unintended consequence is that enabling variations to be indexed can result in uncertainty as to which will be visible in the search results.
A/B Testing Traffic From Live Search Results
A/B testing is when one or more versions of a web page is shown to users. The reason for doing this is generally for testing conversion rates and user responses.
The important takeaway from the guidelines is that A/B testing live web pages is the guidelines were created to minimize impact on search performance.
The guideline begins:
“This page covers how to ensure that testing variations in page content or page URLs has minimal impact on your Google Search performance.”
While Google does not explicitly forbid using A/B testing to test which page ranks better, the context of the guidelines itself is defined as protecting search performance; measuring search performance is not in the guidelines.
What Google’s document describes getting measured is consistently user behavior, not rankings.
On a side note, something that’s not in the guidelines is that there is no “right” button color and size for improving clicks on a call to action button. Longstanding SEO knowledge and experience about this is that large buttons and/or colors that contrast strongly against the web page backgrounds tend to get more clicks. This likely explains why Amazon’s Add To Cart button is a bright mustard color and Walmart’s version is bright blue contrasted against a solid white background.
Google’s Guidelines On A/B Testing
Google’s guidelines on A/B testing describe it as showing different versions of a website and collecting data on how users react to them. In terms of SEO performance it says not to expect any disruption but by allowing Google to index the slightly different pages once the testing is over the winning combination will be indexed much sooner.
There are two kinds of A/B testing:
- A/B Testing
Testing two or more changes to a web page. Google uses the example of testing different fonts on buttons. - Multivariate Testing
This is a test of multiple changes all at once in order to identify which combination of factors work best together. Google uses the example of testing different combinations of different fonts on buttons and on the web page itself.
Four Considerations For A/B Testing
Google also recommends four best practices:
1. Use The rel=”canonical” Link Attribute
This is probably the most important factor to consider. Using the rel=canonical link attribute enables site owners to put all kinds of variations of a web page online and still include a strong hint about which version of a web page is best.
2. Use 302 redirects
If you’re randomly redirecting users to different versions of a web page you should be using a 302 redirect, not 301 redirects. 302 means that a resource (like a web page) has been temporarily moved. That’s different from a 301 redirect which means that a move or change in URL is permanent.
3. Don’t Cloak
Cloaking is the practice of showing one thing to Google and something else to users. If you’re testing different web pages to see how users react when they click through from search then Google insists that site owners show the same thing to Google, even if the page elements are constantly changing.
4. Don’t A/B Test For A Long Time
Google warns site owners to limit how long A/B testing goes on. They warn that excessive testing could get a site in trouble:
“If we discover a site running an experiment for an unnecessarily long time, we may interpret this as an attempt to deceive search engines and take action accordingly. This is especially true if you’re serving one content variant to a large percentage of your users.”
That last warning relates directly to the question asked on the Bluesky social network.
Google Answers Question About Long-term A/B Testing
The person asking the question specifically wanted to know about how Google handles A/B testing that lasts for as long as a year.
They asked:
“Hey @johnmu.com, As Google’s A/B testing guide suggests to avoid running same A/B test for long durations, I was wondering how does Google handle long term holdouts (eg. 10% for 6-12 months), especially for a large scale marketplace with 10s of millions of crawls to similar amount of pages.”
Google’s John Mueller answered:
“Depending on your setup, what might happen is that one or the other version is used for indexing. If they’re close enough, probably that doesn’t matter. If they’re significantly different, that could be visible in search results too.”
The person who asked the original question then followed up with an additional question that revealed more about how much the web pages are changing.
They asked:
“…what if it’s fully different like a redesigned page, and since Googlebot is getting alternative versions with each crawl (sometimes in a day). Can that rapid change in core HTML structure cause issues with indexing and lead to Google potentially dropping the pages from index?”
Mueller responded:
“We’d take the content into account the way that we crawl it for indexing. There’s no (as far as I know) “penalty” or “demotion” for having varying content (lots of sites have that), but it can make it harder for you to debug & monitor if the content constantly changes.”
The person asking the question wanted to know how Google handled long-term A/B testing. They did not ask how Google handles indexing, but that’s the question Mueller answered. That may explain why the person followed up with a second question that was more precise about the extent of their A/B testing and Mueller again focused on indexing.
No Penalty For Having Varying Content?
Mueller’s statement seems to contradict Google’s own guidance about long-term A/B experiments.
The relevant context of Google’s guidelines is:
- It confirms that A/B testing is legitimate.
- Normal experiments are reasonably assumed to be temporary.
- Once enough data is collected to reach conclusions the A/B test it’s normal that it ends.
That’s where we get to the warning part of the guidance:
“If we discover a site running an experiment for an unnecessarily long time, we may interpret this as an attempt to deceive search engines and take action accordingly. This is especially true if you’re serving one content variant to a large percentage of your users.”
So the point of where things get fishy is when the experiment goes on longer than what seems reasonable and where one variation of the content becomes the prime version for most users as part of an attempt to “deceive search engines.”
Featured Image by Shutterstock/logofank
