Definitive SEO Guide to Shopify's Web Pixel Manager Problem
In late 2022, Shopify officially released their new Pixel and Customer Events product. Unfortunately, Shopify failed to properly take into account the SEO implications of this change.
Soon after, SEO indexing problems began being noticed, and prominently made it to the Shopify forums on January 17, 2023 with user J_19 reporting details in a thread titled Shopify Bug - Web Pixels Manager Sandbox.
This article will detail the SEO problem, how Shopify's response created additional problems, and conclude with a definitive fix to the issue.
Table of Contents
- Overview of the Problem
- Understanding the Evolution of the Problem
- The Solution to the Issue
- Frequently Asked Questions
Overview of the Problem
The Shopify Pixel Manager implements tracking URLs that are injected into Shopify pages using an iframe.
Google follows and indexes these URLs which are fully formed HTML pages, complete with html, body and head tags, including a title:
As these pixel tracking URLs are unique to each page, Shopify began offering double the number of pages for Google to index – which Google did. The problem is, these are junk pages (thin content) that should not be indexed.
Since these URLs were also dynamic by pixel manager version, the number of indexable pages could grow exponentially over time; the larger the Shopify site, the greater the number of problem URLs.
As a result, Shopify stores have seen their number of indexed pages increase drastically.
Thin Content and SEO Implications
When search engines crawl your website, they aim to index high-quality, valuable content that provides a positive user experience.
Thin content refers to pages with little or no value to users. Thin content can hinder your website's overall SEO performance as websites with a significant amount of thin content can be deemed to be low quality/untrustworthy and have their search rankings lowered as a result.
All of the indexed Shopify Web Pixel Pages are considered thin content.
Clutter in Google Search Console (GSC)
Another issue with the Web Pixel URLs is that they create a lot of clutter in Google Search Console which makes it more difficult to identify problems.
Understanding the Evolution of the Problem
Shopify made several attempts to address this SEO issue, but some of these solutions failed to fully resolve the problem or introduced new problems, while creating confusion for shop owners who wanted to address the issue themselves.
The initial Web Pixel URLs were engineered by Shopify using the following format:
These were formatted with versioning as follows:
In the above code, 0.0.186 represents the version of Web Pixels Manager, and each version change would duplicate the number of web pixel URLs.
Shopify Attempt #1: Noindex/Nofollow
Shopify added a "noindex, nofollow" directive to these URLs.
This could have resolved the issue, however a few days later these pages began generating 404 errors as the /web-pixels-manager URLs were replaced with /wpm URLs.
At the time of this article, Google is showing roughly 500,000 instances of these 404 error pages still indexed in Google Search Results, including brands like Gymshark, The Brick, Staples, Red Bull and Heinz.
Shopify Attempt #2: Changed URLs + Noindex X-Robots-Tag + Robots.txt Block
Shopify changed the URL format from /web-pixels-manager to /wpm with the following URL convention:
They also changed the versioning number, and added an additional hash.
The URL change created an entirely new set of web pixel URLs.
The idea - presumably - was to let the old /web-pixels-manager URLs no longer work so they would 404 error and eventually be removed from the index, while fixing the problem for the new /wpm URL format.
Shopify choose to implement a noindex using an X-Robots-Tag, which is an HTTP header that can be used to send instructions to Google.
In this case, Shopify sent a noindex, nofollow directive, which tells Google not to index these pages, and not to follow any other links on these pages either.
This implementation makes sense, as it could be implemented fairly quickly and easily against all the /wpm URLs and wouldn't modify the pages source code.
Unfortunately, Shopify blocked these URLs in robots.txt as well.
When URLs are blocked in robots.txt, they are prevented from being crawled, which prevents the noindex directive from being read by Google.
This effectively trapped the URLs in the index.
This resulted in the pages being marked as Indexed, though blocked by robots.txt in Google Search Console and appearing in Google Search Results (SERPs) with No information is available for this page.
Per Google's documentation, "No information is available for this page" means:
"that the website prevented Google from creating a page description, but didn't actually hide the page from Google."
Shopify Attempt #3: Updating robots.txt
After continued reports of SEO issues, Shopify realized that the robots.txt block needed to be removed for Google to read the noindex directive and updated their default robots.txt to remove the /wpm and /web-pixels-manager blocks.
These changes would work to keep the /wpm and /web-pixels-manager pages out of the Google index, however the urls would continue to be identified by Google as pages and appear in Google Search Console under Excluded by 'noindex' tag
Additionally, previously indexed pages remain in the index and Google will need to recrawl any indexed pages before they will view the noindex directive and remove them.
However, since Google doesn't spend resources on indexing thin-content and 404 errors, it could take months or even up to a year or more, depending on the size of your store and the extent of your problem.
Instead of waiting on Google to reindex, we recommend the process outlined below.
Shopify Attempt #4: Blocking the Javascript that Creates the iframe
On May 15, 2023, Shopify rolled out a change that moved the javascript that creates the iframe to its own js file and blocked that file from indexing via robots.txt.
Previously, javascript included in common files created the iframe and Shopify attempted to block the created iframe.
With this change, Shopify added an intermediary javascript js file that is included from the common javascript and then blocked that file in robots.txt to prevent Google rendering of the iframe altogether:
This javascript then loads the analytics iframe to the page. The javascript file exists on the Shopify CDN and is blocked in the robots.txt of https://cdn.shopify.com like so:
Disallow: */blog-article-remove-faq-utms-*.js
Disallow: /wpm/*.js
This block prevents Google from rendering the javascript that generates the iframe, so the pixel urls of the iframe shouldn't be seen by Google. If however, Google came across the /wpm@ urls directly, they remain blocked with a noindex X-Robots tag.
This change should correctly fix the issue moving forward, but it will take some time for indexing to stabalize so we still recommend the process outlined below while we wait for Google Search Console to normalize.
The Solution to the Issue
Step 1: Unblock URLs in robots.txt
If you made previous changes to your robots.txt file to prevent /wpm@ and /web-pixels-manager@ from being indexed, these changes now need to be undone.
Make sure these URLs are NOT blocked in your robots.txt file.
Step 2: Use Google's Removal Tool
Request the temporary removal of your /wpm and /web-pixels-manager URL patterns in Google Search Console.
This will prevent these pages from appearing in search results for about six months, after which time Google should recrawl and recognize the noindex directive and keep them out of the index.
Typically, Google's best practices are to not use the Removal Tool and instead wait for the pages to drop naturally. Google however, has been very slow to drop these thin-content pages, and we do not suggest you wait, as your rankings could potentially be negatively impacted in the interim.
The Removal Process:
-
Log into Google Search Console and access Removals under the Indexing menu.
-
Click the NEW REQUEST button:
-
Enter https://example.com/wpm@ – replacing example.com with your website – and select Remove All URLs with this prefix:
Press NEXT.
-
Confirm the removal by clicking SUBMIT REQUEST:
-
Repeat steps 2-4 above for the URL https://example.com/web-pixels-manager@ – again replacing example.com with your website URL.
-
Your Submitted requests should now look similar to this:
While the removal could take up to a week, often it takes place within 24 hours and sometimes even faster. Keep an eye on the Removal Requests and the status should change from Processing request to Temporarily removed when complete.
Frequently Asked Questions
Is this a Shopify problem or a Google problem?
Google will index pages it comes across when it's crawling – it is everyone else's responsibility to control what Google does and doesn't index.
Why does the Google Removal Tool says the removal is only temporary?
Google makes the URL removal temporary because rather than masking the problem in their index, they want webmasters to properly address and fix issues.
While the removal tool will pull the web pixel URLs from the index right away, when the removal is over and the pages are reindexed, the pages won't reappear in search engine results due to the noindex directive.
In this sense, the pages are being both quickly removed and properly addressed.
Should I do redirects on the wpm and web-pixel-manager pages?
Absolutely not – you would create all sorts of problems by doing so.