In this first example, we’re going to show you how code can differ based on the user-agent. This is a very slight change, but slight changes can be important.
If you head over to Papajohns.com, you’ll see slight differences between a default user-agent (your browser) and a Googlebot user-agent. This isn’t a major change, but you can see that a script will load on the default browser that will not show up for Googlebot. We’ve highlighted it in the images below. This is a fairly banal change, but you’d be surprised at what simple changes may do, especially if you fail to load the one script that injects all of your content.
Now that you’ve seen a smaller example of this in play, let’s look at a full code swap. We’ve headed over to shop.wegmans.com to see this in action. We’ve also saved the code in case they swap methods.
This time, let’s walk through the steps on how we would approach this. We recommend you follow along and perform the same steps so that it sets in. Remember, this should be familiar if you’ve read the first part of this article.
- Head over to a category on shop.wegmans.com
- Load the source code of the site. CTRL+U on Chrome/brave or right click and select view-source.
- Review the source code and copy/paste it into jsbin.com or emulate it in a new tab.
- Use a user-agent switcher extension (I used this) and swap to Googlebot (pull the user-agent from Google’s list of crawlers.
- Revisit the source code as Googlebot, and copy/paste it into jsbin.com or emulate it in a new tab.
Here’s a couple of gifs so that you can quickly see the difference between the two. We’ve also included the actual source code so you can review it yourself.
Browser User-Agent Source Code
Googlebot User-Agent Source Code
Now, when you emulate the Googlebot user agent source code, you’ll have a completely different story. While it doesn’t look pretty, you’ll be able to see content.
In this case, Wegman’s is using a dynamic rendering solution in order to make sure that Googlebot get’s all of the content it needs, whereas user’s load the regular version of the site. One thing to keep in mind is that Wegman’s is not using any form of verification when it comes to Googlebot, which means anyone who changes their user-agent can get this.
If a site is using verification, you’ll likely experience something like this when you try and spoof Googlebot.
We usually find this is the case with bigger websites, such as the Stubhub example here. This is also why you should always try and emulate code that’s pulled from GSC’s URL Inspection tool.