Robots Exclusion Checker

The Google Chrome extension that alerts you to Search Engine indexation or crawl restrictions

20,000+ Installs

Thousands of digital professionals have installed Robots Exclusion Checker

3 languages

The Chrome extension is currently available in English, French and German.

5-star rating

REC has a 5 star rating on the Google Chrome store

Why you should use this extension

I created Robots Exclusion Checker to help digital professionals save time in understanding whether a particular page on a website was accessible to Search Engines. It’s the first extension to combine all elements that affect indexation and crawling together, in one simple to use and visibly clear format.

It’s free to download in Google Chrome and is a must-have for digital professionals, particularly if you work within eCommerce or on large, complex sites.

DOWNLOAD NOW

Robots Exclusion Checker Chrome Extension 1

Which SEO elements does the extension check?

Robots.txt
If a URL you are visiting is being affected by an “Allow” or “Disallow” within robots.txt, the extension will show you the specific rule within the extension, making it easy to copy or visit the live robots.txt. You will also be shown the full robots.txt with the specific rule highlighted (if applicable). Cool eh!
Meta Robots tag
Any Robots Meta tags that direct robots to “index”, “noindex”, “follow” or “nofollow” will flag the appropriate Red, Amber or Green icons. Directives that won’t affect Search Engine indexation, such as “nosnippet” or “noodp” will be shown but won’t be factored into the alerts. The extension makes it easy to view all directives, along with showing you any HTML meta robots tags in full that appear in the source code.
X-robots-tag
Spotting any robots directives in the HTTP header has been a bit of a pain in the past but no longer with this extension. Any specific exclusions will be made very visible, as well as the full HTTP Header – with the specific exclusions highlighted too!
Canonical tags
Although the canonical tag doesn’t directly impact indexation, it can still impact how your URLs behave within SERPS (Search Engine Results Pages). If the page you are viewing is Allowed to bots but a Canonical mismatch has been detected (the current URL is different to the Canonical URL) then the extension will flag an Amber icon. Canonical information is collected on every page from within the HTML and HTTP header response.
Nofollow tags *NEW*
A new addition to the extension gives you the option to highlight any visible links that use a “nofollow”, “ugc” or “sponsored” rel attribute value. You can control which links are highlighted and set your preferred colour for each. I’d you’d prefer this is disabled, you can switch off entirely.

Robots Exclusion Checker Chrome Extension 2

What the professionals say

“Super useful and better than my previous extension ‘see robots’…more extensive when checking if a page is actually no-indexed”

Siobhan SwainstonHead of SEO - performance.io

“Amazing extension! Saves me so much time and I love the new canonical feature. Highly recommended for SEO’s and devs to use!”

Janaya WilkinsOwner - sloactive.com

“I have needed something handy like this for a long time. It’s a real time saver! Highly recommended to all SEO’s out there”

Michael Carden-EdwardsHead of SEO - Ballou

“Top-notch plugin…best robots plugin on chrome store”

Christopher PitcherSenior SEO & UX Developer

“I never write reviews…but I use this bad boy every day and it’s excellent.”

Adam DurrantWeb marketer

“A must-have Chrome extension for your SEO toolkit”

Anthony LavallVP of Strategic Partnerships - Croud

Frequently asked questions

How does the extension handle multiple meta robot tags?

Multiple meta robots tags on a single page is a common issue with many platforms, such as WordPress or Magento. If you have more than one plugin installed offering robots control it can often result in multiple tags being inserted into the page, sometimes with conflicting directives. If the extension detects multiple tags, it will consolidate all values and show them to you within the extension.

But what if there are conflicting values? The extension simulates the way Google treats multiple values and uses the most restrictive value.

For instance, if the extension detects these tags:

<META NAME="ROBOTS" CONTENT="NOINDEX">
<META NAME="ROBOTS" CONTENT="INDEX">

We will flag the NOINDEX value.

Source: https://webmasters.googleblog.com/2007/03/using-robots-meta-tag.html.

Will a meta robots ‘noindex’ prevent Google from crawling my site?

Using a meta NOINDEX tag at a URL level will prevent Search Engines from indexing that URL but in order to detect the tag, which is located within the HTML, the page needs to be crawled.

If you are linking to a large number of pages from within your website that have a NOINDEX tag, they will all be crawled on a regular basis. A NOINDEX tag therefore does not impact Search Engine crawling but instead the indexation of pages.

Why is a robots.txt exclusion flagged instead of a URL that has a meta NOINDEX tag?

In order for Search Engines to detect a meta NOINDEX tag, the URL must be crawled and HTML processed. When there is an exclusion in the robots.txt file, this prevents Search Engines from crawling and processing the URL, meaning they don’t get a chance to discover the NOINDEX tag.

Does blocking a URL within robots.txt prevent Google from indexing it?

The robots.txt file will prevent Google from crawling a specific page or set of pages. However, Google will still be aware that the URL exists – especially if you link to that URL internally, include the URL within your sitemap.xml or that URL is linked to from an external website.

In this scenario, a robots.txt will not prevent Google from indexing the URL itself. This could lead to the URL being returned in the Search Engine Results Pages (SERPs) with a warning that robotst.txt prevented them from crawling the page.