.Google's John Mueller addressed a concern about why Google.com indexes webpages that are actually disallowed from crawling through robots.txt as well as why the it is actually safe to ignore the similar Search Console files regarding those crawls.Robot Visitor Traffic To Query Criterion URLs.The person talking to the inquiry documented that bots were actually producing web links to non-existent query guideline Links (? q= xyz) to webpages along with noindex meta tags that are likewise blocked out in robots.txt. What urged the question is actually that Google is crawling the hyperlinks to those web pages, getting obstructed by robots.txt (without noticing a noindex robots meta tag) at that point obtaining turned up in Google Search Console as "Indexed, though shut out by robots.txt.".The individual asked the adhering to inquiry:." Yet here's the large concern: why would Google index webpages when they can't even see the information? What is actually the advantage in that?".Google.com's John Mueller verified that if they can not creep the page they can't find the noindex meta tag. He additionally creates an exciting reference of the site: search operator, advising to disregard the results due to the fact that the "average" consumers won't find those end results.He created:." Yes, you are actually right: if our experts can not crawl the webpage, our team can't find the noindex. That said, if our team can't creep the webpages, then there's certainly not a whole lot for our company to index. So while you could observe several of those pages along with a targeted internet site:- inquiry, the typical consumer will not view them, so I definitely would not fuss over it. Noindex is also alright (without robots.txt disallow), it simply means the URLs will find yourself being actually crawled (as well as end up in the Search Console document for crawled/not catalogued-- neither of these standings lead to concerns to the rest of the website). The essential part is actually that you do not make them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the constraints being used the Site: hunt accelerated hunt driver for analysis main reasons. Some of those factors is considering that it is actually certainly not linked to the normal search index, it's a separate point altogether.Google.com's John Mueller discussed the web site search operator in 2021:." The brief answer is that an internet site: concern is not suggested to become full, neither used for diagnostics reasons.A web site question is actually a particular kind of search that limits the end results to a certain internet site. It is actually primarily merely the word site, a digestive tract, and then the internet site's domain name.This inquiry restricts the end results to a particular website. It's not implied to be a detailed compilation of all the webpages from that internet site.".2. Noindex tag without making use of a robots.txt is actually fine for these kinds of situations where a bot is connecting to non-existent pages that are receiving discovered through Googlebot.3. URLs along with the noindex tag will produce a "crawled/not recorded" item in Search Console which those won't possess a negative effect on the remainder of the web site.Review the inquiry and respond to on LinkedIn:.Why will Google mark web pages when they can't even find the web content?Featured Graphic by Shutterstock/Krakenimages. com.