Search engines crawling non-existent subdomains

Search engines crawling non-existent subdomains - Google Search Console is a free application that allows you to identify, troubleshoot, and resolve any issues that Google may encounter as it crawls and attempts to index your website in search results. If you’re not the most technical person in the world, some of the errors you’re likely to encounter there may leave you scratching your head. We wanted to make it a bit easier, so we put together this handy set of tips about domains, search-engines, google-search, subdomain to guide you along the way. Read the discuss below, we share some tips to fix the issue about Search engines crawling non-existent subdomains.Problem :

A few years ago, a previous developer added my company's website to a link farm he ran. My company has a wildcard subdomain, so any subdomains will return the user to the homepage. As such, the dev added multiple subdomains to the link farm (e.g., foo.example.com, bar.example.com, foobar.example.com) Before this developer left, he claims to have removed all our links from the link farm. However, to this day, we still get search engines crawling sub-domains that do not exists. I should add that the developer added many, many, subdomain links.

What can I do to prevent these subdomains from getting crawled?

Solution :

If the links still exist on other sites then Google will continue to look for them on your site, there isn't much you can do to stop it if you can't remove the links.

The only thing that you can do is add the code below to your htaccess file, this 410 gone message will let Google know the content is gone for good. Google may not stop looking but at least it will stop generating 404's each time it does look.

#Stuff to 410

Redirect gone /path/to/page.html
Redirect gone /directory-path/
Redirect gone foo.domain.com

If the issue about domains, search-engines, google-search, subdomain is resolved, there’s a good chance that your content will get indexed and you’ll start to show up in Google search results. This means a greater chance to drive organic search traffic to your site.