sitemap.xml for huge online store

sitemap.xml for huge online store - Google Search Console is a free application that allows you to identify, troubleshoot, and resolve any issues that Google may encounter as it crawls and attempts to index your website in search results. If you’re not the most technical person in the world, some of the errors you’re likely to encounter there may leave you scratching your head. We wanted to make it a bit easier, so we put together this handy set of tips about google-search, search-engine-indexing, google-index, xml-sitemap to guide you along the way. Read the discuss below, we share some tips to fix the issue about sitemap.xml for huge online store.Problem :

I'm running online store with 30,000+ products (having unique urls) which are updated every day. I want google to index my products faster. Now crawler index ~500 pages/day (reports from google webmaster tools). Would generating sitemap xml improve indexing? Should I include in sitemap static pages like contact, shipping info etc (I have about 10 such pages)? Does google bot works independently and index pages that are not listed in sitemap?

Solution :

Would generating sitemap xml improve indexing?

In theory, but there are no promises, either. ("Google doesn't guarantee that we'll crawl or index all of your URLs." etc.) Sitemaps can be helpful for bots to find all your pages, but they're informational; they aren't required to do anything with them. The search engines may decide to disregard some documents for whatever reasons of their own.

Should I include in sitemap static pages

There's no real reason to leave any URLs out.

Does google bot works independently and index pages that are not listed in sitemap?

Yes, all the spiders do. Sitemaps are only there to help them do their jobs, especially for documents that may not be linked yet, or not very well.

It sounds like your type of site was exactly what the Sitemap protocol was created for. I'd say that a sitemap file could do nothing but improve your site's indexing.

Using the <lastmod> and <changefreq> tags should allow search engine crawlers to better determine how often they re-crawl each of your product pages. Worst case scenario would be that these aren't taken into account, and you're back to square one - but you're also no worse for wear either.

Adding your static content is also not a bad idea, and you can use the <changefreq> value to let search engines know that they don't have to check that page every day - so maybe they'll spend more time indexing your product pages.

It's also worth looking into tools like Google Webmaster Tools - Google in particular have a section where you can tweak your crawl settings to see if that helps.

If the issue about google-search, search-engine-indexing, google-index, xml-sitemap is resolved, there’s a good chance that your content will get indexed and you’ll start to show up in Google search results. This means a greater chance to drive organic search traffic to your site.