X-Cart forums

[BrandonLR]

I was just poking around at the source code of one of my product listings and I see that there is no meta information for robots with follow instructions.
Why is that?

[BrandonLR]

bump...no one answered this.

Nonetheless, I have another question along the same lines.

I downloaded my robots.txt file and noticed that it says
User-agent: *
# Sitemap example
# Sitemap: http://example.com/sitemap.xml

and not the actual URL to our sitemap xml file.
Why is this? Our site has been up for almost a year and who knows what kind of indexing we have been missing out on.

How do I solve this? Should I change the URL in the robots file to https://razorblade.pro/?target=sitemap or should I use https://razorblade.pro/sitemap.xml ?

[qualiteam]

It is not required to have the Sitemap XML link in your robots.txt

There are 3 methods of getting search engines aware of your sitemap XML files:
https://www.sitemaps.org/protocol.html#informing

The recommended way is submitting the link manually. Please check this article:
http://kb.x-cart.com/en/seo_and_promotion/submitting_your_shop_to_search_engines.html

Also, on the Sitemap XML configuration page there are two checkboxes that will make XC5 inform Google and Bing servers about new sitemaps when you generate them (this is the third method from the link to sitemaps.org site above).

You may also edit your robots.txt and add the link manually.

[BrandonLR]

Originally Posted by qualiteam

It is not required to have the Sitemap XML link in your robots.txt

Apparently it is because Google, to name just one, had no idea of my sitemap.

Originally Posted by qualiteam

The recommended way is submitting the link manually. Please check this article:
http://kb.x-cart.com/en/seo_and_promotion/submitting_your_shop_to_search_engines.html

Also, on the Sitemap XML configuration page there are two checkboxes that will make XC5 inform Google and Bing servers about new sitemaps when you generate them (this is the third method from the link to sitemaps.org site above).

Did this on multiple occassions over the last 11 months and yet, again, Google had no record of my sitemap and my robots.txt file still said "example.com"
I'm sure I could submit a bug report or something along those lines about it and then be asked to pay a $99 fee to find out why...

Originally Posted by qualiteam

You may also edit your robots.txt and add the link manually.

That's what I did and everything seems to be working fine now.

[qualiteam]

Originally Posted by BrandonLR

Did this on multiple occassions over the last 11 months and yet, again, Google had no record of my sitemap and my robots.txt file still said "example.com"

Did you submit your sitemap xml manually via Google Webmast Tools? I believe their site immediately informs whether the submitted sitemap is accepted, or not.

[GlasPak]

Originally Posted by qualiteam

Did you submit your sitemap xml manually via Google Webmast Tools? I believe their site immediately informs whether the submitted sitemap is accepted, or not.

Google renamed the service to "Search Console"... But yes it will say under Search Console how many URLs you submitted and how many URL's google indexed from that submission (doesn't include other pages they may have indexed via deep link crawling or other methods)..

For example here is what mine says

Sitemaps
655 URLs submitted
221 URLs indexed

Since I may have the same url in 2 different site maps, that is why the submitted and indexed are different.

[lostdog9]

I also checked code today and there is no meta robots tag on any pages?

Example: <meta name="robots" content="noindex" />

Why?

Can we add this and if so where?

[cflsystems]

The meta tag can easily be added with a custom module. Or you can modify one of the twig files that adds another meta tag and add your to it (you need to do this thought the template edit feature for the changes to stick). Or you can use this module https://market.x-cart.com/addons/custom-http-headers-by-cfl-systems.html to add x-robots-tag header tag - more info https://developers.google.com/search/docs/advanced/robots/robots_meta_tag


robots.txt file is optional and you need to add any entries in there manually. Keep in mind though most bots ignore this file especially the malware bots.