X-Cart forums

[keystone]

I started gsite crawler this morning around 9:30am I imported my robots.txt to make sure it wasn't going to crawl my entire site. I ended up with around 6400 files waiting and I'm now down to 4500 but it is 4:30pm!!!! Is it supposed to take that long???? Anybody know what settings I should have adjusted or any other tips?
thanks

[nevets1219]

We have something like

*?=productcode&=1
/*.html?js=*
/*?=productcode&=1
/?
/?*
/?=productcode&=1
/?sort=*
/admin/
/antibot_image.php
/cart.php
/catalog/
/error_message.php
/files/
/giftcert.php
/giftreg_manage.php
/giftregs.php
/help.php
/home.php?cat=*
/icon.php
/image.php
/include/
/modules/
/offers.php
/orders.php
/payment/
/product.php
/product.php*
/product_image.php
/register.php
/search.php
/shop_closed.html
/sql/
/upgrade/
/var/
?*

Some are probably duplicates and some are not necessary but we figure this works so far This includes more than what is in our robots.txt.

It takes about less than 5 minutes (2-3 minutes) for it to go through and process (generated 300ish links). Though you may want to limit it on a shared hosting since it can be pretty intense on your website. Also when your host locks your site for CPU overuse, GSiteCrawler doesn't realize that and continues to crawl.

[Jon]

Some crawlers don't respect the base href tag so having CDSEO installed can cause problems if they don't.

If you find that the urls they are coming up with are "looping", it means that the base href tag is not being followed.

You can solve this problem with the following code in .htaccess:

RewriteEngine on 
RewriteCond %{REQUEST_URI} ^/[^\.]+[^/]$
RewriteRule ^(.*)$ http://%{HTTP_HOST}/$1/ [R=301,L]

[nfc5382]

take a look at http://www.xml-sitemaps.com/standalone-google-sitemap-generator.html . it can be automated on your server. works fine with CDSEO

[keystone]

Is this code in my .htaccess file the same as what you suggested? If so I already have it there.

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_HOST} !^www\.uscandleco\.com [NC]
RewriteRule ^(.*)$ http://www.uscandleco.com/$1 [R=301,L]
RewriteCond %{SERVER_PORT} 80
#RewriteRule ^admin/ https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
</IfModule>

I ran gsitecrawler again and it ran overnight and when I checked it there were150000 files waiting and it still was going. I shut it down. I'm assuming it was just looping again. Any help would be great.

[Jon]

The code isn't the same, please try the code I provided.