View Single Post
  #3  
Old 06-22-2007, 02:57 PM
  Jon's Avatar 
Jon Jon is offline
 

X-Guru
  
Join Date: Oct 2002
Location: Vancouver, Canada
Posts: 4,200
 

Default Re: gsite crawler taking forever!!!

Some crawlers don't respect the base href tag so having CDSEO installed can cause problems if they don't.

If you find that the urls they are coming up with are "looping", it means that the base href tag is not being followed.

You can solve this problem with the following code in .htaccess:

Code:
RewriteEngine on RewriteCond %{REQUEST_URI} ^/[^\.]+[^/]$ RewriteRule ^(.*)$ http://%{HTTP_HOST}/$1/ [R=301,L]
Reply With Quote