Hello,
OK...still have a few more questions.
I did place a robots.txt file in the root directory.
# All robots will spider the domain
User-agent:*
Disallow:
User-agent: *
Disallow: /lab/
User-agent: GoogleBot
Disallow: /*.php$
Is this correct?
Also, I generated the catalog using the default /catalog. If I change it to the root directory (/), then do I need to mark the box that says: "Drop the old catalog before generating the new one". ??
|