Skip to Content
Former Member
Nov 15, 2006 at 01:10 PM

Web Repository Manager and robots.txt



I would like to search an intranet site and therefore set up a crawler according to the guide "How to set up a Web Repository and Crawl It for Indexing".

Everything works fine.

Now this web site uses a robots.txt as follows:

<i>User-agent: googlebot

Disallow: /folder_a/folder_b/

User-agent: *

Disallow: /</i>

So obviously, only google is allowed to crawl (parts of) that web site.

My question: If I'd like to add the TRex crawler to the robots.txt what's the name of the "User-agent" I have to specify here?

Maybe the name I defined in the SystemConfiguration > ... > Global Services > Crawler Parameters > Index Management Crawler?

Thanks in advance,