deadlock Message Board
[ July Full List | Reply ]

From Ari Manor at ZOOZ on 17 July '99
Hi there!

In the monthly visitors report of my site,,
most failures are related to requests for robots.txt
Does this mean I have to create some special robots.txt for spiders?
Which text should be written in this robots.txt file?

A quote from the failure report is below.

Thanks for your help,

Failure Report

Listing files, sorted by the number of failed requests.

#reqs: file
-----: ----
18: /robots.txt
1: /heb/intro.html
1: /eng/intro.html
1: /intro.htm
1: /

Reply to this message

Required Details
Your name:

Optional Details
Home page title:
Home page URL:
(if you'd like to receive
automatic replies by email)

Your Message