In the monthly visitors report of my site, http://www.zooz.co.il,
most failures are related to requests for robots.txt
Does this mean I have to create some special robots.txt for spiders?
Which text should be written in this robots.txt file?
A quote from the failure report is below.
Thanks for your help,
Listing files, sorted by the number of failed requests.