robots.txt & freefind.com

[http://dummy.us.eu.org/robert (Robert)]

Hi.

I do not have a robots.txt in my directories.

 > From: http://www.freefind.com/~robot
 > Date: Fri, 29 Mar 2002 22:46:54 -0800 (PST)
 > Subject: The site, http://www.csoft.net/~dummy/robert/, has not been spidered.
 >
 > Note: Do not reply to this email!
 > This message was automatically generated by your FreeFind
 > search engine. For help or to contact a human, see the 
 > information at the very end of this email. Thanks!

 > Pages read: 0
 > 
 > No pages on your site were indexed.
 > 
 >    Your site is protected by a robots.txt
 >    file which does not allow our FreeFind
 >    agent to access your site.
 >    
 >    fix: adjust your robots.txt file to allow
 >    access to FreeFind.  
 > 
 >    What is robots.txt?
 >    robots.txt is a file that advises robots (like our
 >    web spider which indexes your site) about which
 >    areas of a site (if any) the web site owner allows
 >    robotic access to.  As we respect the wishes of the
 >    site owner, we always honor robots.txt
 > 
 >    Didn't know you had a robots.txt file?
 >    You can find your robots.txt in the root of your
 >    domain. For example, if your site is located at:
 >    
 >    http://www.adomain.com/yoursite/index.html
 > 
 >    your robots.txt file is at
 >   
 >    http://www.adomain.com/robots.txt
 > 
 >    More info about robots.txt can be found at:
 >    See: http://www.robotstxt.org/wc/norobots.html
 > 
 >    Don't have control over your robots.txt?
 >    Contact your site administrator, ask them to add
 >    the following two lines to the robots.txt file:
 >       
 >    user-agent: FreeFind
 >    allow: /
 > 
 > Questions about what pages get indexed?
 > Check the help and faq in the FreeFind library at:
 > 
 > http://www.freefind.com/library
 > 
 > Sincerely,
 > Alan and Jim
 > FreeFind.com
 > 
 > ______________ FreeFind Help & Contact Information ______________
 > 
 > Help and FAQ:     http://www.freefind.com/library
 > Contact Info:     http://www.freefind.com/contact.html
 > 
 > Getting this email too often? 
 > Then re-schedule the indexing of your site at: 
 > http://www.freefind.com/control.html
 > Press the "build index" tab, then click "schedule re-indexing".