Newcastle Music Directory

NMbot

The Newcastle Music Directory Web Robot

NMbot is the Newcastle Music Directory's web-crawling robot. It regularly checks each link in the directory to determine if it still exists. Information collected by NMbot helps build a searchable index for the Newcastle Music Directory. On this page, you'll find answers some commonly asked questions about NMbot.



Why is NMbot asking for a file called robots.txt which isn't on my server?

robots.txt is a standard document that can tell NMbot not to download some or all information from your web server. For information on how to create a robots.txt file, see The Robot Exclusion Standard. If you just want to prevent the "file not found" error messages in your webserver log, create an empty file named robots.txt.


How often will NMbot try to access my webpages?

For most sites, NMbot should not access your site more than once every few seconds on average. Since network delays are involved it is possible over short periods the rate will appear to be slightly higher. If you find that we are placing too high a load on your site, please let us know by sending us e-mail at contact@newcastlemusic.net.


How do I request the NMbot to not crawl parts or all of my site?

There is a standard for robot exclusion documented at robotstxt.org. You can put a file on your server called robots.txt that can exclude NMbot or other "web crawlers." NMbot has a user-agent of "NMbot".


Where can I find out more information about robots.txt?

For more info, see the Robots FAQ.


Why isn't NMbot obeying my robots.txt file?

In order to save bandwidth NMbot only downloads the robots.txt file once a week. So, it may take a while for us to become aware of any changes that might have been made to your robots.txt file. Also, check that your syntax is correct against the standard at http://www.robotstxt.org/wc/exclusion.html#robotstxt. A common source of problems is that the robots.txt file must be placed in the top directory of the server (e.g., www.example.com/robots.txt); placing the file in any subdirectory will not have any effect.

If there still seems to be a problem, please let us know, and we will correct it.





Google
 


© 1999-2017 : Legal : Privacy : Contact : About : Sitemap : RSS