public marks

PUBLIC MARKS from znarf with tags server & search

January 2006

July 2005

Robots.txt, The Big Crawl

(via)
While testing our new robots.txt validator and checker, we needed some test fodder. We found more than 5% of the robots.txt used bad style and up to 2% were so badly formed that they would not be recognized by any spider. The following lists some of the problems we discovered.

François Hodierne's TAGS related to tag server

ajax +   amazon +   apache +   arm +   backup +   blogmarks.net +   browsers +   debian +   del.icio.us +   delicious +   development +   EC2 +   email +   firefox sync +   flickr +   gandi +   gtd +   hardware +   hotlinked +   http +   internet +   jabber +   javascript +   knockd +   linux +   lua +   mozilla +   msn +   mysql +   network +   nginx +   performance +   php +   python +   rails +   rss +   search +   security +   social bookmarking +   spam +   sql +   sqlite +   ubuntu +   unix +   weave +   windows +   wordpress +   xml +   xmlhttprequest +