public marks

PUBLIC MARKS from znarf with tags search & server

January 2006

July 2005

Robots.txt, The Big Crawl

(via)
While testing our new robots.txt validator and checker, we needed some test fodder. We found more than 5% of the robots.txt used bad style and up to 2% were so badly formed that they would not be recognized by any spider. The following lists some of the problems we discovered.

François Hodierne's TAGS related to tag search

amazon +   atom +   beagle +   blog +   bloglines +   blogmarks.net +   business +   desktop +   development +   digg +   flickr +   gnome +   google +   hotlinked +   linux +   microformats +   minibarcamp +   mp3 +   msn +   music +   open directory +   opensearch +   photo +   photography +   pierre chappaz +   ping +   privacy +   rest +   rss +   server +   six apart +   social bookmarking +   social software +   tags +   Tantek Çelik +   technorati +   web 2.0 +   wikio +   windows +   xml-rpc +   yahoo +   youtube +