Archive of UserLand's first discussion group, started October 5, 1998.
Re: Some advise for newschannel tool writers...
Author: Ian Davis Posted: 7/14/1999; 4:29:58 AM Topic: Some advise for newschannel tool writers... Msg #: 8456 (In response to 8454) Prev/Next: 8455 / 8457
I would add:
- If you're writing a robot to grab channels, make sure you provide a contact address in the user agent string so the owner of the site knows who to contact if there is a problem. Otherwise you risk being excluded from the site completely
- Read up on the robots exclusion protocol and build that into your robot - it's easy to do with many packages.
In the interests of making this information public I run two robots, one active, the other under development, they identify themselves as:
rssmaker/1.0;webmaster@openjava.org
channelmirror/1.0;webmaster@startshere.netI'd like other bot owners to do the same, I have a number of Frontier/6.0 entries in my logs that it would be nice to identify.
.id.
This page was archived on 6/13/2001; 4:51:23 PM.
© Copyright 1998-2001 UserLand Software, Inc.