Archive of UserLand's first discussion group, started October 5, 1998.
Re: 130 RSS newsfeeds from Moreover.com
Author: Ian Davis Posted: 8/31/1999; 4:53:04 PM Topic: 130 RSS newsfeeds from Moreover.com Msg #: 10355 (In response to 10353) Prev/Next: 10354 / 10356
If sites don't want to be scraped they can just put the following lines in their robots.txt:User-agent: ScraperName
Disallow: /This will stop any well behaved robot.
I get more robots visiting Internet Alchemy than real people, mostly for RSS files but a lot of search engines as well. How is syndicating news headlines any different from indexing web pages. Should I refrain from listing the headlines and just provide a search box?
Ian
There are responses to this message:
- Re: 130 RSS newsfeeds from Moreover.com, Dave Winer, 8/31/1999; 5:05:39 PM
This page was archived on 6/13/2001; 4:52:19 PM.
© Copyright 1998-2001 UserLand Software, Inc.