Archive of UserLand's first discussion group, started October 5, 1998.

Re: 130 RSS newsfeeds from Moreover.com

Author:Ian Davis
Posted:8/31/1999; 4:53:04 PM
Topic:130 RSS newsfeeds from Moreover.com
Msg #:10355 (In response to 10353)
Prev/Next:10354 / 10356

If sites don't want to be scraped they can just put the following lines in their robots.txt:

User-agent: ScraperName
Disallow: /

This will stop any well behaved robot.

I get more robots visiting Internet Alchemy than real people, mostly for RSS files but a lot of search engines as well. How is syndicating news headlines any different from indexing web pages. Should I refrain from listing the headlines and just provide a search box?

Ian

There are responses to this message:

This page was archived on 6/13/2001; 4:52:19 PM.

© Copyright 1998-2001 UserLand Software, Inc.