Archive of UserLand's first discussion group, started October 5, 1998.

Re: 130 RSS newsfeeds from Moreover.com

Author:Andrew Wooldridge
Posted:8/31/1999; 2:52:15 PM
Topic:130 RSS newsfeeds from Moreover.com
Msg #:10345 (In response to 10340)
Prev/Next:10344 / 10346

There should be a sort of robots.txt file for sydication - where the syndication bot asks if its ok to scrape the site or not, just like search engines ask if it's ok to search the site or not. There is really not much difference. I could imagine search engines evolving to become 'syndication' engines where they walk the net and package everything.

There are responses to this message:

This page was archived on 6/13/2001; 4:52:19 PM.

© Copyright 1998-2001 UserLand Software, Inc.