Archive of UserLand's first discussion group, started October 5, 1998.

NewsNow: an example of unsolicited "scraping"

Author:Edd Dumbill
Posted:8/31/1999; 5:06:29 PM
Topic:130 RSS newsfeeds from Moreover.com
Msg #:10357 (In response to 10337)
Prev/Next:10356 / 10358

My acid test for this is "How would I feel if he scraped my site?" My answer is "I wouldn't like it." Would I object? I'm not sure. I certainly would object if he hadn't asked permission.

There are interesting questions which are going to make some lawyer a lot of money here.

When do search engines and web directories become more than that? Is moreover.com just a really clever search engine, specifically tuned for its target sites? Certainly few people would ever object to a search engine spidering them (although you do get the chance to say "no" through robots.txt).

The NewsNow site http://www.newsnow.co.uk/ has been doing this kind of thing for over a year in the UK and is now even selling its results (see http://www.newsnow.co.uk/info/NewsNowDirectQuote.htm) and undercutting traditional content providers. This is from their FAQ:

Q: Is NewsNowDirect legal?
A: Under UK law, the Copyright Act allows fair use when copying an insubstantial portion of a copyrighted work, especially for the purposes of reporting news and current affairs. Given that a headline is an insubstantial portion of an entire news story, and that NewsNowDirect only involves the copying of headlines, we have no reason to believe that NewsNowDirect is anything but legitimate and legal.

That is not to say that all UK content providers like what NewsNow are doing, though... However it does generate clickthroughs onto their sites.

So if one trains a search engine to look at only news sites and clean up the results, it's all of a sudden something different.

On the question of agreement, NewsNow says:

Since we have every confidence that NewsNowDirect is legitimate and legal, we do not believe the onus rests with NewsNow Publishing to make agreements with web publishers. Furthermore, web publishers that carry advertising increase revenues directly as a result of users that NewsNowDirect sends to their sites. However, many news providers - including the BBC, Ziff-Davis UK, VNU Business Publications, Associated New Media, The Register and Silicon News - have encouraged us to link to more of their content.

I thought this UK perspective might be an interesting addition to the discussion. It certainly is a precedent -- these guys don't seem to have been sued or closed down yet. However, neither do they seem to have eroded the traditional content provider market on mainstream sites.

One wonders though if moreover.com is the kiss of death for NewsNow.

There are responses to this message:

This page was archived on 6/13/2001; 4:52:19 PM.

© Copyright 1998-2001 UserLand Software, Inc.