Archive of UserLand's first discussion group, started October 5, 1998.
Re: Current Legal Issue
Author: Jeremy Bowers Posted: 8/2/2000; 12:17:33 PM Topic: Who owns what parts of your webpage? Msg #: 19405 (In response to 19403) Prev/Next: 19404 / 19406
How do you block a bot from accessing your page?Polite bots can be blocked using a ROBOTS.TXT file. Since your average search engine is a polite bot, you should be aware that if you are not careful, this will prevent your site from being indexed, generally considered a bad thing.
What software is available to "scrape" web pages?
If you can imagine it, it's probably available, from several sources.
You'll need to be more specific. Software that grabs data and does something with it? Software that can include a piece of several pages into one? Sites that frame your site? All of these exist, and they require different actions to block.
This page was archived on 6/13/2001; 4:55:57 PM.
© Copyright 1998-2001 UserLand Software, Inc.