Archive of UserLand's first discussion group, started October 5, 1998.
Re: Inktomi's Crawler and EditThisPage.Com Performance
Author: Brent Simmons Posted: 9/18/2000; 6:03:27 PM Topic: Inktomi's Crawler and EditThisPage.Com Performance Msg #: 21489 (In response to 21486) Prev/Next: 21488 / 21490
Are you sure it's a crawler and not some sort of WAP gateway?The User-Agent specifies that it's Slurp:
http://www.inktomi.com/slurp.html
The page on Slurp says it's a robot. (It also says it doesn't do some of the things it's been doing. Which isn't to say it's always badly behaved.)
This page was archived on 6/13/2001; 4:56:45 PM.
© Copyright 1998-2001 UserLand Software, Inc.