Archive of UserLand's first discussion group, started October 5, 1998.

Inktomi's Crawler and EditThisPage.Com Performance

Author:Brent Simmons
Posted:9/18/2000; 5:48:15 PM
Topic:Inktomi's Crawler and EditThisPage.Com Performance
Msg #:21485
Prev/Next:21484 / 21486

Inktomi's crawler has been hitting the EditThisPage.Com server hard for at least the second day in a row.

It's repeatedly, rapidly requesting the same documents over and over. Interestingly, it's always WAP pages.

If it was just crawling, it wouldn't be such a problem, it would crawl and move on -- but it's as if it's stuck on a few pages, and it just keeps hitting them.

Naturally, we've taken steps to deal with the issue. But what would be ideal is if their crawler stopped sending these requests.

There are responses to this message:

This page was archived on 6/13/2001; 4:56:45 PM.

© Copyright 1998-2001 UserLand Software, Inc.