Archive of UserLand's first discussion group, started October 5, 1998.

---

Robots.txt directs search engines what to index

Author:	Jeff Willden
Posted:	12/7/1998; 4:49:43 PM
Topic:	Would you do some work for more hits?
Msg #:	935 (In response to 864)
Prev/Next:	934 / 936

I found out about robots.txt when I saw it popping up as a "file not found" in the logs for my web servers. So I went and found out what was supposed to be in a robots.txt file and what implications that has on how they index your site.

If search engines start looking for it, then people will want to find out what this new thing is that search engines are looking for.

If you build it they will come?

---

This page was archived on 6/13/2001; 4:46:22 PM.

© Copyright 1998-2001 UserLand Software, Inc.