Archive of UserLand's first discussion group, started October 5, 1998.

---

robots.txt and F6,MR...

Author:
Posted:	4/15/1999; 10:15:09 AM
Topic:	robots.txt and F6,MR...
Msg #:	5043
Prev/Next:	5042 / 5044

If you are using the shortening URLs technique (<http://frontier.userland.com/tree$3.2.6.3>), you can pretty easily set up your own robots.txt file. Of course you have to have access to the top level of your web servers site. robots.txt only works at the top level of a website. For example <http://my.site.com/robots.txt>

Here's what I did. In my scratchpad.mySite table, I created an wp text object named "robots.txt".

In there I typed in...

# go away
User-agent: *
Disallow: /

That's it!!!

There are tons of sources on how the robots.txt file works. Here is one <http://info.webcrawler.com/mak/projects/robots/robots.html>.

Toodle-loooooo........ Thomas

---

This page was archived on 6/13/2001; 4:49:22 PM.

© Copyright 1998-2001 UserLand Software, Inc.