Add a robots.txt URL to the site root

Adds a robots.txt file to the site root. Defaults defined by
'robotstxt' section of config. New events StartRobotsTxt and
EndRobotsTxt to let plugins add information. Probably not
useful if path is not /, but won't hurt anything, either.

1 files changed, 14 insertions, 0 deletions

diff --git a/README b/README
index da278f741..4e576dcdd 100644
--- a/README
+++ b/README
@@ -1496,6 +1496,20 @@ interface. It also makes the user's profile the root URL.
 enabled: Whether to run in "single user mode". Default false.
 nickname: nickname of the single user.
 
+robotstxt
+---------
+
+We put out a default robots.txt file to guide the processing of
+Web crawlers. See http://www.robotstxt.org/ for more information
+on the format of this file.
+
+crawldelay: if non-empty, this value is provided as the Crawl-Delay:
+            for the robots.txt file. see http://ur1.ca/l5a0
+            for more information. Default is zero, no explicit delay.
+disallow: Array of (virtual) directories to disallow. Default is 'main',
+          'search', 'message', 'settings', 'admin'. Ignored when site
+          is private, in which case the entire site ('/') is disallowed.
+
 Plugins
 =======