From dc62246443e3584ef5267505275f618f6fa86bf7 Mon Sep 17 00:00:00 2001 From: Evan Prodromou Date: Sun, 31 Jan 2010 10:12:26 -0500 Subject: Add a robots.txt URL to the site root Adds a robots.txt file to the site root. Defaults defined by 'robotstxt' section of config. New events StartRobotsTxt and EndRobotsTxt to let plugins add information. Probably not useful if path is not /, but won't hurt anything, either. --- README | 14 ++++++++++++++ 1 file changed, 14 insertions(+) (limited to 'README') diff --git a/README b/README index da278f741..4e576dcdd 100644 --- a/README +++ b/README @@ -1496,6 +1496,20 @@ interface. It also makes the user's profile the root URL. enabled: Whether to run in "single user mode". Default false. nickname: nickname of the single user. +robotstxt +--------- + +We put out a default robots.txt file to guide the processing of +Web crawlers. See http://www.robotstxt.org/ for more information +on the format of this file. + +crawldelay: if non-empty, this value is provided as the Crawl-Delay: + for the robots.txt file. see http://ur1.ca/l5a0 + for more information. Default is zero, no explicit delay. +disallow: Array of (virtual) directories to disallow. Default is 'main', + 'search', 'message', 'settings', 'admin'. Ignored when site + is private, in which case the entire site ('/') is disallowed. + Plugins ======= -- cgit v1.2.3-54-g00ecf