Stomp queue restructuring for mass scalability:

- Multiplexing queues into groups and for multiple sites.
- Sharing vs breakout configurable per site and per queue via $config['queue']['breakout']
- Detect how many times a message is redelivered, discard if it's killed too many daemons
 - count configurable with $config['queue']['max_retries']
 - can dump the items to files in $config['queue']['dead_letter_dir']

Queue daemon memory & resource leak fixes:
- avoid unnecessary reconnections to memcached server (switch persistent connections back in on second initialization, assuming it's child process)
- monkey-patch for leaky .ini loads in DB_DataObject::databaseStructure() - was leaking 200k per active switch
- applied leak fixes to Status_network as well, using intermediate base Safe_DataObject for both it and Memcache_DataObject

Misc queue fixes:
- correct handling of child processes exiting due to signal termination instead of regular exit
- shutdown instead of infinite respawn loop if we're already past the soft memory limit at startup
- Added --all option for xmppdaemon... still opens one xmpp connection per site that has xmpp active

Cache updates:
- add Cache::increment() method with native support for memcached atomic increment

1 files changed, 11 insertions, 5 deletions

diff --git a/classes/Status_network.php b/classes/Status_network.php
index 4bda24b6a..a452c32ce 100644
--- a/classes/Status_network.php
+++ b/classes/Status_network.php
@@ -21,7 +21,7 @@
 
 if (!defined('STATUSNET') && !defined('LACONICA')) { exit(1); }
 
-class Status_network extends DB_DataObject
+class Status_network extends Safe_DataObject
 {
     ###START_AUTOCODE
     /* the code below is auto generated do not remove the above tag */
@@ -57,6 +57,7 @@ class Status_network extends DB_DataObject
     ###END_AUTOCODE
 
     static $cache = null;
+    static $cacheInitialized = false;
     static $base = null;
     static $wildcard = null;
 
@@ -78,11 +79,15 @@ class Status_network extends DB_DataObject
         if (class_exists('Memcache')) {
             self::$cache = new Memcache();
 
-            // Can't close persistent connections, making forking painful.
+            // If we're a parent command-line process we need
+            // to be able to close out the connection after
+            // forking, so disable persistence.
             //
-            // @fixme only do this in *parent* CLI processes.
-            // single-process and child-processes *should* use persistent.
-            $persist = php_sapi_name() != 'cli';
+            // We'll turn it back on again the second time
+            // through which will either be in a child process,
+            // or a single-process script which is switching
+            // configurations.
+            $persist = php_sapi_name() != 'cli' || self::$cacheInitialized;
             if (is_array($servers)) {
                 foreach($servers as $server) {
                     self::$cache->addServer($server, 11211, $persist);
@@ -90,6 +95,7 @@ class Status_network extends DB_DataObject
             } else {
                 self::$cache->addServer($servers, 11211, $persist);
             }
+            self::$cacheInitialized = true;
         }
 
         self::$base = $dbname;