summaryrefslogtreecommitdiff
path: root/classes/Status_network.php
AgeCommit message (Collapse)Author
2010-02-16Stomp queue restructuring for mass scalability:Brion Vibber
- Multiplexing queues into groups and for multiple sites. - Sharing vs breakout configurable per site and per queue via $config['queue']['breakout'] - Detect how many times a message is redelivered, discard if it's killed too many daemons - count configurable with $config['queue']['max_retries'] - can dump the items to files in $config['queue']['dead_letter_dir'] Queue daemon memory & resource leak fixes: - avoid unnecessary reconnections to memcached server (switch persistent connections back in on second initialization, assuming it's child process) - monkey-patch for leaky .ini loads in DB_DataObject::databaseStructure() - was leaking 200k per active switch - applied leak fixes to Status_network as well, using intermediate base Safe_DataObject for both it and Memcache_DataObject Misc queue fixes: - correct handling of child processes exiting due to signal termination instead of regular exit - shutdown instead of infinite respawn loop if we're already past the soft memory limit at startup - Added --all option for xmppdaemon... still opens one xmpp connection per site that has xmpp active Cache updates: - add Cache::increment() method with native support for memcached atomic increment
2010-01-26Control channel for queue daemons to request graceful shutdown, restart, or ↵Brion Vibber
update to listen to a newly added or reconfigured site. queuectl.php --update -s<site> queuectl.php --stop queuectl.php --restart Default control channel is /topic/statusnet-control. For external utilities to send a site update ping direct to the queue server, connect via Stomp and send a message formatted thus: update:<nickname> (Nickname here, *not* server hostname! The rest of the queues will be updated to use nicknames later.) Note that all currently-connected queue daemons will get these notifications, including both queuedaemon.php and xmppdaemon.php. (XMPP will ignore site update requests for sites that it's not handling.) Limitations: * only implemented for stomp queue manager so far * --update may not yet handle a changed server name properly * --restart won't reload PHP code files that were already loaded at startup. Still need to stop and restart the daemons from 'outside' when updating code base.
2010-01-26Site metadata tags in status_network: single 'tags' field, pipe-separated.Brion Vibber
$sn->tags() returns tag list as array; $sn->hasTag('blah') to check for a particular tag only Could be used to control things in config file: $sn = Status_network::setupSite($_server, $_path, $_wildcard); if (!$sn) { die("No such site"); } if ($sn->hasTag('individual')) { /* blah */ } Note memcached keys are unchanged; if tags are changed from an external tool clear: statusnet:<dbname>:status_network:<key>:<val> for <key>s 'nickname', 'hostname', and 'pathname'
2010-01-24save nickname and wildcard when setting up status networkEvan Prodromou
2010-01-12Major refactoring of queue handlers to support running multiple sites in one ↵Brion Vibber
daemon. Key changes: * Initialization code moved from common.php to StatusNet class; can now switch configurations during runtime. * As a consequence, configuration files must now be idempotent... Be careful with constant, function or class definitions. * Control structure for daemons/QueueManager/QueueHandler has been refactored; the run loop is now managed by IoMaster run via scripts/queuedaemon.php IoManager subclasses are woken to handle socket input or polling, and may cover multiple sites. * Plugins can implement notice queue handlers more easily by registering a QueueHandler class; no more need to add a daemon. The new QueueDaemon runs from scripts/queuedaemon.php: * This replaces most of the old *handler.php scripts; they've been refactored to the bare handler classes. * Spawns multiple child processes to spread load; defaults to CPU count on Linux and Mac OS X systems, or override with --threads=N * When multithreaded, child processes are automatically respawned on failure. * Threads gracefully shut down and restart when passing a soft memory limit (defaults to 90% of memory_limit), limiting damage from memory leaks. * Support for UDP-based monitoring: http://www.gitorious.org/snqmon Rough control flow diagram: QueueDaemon -> IoMaster -> IoManager QueueManager [listen or poll] -> QueueHandler XmppManager [ping & keepalive] XmppConfirmManager [poll updates] Todo: * Respawning features not currently available running single-threaded. * When running single-site, configuration changes aren't picked up. * New sites or config changes affecting queue subscriptions are not yet handled without a daemon restart. * SNMP monitoring output to integrate with general tools (nagios, ganglia) * Convert XMPP confirmation message sends to use stomp queue instead of polling * Convert xmppdaemon.php to IoManager? * Convert Twitter status, friends import polling daemons to IoManager * Clean up some error reporting and failure modes * May need to adjust queue priorities for best perf in backlog/flood cases Detailed code history available in my daemon-work branch: http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
2010-01-10always set site/server to hostname if it existsEvan Prodromou
2010-01-09allow hostname with SSLEvan Prodromou
2010-01-09use nickname, not sitename, in domain for SSLEvan Prodromou
2010-01-09correct superglobal variable nameEvan Prodromou
2010-01-09redirect to sitename.wildcard for SSLEvan Prodromou
2009-11-10Bringing Sphinx search support up to code: broken out to a plugin, now ↵Brion Vibber
supports multiple sites on a single server. Upgrade notes: * Index names have changed from hardcoded 'Identica_people' and 'Identica_notices' to use the database name and actual table names. Must reindex. New events: * GetSearchEngine to override default search engine class selection from plugins New scripts: * gen_config.php generates a sphinx.conf from database configuration (with theoretical support for status_network table, but it doesn't seem to be cleanly queriable right now without knowing the db setup info for that. Needs generalized support.) * Replaced old sphinx-indexer.sh and sphinx-cron.sh with index_update.php Other fixes: * sphinx.conf.sample better matches our live config, skipping unused stopword list and using a more realistic indexer memory limit Further notes: * Probably doesn't work right with PostgreSQL yet; Sphinx can pull from PG but the extraction queries currently look like they use some MySQL-specific functions.
2009-08-28Status_network had wrong ini fileEvan Prodromou
2009-08-26define LACONICA and accept LACONICA for backwards compatibilityEvan Prodromou
2009-08-25global search and replace for laconica -> statusnetEvan Prodromou
2009-08-25change LACONICA to STATUSNETEvan Prodromou
2009-08-25a distributed -> the distributedEvan Prodromou
2009-08-25change Laconica and Control Yourself to StatusNet in PHP filesEvan Prodromou
2009-06-28bad string compare in Status_networkEvan Prodromou
2009-06-28fallback for www. addressesEvan Prodromou
2009-06-15forgot to disinherit Memcached_DataObject in Status_networkEvan Prodromou
2009-06-15Add some basic memcached handling to status_networkEvan Prodromou
Status_network can't be a subclass of Memcached_DataObject -- the latter is too entrenched in Laconica's memc handling functions, which aren't loaded when Status_network is running! But the importance of caching these values can't be overstated. So, a considerably slimmed-down version of the Memcached_DataObject code is transcribed into Status_network.
2009-06-15a little better query handling in redirect codeEvan Prodromou
2009-06-15a little better query handling in redirect codeEvan Prodromou
2009-06-15forgot some functions aren't available at status timeEvan Prodromou
2009-06-15redirect on non-canonical server nameEvan Prodromou
2009-06-15strncmp -> strcasecmpEvan Prodromou
2009-06-15Return network from network setup functionEvan Prodromou
Return the network from the network setup function. Also, special-case for when we get a server name the same as the wildcard.
2009-06-15updates to Status_networkEvan Prodromou
2009-05-22some class files had x bit setEvan Prodromou
2009-04-07Host multiple sites with the same codebaseEvan Prodromou
This is the beginning of the code for status.net and related status farms. It will read basic information about a site from a shared, central database and use the data stored there to switch on the hostname.