summaryrefslogtreecommitdiff
path: root/mirrors
AgeCommit message (Collapse)Author
2011-02-22Slight tweaks to mirror commandsDan McGee
We know we are doing updates when setting IPv4/IPv6 information, so set force_update to True to save the useless select query on each save(). For mirror checks, use a less cumbersome deque for the results since it is also thread-safe, and have all the log entries committed in one go. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-02-22Modularize URLsDan McGee
Make some additional URL config files that can be included so we aren't trying to do so much in the top level config. This also allows us to branch a bit more rather than go linear down the rather lengthy list. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-01-31Defer format string substitution to loggerDan McGee
Don't use 'fmtstr % (arg1, arg2)' type format; logger can be passed a format string and the arguments to populate it. Saves a bit of work for strings that never end up getting displayed anyway. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-13Add a mirror status JSON viewDan McGee
Requested in FS#21144. This should provide most if not all of the data that was provided on the archlinux.de website, although there are some differences in what is returned to the user. It is nearly the same data as that provided in the HTML view, the difference being things are a bit more machine-friendly and the list is not split into good and bad portions. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-13Move database interaction out of the threaded sectionDan McGee
We were seeing a lot of hangs and long-running never-ending processes. This might be due to some multithreading issues within Django, so move the save() calls to a loop after the join() on the threads doing the mirror polling. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-12Use check count for this URL, not max of all mirrorsDan McGee
Prevents a recently enabled mirror from getting unfairly represented as far as completion percentage goes. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-07Fix mirror check unicode functionDan McGee
Needed a few self. qualifiers in there. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-07Don't require country on mirrorlist submissionDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2010-11-04Ensure mirrorlist generator works if no status availableDan McGee
Saw this error come through on the live site today, as well as being reproducible when no mirror check runs have happened in the last 24 hours on a development machine. Let mirrors that have no available checks show up on this page, but be sorted last and show a score of unknown. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-13Show all mirrors if no IP version specifiedrelease_2010-10-13Dan McGee
I'm eating my words on this one- we don't want to filter to none() if we don't have an IP version specified, because people hitting the URL without a ip_version parameter would no longer get any mirrors back. Filter if they are provided by using a Q() object. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-13Allow filtering of mirrorlist by IP-versionPyroPeter
Dan: Use a forms.MultipleChoiceField to match the protocol selection. Signed-off-by: PyroPeter <abi1789@googlemail.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-13Mirror check script cleanupDan McGee
Don't import all the constants from logging, just use logging.* instead. Also, fix some typos that somehow snuck into one of my commits. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-13Add mirrorresolv manage.py commandPyroPeter
Dan: * Fix up some style issues such as spacing between operators * Ensure one failed lookup doesn't crash the whole script * Be silent out of the box if there are no errors, just like mirrorcheck Signed-off-by: PyroPeter <abi1789@googlemail.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-13Add has_ipv{4,6} to MirrorUrlPyroPeter
Dan: Add some more useful labels for use in the admin. Signed-off-by: PyroPeter <abi1789@googlemail.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-13Add 'All' choice and make it the defaultPyroPeter
Signed-off-by: PyroPeter <abi1789@googlemail.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-11mirrorcheck: catch and handle another socket timeout caseDan McGee
We were seeing processes hang on the Arch server. It looks like there are ways for socket.timeout to come out of the main check code, so add another except block to catch this case. In addition, make sure we always call task_done() even on failures so processes eventually die. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-07Factor check completion pct into mirror scorerelease_2010-10-07Dan McGee
Use it as the divisor in our slightly longer equation. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-07mirrorcheck: record duration on file not found errorsDan McGee
On an HTTP 404, FTP 550, or inability to parse the lastsync file, record the duration of the check even though we couldn't get a time from the mirror. This allows for these checks to show up as completed but in error, which is more what. Previously, inability to parse the date was also recorded as a success, so change that to be a failure and record an error message with it. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-01Fix an off by one error in math for check intervalDan McGee
Because we are averaging the interval and not the value, we need to subtract one from the total we are dividing by. Whoops. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-30Update MirrorProtocol fixture dataDan McGee
Move it to the correct location and add the new field. Also tidy up some of the instructions dealing with loading this data. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-30Use new is_download fieldDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-30Add is_download field to mirror protocolsDan McGee
This will replace all the usages of '!= rsync' and 'is ftp or http' we have in the code with one check on a boolean flag. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-30Mirror status improvementsDan McGee
* Fix sorting issues. '', 'unknown', and '∞' should now always sort after anything else in the list. * Add a completion percentage column; this will tell you at a glance if a mirror is sometimes unresponsive. This should probably be incorporated into the mirror score. * Make a few more things dynamic in the template, like the time back the page reflects. * Add some additional template tags for formatting things. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-29Fix migration dependencies exposed after moving modelsDan McGee
When we moved some models from one app to another, we didn't do anything to ensure the tables were created at all initially. Enforce this by adding the minimal required dependencies- those migrations in the 'main' model that last touched the involved models moving between apps. Noticed-by: Angel Velasquez <angvp@archlinux.org> Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-24Add a basic mirror details pageDan McGee
Still some work to do here, but this covers the basics of the public view we can show for mirrors and their associated data. The upstream and downstream links should be working OK to aid navigation, but right now we have some potential dead links for non-authenticated users if they click a link to a "private" mirror. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-24Make general mirror list view publicDan McGee
Hide some columns when not logged in because they aren't relevant for the general public, but this will work nicely as a base page for all of our known mirrors. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-24Give more information about mirror check runs and frequencyDan McGee
Show how many times the check has ran in the last 24 hours, as well as the average interval between checks. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-24Mirror status query refinementsDan McGee
Only show errors for active and public mirrors, and collapse two filter calls into just one for our normal status query. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-22Switch mirror status delay display to average delayrelease_2010-09-22Dan McGee
This takes a bit more work to compute, but since we cache all of this anyway it isn't too big of deal. Using average delay instead of last delay will be a bit more fair on mirrors that have odd syncing schedules, as well as exposing those that only sync once a day. Also fix an issue that will arise with cutoff_time being calculated once, and adjust mirror score to treat hours delay as a float rather than an integer. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-21Allow generated mirrorlist to take status info into accountDan McGee
By using the mirror score we calculate, we can sort the mirrors in the generated mirrorlist for people. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-21Allow caching of mirror status infoDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-21Prevent 500 if no mirror checks have ranrelease_2010-09-21Dan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-21Add ordering, sorting, and a lot more info to mirror status pageDan McGee
This should get this to the point where it is releasable to the general public for their use and pleasure. Still not sure on how often the check should be run, and we probably want to incorporate this mined data into some other things like the mirror list generator. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-21Small updates to mirrorcheck commandDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-21Add debugger when having to parse lastsync fileDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-21Add mirror status viewDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-21Add 'mirrorcheck' commandDan McGee
This does the actual work of going out and checking the mirror status. In short, it polls every active mirror URL for the 'lastsync' file and then records the appropriate details. These include the contents of that file, how long the total time to retrieve took, and any errors encountered. In order to finish up a bit faster, we spawn several threads to do the actual work. This parallelization allows the whole check process to take around 30 seconds rather than several minutes. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-21Rename generate mirrorlist viewDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-21Add new MirrorLog model and associated migrationDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-16Remove some leftover print statementsDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-06Move mirror models out of main appDan McGee
South actually makes this relatively painless if you get everything right, so might as well start getting these out of the legacy main application to eventually eliminate models being separate from their views. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-06Put mirrors app under South controlDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-03Use direct_to_template in all remaining possible placesDan McGee
Rather than the need to include RequestContext() calls directly, we can just use direct_to_template to do all the work for us. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-08-28PyLint suggested cleanupsDan McGee
We had a bunch of extra imports, non-conventional variable names, spacing issues, etc. that were relatively low-hanging fruit to clean up. Fix them and make the code a bit cleaner in the process. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-07-02Default country selection to AnyDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2010-07-02mirrorlist: allow filtering by protocolDan McGee
And add some more "static" URLs to access the non-country filtered lists. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-07-02Get mirrorlist view ready for general useDan McGee
Make the page much more flexible- allow multiple countries to be selected rather than just one in the form. Also add a lot more text to the page, and move the 'all' option out into its own subheading rather than being in the same form. Both GET and POST requests are now allowed for ease of use from non-browser scenarios or those that wish to update their mirrorlist automatically and submit parameters to the URL. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-07-02Remove arch from mirrorlist generationDan McGee
With pacman 3.4.0, we can now use the $arch variable so we don't need architecture-specific mirrorlists. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-02-10Remove archweb prefix from all importsDan McGee
Unnecessary, and lets us standardize on not using it everywhere. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-01-30Use select_related() for some mirror pagesDan McGee
We also traverse relationships here, so select the associated items. Signed-off-by: Dan McGee <dan@archlinux.org>