summaryrefslogtreecommitdiff
path: root/mirrors
AgeCommit message (Collapse)Author
2012-04-08Don't blow up when no mirror status data is availableDan McGee
The check here was wrong before; in the case of no mirror log entries the returned value will not be empty, but will contain two empty values. Check the values instead to see if we have valid data available. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-08Add migrations for on_delete attributesDan McGee
I don't think these end up doing anything at the database level, but since South insists on generating alter_column() calls for these changes, do them once so they don't keep reappearing in auto-generated future migrations. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-07Choose an up-to-date mirror for download URLsDan McGee
Given that we collect a lot of mirror status data, we can utilize it to ensure the download link on the website actually works and newly-added packages have actually been mirrored out. Add a method that attempts to use the mirror status data to determine a mirror we should redirect our download requests to. This can change on a regular basis, and falls back to the old method if no mirror status data is available. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-26Bump signoffs and mirror status JSON API versionsDan McGee
Due to datetime formatting changes in Django 1.4, we know follow the ECMA specification more closely and use 'yyyy-mm-ddThh:mm_ssZ' format. As this could break existing users of the JSON data, bump the version. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-24Use bulk_create() when inserting mirror log entriesrelease_2012-03-25Dan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-23Make all datetime objects fully timezone awareDan McGee
This is most of the transition to Django 1.4 `USE_TZ = True`. We need to ensure we don't mix aware and non-aware datetime objects when dealing with datetimes in the code. Add a utc_now() helper method that we can use most places, and ensure there is always a timezone attached when necessary. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-23Change Django urls.py importDan McGee
Until Django 1.3, the functions include(), patterns() and url() plus handler404, handler500 were located in a django.conf.urls.defaults module. In Django 1.4, they live in django.conf.urls. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-02-13Don't select FTP by default in mirrorlistsDan McGee
More generally, add a new 'default' column to the mirror protocol model so we can determine what is selected and shown by default. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-01-05Adjust page and content caching lengths and decoratorsDan McGee
Remove never_cache from many places now that we don't actually need it since we aren't caching by default. Adjust our cache_function decorator times be shorter values, and also randomize them a bit to make cache invalidations not all line up. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-11PyLint suggested cleanupsDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-11Switch back to using standard deviation in mirror check pageDan McGee
This got checked in by default, whoops. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-07Various small admin touchupsDan McGee
* Don't capitalize things in verbose_name, Django does this automatically and uses title case * Add overrides for IP, URL, ISO, etc. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-10Add package signoffs JSON viewDan McGee
This allows access to the same data (and even a bit more) from the signoffs overview page in a machine-friendly way. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-10-26Don't require mirrorlist URLs to have ISOsrelease_2011-10-26Dan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-10-11Pylint suggested and other cleanupsDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-22Admin setup tweaksDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-16Small Python convention fixupDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-07-29mirrors: do protocol formatting in templateDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-14Add JSON package details and files viewsDan McGee
These are retrieved by adding 'json/' to the normal package details or files view. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-14Allow mirror country field to be persisted to DB as NULLDan McGee
You need a custom field type in Django to allow this. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-09Management command cleanupDan McGee
Now that we aren't seeing odd segfaults and hung tasks, we can remove the traceback stuff from the scripts. Also use the 'io' module only, it has been long enough. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-18mirrors: pylint discovered cleanupsDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-18Never parse generated lastsync fileDan McGee
We should be encouraging our mirrors to serve us the original file, not something they create and come up with. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-12Add optional country override for individual mirror URLsDan McGee
This allows a named top-level mirror to have geographically distributed URLs, e.g. kernel.org and the geo-DNS setup. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-07Utilize Django 1.3 'on_delete' feature on several foreign keysDan McGee
The most important one here is PROTECT to keep people from making bone-headed plays and deleting an Arch or Repo and every package along with it. We can use this in a few other places, as well as some carefully placed SET_NULL indicators. Note that nothing here pushes deletion responsibilities down to the database, although that will probably happen in a future commit. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-05Make mirror name and URLs uniqueDan McGee
So we don't create duplicates without knowing it in our database, like we did earlier today. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-28Enhance the mirror details pageDan McGee
Get the URLs with their performance data showing up, and simplify the top part for non-authenticated users while adding more detail for logged-in users. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-28Move country filter last in mirror adminDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-03Add ordering to mirror protocolsDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-02-27Auto-resolve mirror URLs on saveDan McGee
This prevents people from having to mess with these checkboxes at all in the admin, and we incur no delay on their initial values being correct waiting for the cron job to run. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-02-27Auto map the protocol URL fieldDan McGee
And perform better validation when doing so. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-02-22Slight tweaks to mirror commandsDan McGee
We know we are doing updates when setting IPv4/IPv6 information, so set force_update to True to save the useless select query on each save(). For mirror checks, use a less cumbersome deque for the results since it is also thread-safe, and have all the log entries committed in one go. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-02-22Modularize URLsDan McGee
Make some additional URL config files that can be included so we aren't trying to do so much in the top level config. This also allows us to branch a bit more rather than go linear down the rather lengthy list. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-01-31Defer format string substitution to loggerDan McGee
Don't use 'fmtstr % (arg1, arg2)' type format; logger can be passed a format string and the arguments to populate it. Saves a bit of work for strings that never end up getting displayed anyway. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-13Add a mirror status JSON viewDan McGee
Requested in FS#21144. This should provide most if not all of the data that was provided on the archlinux.de website, although there are some differences in what is returned to the user. It is nearly the same data as that provided in the HTML view, the difference being things are a bit more machine-friendly and the list is not split into good and bad portions. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-13Move database interaction out of the threaded sectionDan McGee
We were seeing a lot of hangs and long-running never-ending processes. This might be due to some multithreading issues within Django, so move the save() calls to a loop after the join() on the threads doing the mirror polling. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-12Use check count for this URL, not max of all mirrorsDan McGee
Prevents a recently enabled mirror from getting unfairly represented as far as completion percentage goes. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-07Fix mirror check unicode functionDan McGee
Needed a few self. qualifiers in there. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-07Don't require country on mirrorlist submissionDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2010-11-04Ensure mirrorlist generator works if no status availableDan McGee
Saw this error come through on the live site today, as well as being reproducible when no mirror check runs have happened in the last 24 hours on a development machine. Let mirrors that have no available checks show up on this page, but be sorted last and show a score of unknown. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-13Show all mirrors if no IP version specifiedrelease_2010-10-13Dan McGee
I'm eating my words on this one- we don't want to filter to none() if we don't have an IP version specified, because people hitting the URL without a ip_version parameter would no longer get any mirrors back. Filter if they are provided by using a Q() object. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-13Allow filtering of mirrorlist by IP-versionPyroPeter
Dan: Use a forms.MultipleChoiceField to match the protocol selection. Signed-off-by: PyroPeter <abi1789@googlemail.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-13Mirror check script cleanupDan McGee
Don't import all the constants from logging, just use logging.* instead. Also, fix some typos that somehow snuck into one of my commits. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-13Add mirrorresolv manage.py commandPyroPeter
Dan: * Fix up some style issues such as spacing between operators * Ensure one failed lookup doesn't crash the whole script * Be silent out of the box if there are no errors, just like mirrorcheck Signed-off-by: PyroPeter <abi1789@googlemail.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-13Add has_ipv{4,6} to MirrorUrlPyroPeter
Dan: Add some more useful labels for use in the admin. Signed-off-by: PyroPeter <abi1789@googlemail.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-13Add 'All' choice and make it the defaultPyroPeter
Signed-off-by: PyroPeter <abi1789@googlemail.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-11mirrorcheck: catch and handle another socket timeout caseDan McGee
We were seeing processes hang on the Arch server. It looks like there are ways for socket.timeout to come out of the main check code, so add another except block to catch this case. In addition, make sure we always call task_done() even on failures so processes eventually die. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-07Factor check completion pct into mirror scorerelease_2010-10-07Dan McGee
Use it as the divisor in our slightly longer equation. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-07mirrorcheck: record duration on file not found errorsDan McGee
On an HTTP 404, FTP 550, or inability to parse the lastsync file, record the duration of the check even though we couldn't get a time from the mirror. This allows for these checks to show up as completed but in error, which is more what. Previously, inability to parse the date was also recorded as a success, so change that to be a failure and record an error message with it. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-01Fix an off by one error in math for check intervalDan McGee
Because we are averaging the interval and not the value, we need to subtract one from the total we are dividing by. Whoops. Signed-off-by: Dan McGee <dan@archlinux.org>
generated by cgit v1.2.3-54-g00ecf (git 2.45.2) at 2024-12-02 04:39:10 +0000