summaryrefslogtreecommitdiff
path: root/mirrors
AgeCommit message (Collapse)Author
2013-04-14Remove cache_function decorator from a few spotsDan McGee
The benefit of these storage operations might be outweighed by the cost, especially given how infrequently these functions are called. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-04-14Reduce mirror status query madnessDan McGee
Move completely to custom SQL for this logic. The Django ORM just doesn't play nice with the kind of query we are looking to do, so it is easier to do using raw SQL. The biggest pain factor here is in supporting sqlite as it doesn't have nearly the capabilities in handling datetime types directly in the database, as well as having some different type conversion necessities. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-04-13Support only a single mirror ID in error/status retrievalDan McGee
This simplifies things and makes injecting this single mirror ID into custom SQL a whole lot easier. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-04-13Calculate average URL delay in the databaseDan McGee
Rather than doing this in the Python code and needing 12,000+ rows returned from the database, we can do it in the database and get fewer than 300 rows back. If I recall correctly, the reason this was not done originally was due to our usage of MySQL and some really bad date math/overflow stuff it did when the interval between last_sync and check_time were greater than about a week. Luckily, we have switched to using a more sane database. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-04-10Allow mirror rsync IPs to be IPv4/IPv6 addresses or networksrelease_2013-04-10Dan McGee
This gives us a bunch more flexibility on this field, and now supports all the options that the rsync config file supports. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-29Add IP family lookup to CheckLocation modelDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-13mirrorcheck: s/if/elif/ when determining what check function to runrelease_2013-03-13Dan McGee
This was a silly thinko here; it caused the logs to fill up with a bunch of 'unknown url type: rsync' errors. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-12Don't check FTP + IPv6 combinationDan McGee
Very few, if any, FTP servers support connections over IPv6. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-06Don't add blank options to rsync command linerelease_2013-03-06Dan McGee
Rsync doesn't like this so much: Unexpected remote arg: rsync://mirror.example.com/archlinux/lastsync rsync error: syntax or usage error (code 1) at main.c(1214) [sender=3.0.9] Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-06Teach mirrorcheck management command about check locationsDan McGee
This adds the -l/--location argument to the command in order to pass in a check location that we are currently running from. This locks the IP address family to the one derived from the address on that location, and stores any check results tagged with a location ID. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-06Add location ID to mirror logsDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-06Add family property to mirror check locationDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-06Revert "mirrorcheck: Don't use bulk_create on sqlite3"Dan McGee
This reverts commit 3c4ceb16. We don't need this anymore as bulk_create gets automatic batching now on sqlite3 so it is safe to use. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-06Add mirror CheckLocationAdminDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-06Add 'created' field to more mirror modelsDan McGee
We have been better about doing this to most of our models, but the ones here didn't have a created field. Add it where appropriate and set a reasonably old default value. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-06Add mirror check locations modelDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-04Add HTTPS mirror protocol fixtureDan McGee
For now, it is not included in the default selection, but we have a few existing mirrors that do support it. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-02-09Use 'update_fields' model.save() kwargDan McGee
This was added in Django 1.5 and allows saving only a subset of a model's fields. It makes sense in a few cases to utilize it. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-20Lengthen the mirror rsync IP address fieldDan McGee
Make it long enough to support a full-form IPv6 address with a subnet. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-16Remove debugging print statementDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-14Support mirror status JSON by tierDan McGee
Just as we do for the normal status HTML view. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-14Ensure URLs without check data work on mirror details pageDan McGee
Less noticeable in production as the templates don't show '@@@INVALID@@@' there, but we were trying to access attributes that don't actually exist on certain mirror objects. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-14Drop country column from mirror tableDan McGee
We now always look for this information at the URL level, not the mirror level. This simplifies quite a bit of code in and around the mirror views. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-14Add migration to move country data down to the URL levelDan McGee
Rather than have the weird indirection we need now to find the right country for URLs, just always store it on the URL. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-13Use content_type and not mimetype on HttpResponse()Dan McGee
Bug #16519 in Django deprecates mimetype, so update our code accordingly. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-12Round two of mirror status query improvementsDan McGee
This seems to generate much more performant queries at the database level than what we were previously doing, and also doesn't show duplicate rows. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-12-27Ensure mirror protocols are distinctDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-21Fix mirror URL duplication in status viewDan McGee
We need to ensure we don't duplicate URLs in the status view, so add a distinct() call back in to the queryset when it was inadvertently dropped in commit a2cfa7edbb. This negates a lot of the performance gains we had, unfortunately, so it looks like a nested subquery might be more efficient. Disappointing the planner can't do this for us. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-16Use Python 2.7 dictionary comprehension syntaxDan McGee
Rather than the old idiom of dict((k, v) for <> in <>). Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-16Optimize mirror status data fetchingDan McGee
Now that we have as many mirror URLs as we do, we can do a better job fetching and aggregating this data. The prior method resulted in a rather unwieldy query being pushed down to the database with a horrendously long GROUP BY clause. Instead of trying to group by everything at once so we can retrieve mirror URL info at the same time, separate the two queries- one for getting URL performance data, one for the qualitative data. The impetus behind fixing this is the PostgreSQL slow query log in production; this currently shows up the most of any queries we run in the system. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-12Prefetch the available protocols on the mirror overview pageDan McGee
Otherwise we are doing one query per mirror, which at this point is over 100 separate queries. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-12Add get_latest_by to MirrorLog Meta classDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-11Add title to mirror status data pointsDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-11Mirror graph tweaking after usage with real dataDan McGee
* Clamp y-axis minimum to 0. * Don't plot `is_success == false` values. * Ensure URLs are sorted predictably. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-11Re-add nice() calls for mirror status axesDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-11Mirror status graph, now with points AND linesDan McGee
We might have to tweak the interpolation method once we see this with real data, but for now it looks really pretty locally. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-10Enable mirror status graph resizingDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-10Add initial version of mirror status chartDan McGee
Still have some hardcoded stuff to rip out and replace to make this a bit more dynamic on things like sizing, but for now, this is a great start. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-10Allow filtering retrieved mirror statuses by mirror_idDan McGee
When we don't need them all, no need to fetch them all. Let the database do the work for us, hopefully. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-10Add mirror extended status JSON viewDan McGee
When asking for status for a single mirror, we can include logs from the past 24 hours in addition to the normal information we provide. This is slated for usage by a frontend graph still to come, similar to those on the NTP pool website. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-10-10Make mirror log time query a bit more efficientDan McGee
We don't need the full mirror log objects; we just need a very small subset of values from them here to do the required math and object building. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-09-18Sort mirrorlist by English country name, not ISO codeDan McGee
Fixes FS#31503. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-08-07Push more default values down into the databaseDan McGee
This makes it easier to do manual manipulation/insertion/etc. at the database level, as well as just making things act more sane from an overall software stack perspective. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-31Declare 'enums' at class scopeDan McGee
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-24Replace deprecated direct_to_template() with render() shortcutDan McGee
Now that Django actually provides a concise way to use a RequestContext object without instantiating it, we can use that rather than the old function-based generic view that worked well to do the same. Additionally, these function-based generic views will be gone in Django 1.5, so might as well make the move now. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-24Remove custom utc_now() function, use django.utils.timezone.now()Dan McGee
This was around from the time when we handled timezones sanely and Django did not; now that we are on 1.4 we no longer need our own code to handle this. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-08Correctly reassign queryset with added annotation in mirror statusrelease_2012-07-08Dan McGee
This was a dumb oversight on my part in commit 0f3c894e7a0. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-08mirrorcheck: Don't use bulk_create on sqlite3Dan McGee
It isn't worth it, as we run into the 999 max SQL statement variables issue when using it on any significant amount of mirrors. Since this is just a development database setup, and it isn't a command we need to run especially fast, we can ditch it. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-08Don't include StdDev on sqlite3 mirror status queryDan McGee
Because this function isn't shipped by default, it makes more sense to just omit it completely from the query we do to build the tables on this page when in development. Substitute 0.0 for the value so the rest of the calculations and display work as expected. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-08Handle HTTPException being thrown in mirrorcheckDan McGee
Managed to see this bubble up today when running the mirrorcheck command on a less than ideal connection that was experiencing timeouts at the wrong time. Signed-off-by: Dan McGee <dan@archlinux.org>
generated by cgit v1.2.3-54-g00ecf (git 2.45.2) at 2024-11-09 09:11:34 +0000