| Age | Commit message (Collapse) | Author |
|---|
|
You need a custom field type in Django to allow this.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Now that we aren't seeing odd segfaults and hung tasks, we can remove
the traceback stuff from the scripts. Also use the 'io' module only, it
has been long enough.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
We should be encouraging our mirrors to serve us the original file, not
something they create and come up with.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This allows a named top-level mirror to have geographically distributed
URLs, e.g. kernel.org and the geo-DNS setup.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
The most important one here is PROTECT to keep people from making
bone-headed plays and deleting an Arch or Repo and every package along
with it. We can use this in a few other places, as well as some
carefully placed SET_NULL indicators.
Note that nothing here pushes deletion responsibilities down to the
database, although that will probably happen in a future commit.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
So we don't create duplicates without knowing it in our database, like
we did earlier today.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Get the URLs with their performance data showing up, and simplify the
top part for non-authenticated users while adding more detail for
logged-in users.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This prevents people from having to mess with these checkboxes at all in
the admin, and we incur no delay on their initial values being correct
waiting for the cron job to run.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
And perform better validation when doing so.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
We know we are doing updates when setting IPv4/IPv6 information, so set
force_update to True to save the useless select query on each save(). For
mirror checks, use a less cumbersome deque for the results since it is also
thread-safe, and have all the log entries committed in one go.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Make some additional URL config files that can be included so we aren't
trying to do so much in the top level config. This also allows us to
branch a bit more rather than go linear down the rather lengthy list.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Don't use 'fmtstr % (arg1, arg2)' type format; logger can be passed a format
string and the arguments to populate it. Saves a bit of work for strings
that never end up getting displayed anyway.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Requested in FS#21144. This should provide most if not all of the data that
was provided on the archlinux.de website, although there are some
differences in what is returned to the user. It is nearly the same data as
that provided in the HTML view, the difference being things are a bit more
machine-friendly and the list is not split into good and bad portions.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
We were seeing a lot of hangs and long-running never-ending processes. This
might be due to some multithreading issues within Django, so move the save()
calls to a loop after the join() on the threads doing the mirror polling.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Prevents a recently enabled mirror from getting unfairly represented as far
as completion percentage goes.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Needed a few self. qualifiers in there.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Saw this error come through on the live site today, as well as being
reproducible when no mirror check runs have happened in the last 24 hours on
a development machine. Let mirrors that have no available checks show up on
this page, but be sorted last and show a score of unknown.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
I'm eating my words on this one- we don't want to filter to none() if we
don't have an IP version specified, because people hitting the URL without a
ip_version parameter would no longer get any mirrors back. Filter if they
are provided by using a Q() object.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Dan: Use a forms.MultipleChoiceField to match the protocol selection.
Signed-off-by: PyroPeter <abi1789@googlemail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Don't import all the constants from logging, just use logging.* instead.
Also, fix some typos that somehow snuck into one of my commits.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Dan:
* Fix up some style issues such as spacing between operators
* Ensure one failed lookup doesn't crash the whole script
* Be silent out of the box if there are no errors, just like mirrorcheck
Signed-off-by: PyroPeter <abi1789@googlemail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Dan: Add some more useful labels for use in the admin.
Signed-off-by: PyroPeter <abi1789@googlemail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: PyroPeter <abi1789@googlemail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
We were seeing processes hang on the Arch server. It looks like there are
ways for socket.timeout to come out of the main check code, so add another
except block to catch this case. In addition, make sure we always call
task_done() even on failures so processes eventually die.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Use it as the divisor in our slightly longer equation.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
On an HTTP 404, FTP 550, or inability to parse the lastsync file, record the
duration of the check even though we couldn't get a time from the mirror.
This allows for these checks to show up as completed but in error, which is
more what.
Previously, inability to parse the date was also recorded as a success, so
change that to be a failure and record an error message with it.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Because we are averaging the interval and not the value, we need to subtract
one from the total we are dividing by. Whoops.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Move it to the correct location and add the new field. Also tidy up some of
the instructions dealing with loading this data.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This will replace all the usages of '!= rsync' and 'is ftp or http' we have
in the code with one check on a boolean flag.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
* Fix sorting issues. '', 'unknown', and '∞' should now always sort after
anything else in the list.
* Add a completion percentage column; this will tell you at a glance if a
mirror is sometimes unresponsive. This should probably be incorporated
into the mirror score.
* Make a few more things dynamic in the template, like the time back the
page reflects.
* Add some additional template tags for formatting things.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
When we moved some models from one app to another, we didn't do anything to
ensure the tables were created at all initially. Enforce this by adding the
minimal required dependencies- those migrations in the 'main' model that
last touched the involved models moving between apps.
Noticed-by: Angel Velasquez <angvp@archlinux.org>
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Still some work to do here, but this covers the basics of the public view we
can show for mirrors and their associated data. The upstream and downstream
links should be working OK to aid navigation, but right now we have some
potential dead links for non-authenticated users if they click a link to a
"private" mirror.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Hide some columns when not logged in because they aren't relevant for the
general public, but this will work nicely as a base page for all of our
known mirrors.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Show how many times the check has ran in the last 24 hours, as well as the
average interval between checks.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Only show errors for active and public mirrors, and collapse two filter
calls into just one for our normal status query.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This takes a bit more work to compute, but since we cache all of this anyway
it isn't too big of deal. Using average delay instead of last delay will be
a bit more fair on mirrors that have odd syncing schedules, as well as
exposing those that only sync once a day. Also fix an issue that will arise
with cutoff_time being calculated once, and adjust mirror score to treat
hours delay as a float rather than an integer.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
By using the mirror score we calculate, we can sort the mirrors in the
generated mirrorlist for people.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This should get this to the point where it is releasable to the general
public for their use and pleasure. Still not sure on how often the check
should be run, and we probably want to incorporate this mined data into some
other things like the mirror list generator.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This does the actual work of going out and checking the mirror status. In
short, it polls every active mirror URL for the 'lastsync' file and then
records the appropriate details. These include the contents of that file,
how long the total time to retrieve took, and any errors encountered.
In order to finish up a bit faster, we spawn several threads to do the
actual work. This parallelization allows the whole check process to take
around 30 seconds rather than several minutes.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
