| Age | Commit message (Collapse) | Author |
|---|
|
We can use the easier transaction.commit_on_success() decorator if we be
sure to explicitly mark the transaction dirty. This fixes the issue
where a raised exception in this code called neither commit nor
rollback.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Rather than delegating to Django and batch deletion by ID, force issuing
of a single delete query to clear out all existing file objects when
necessary. This should speed up the deletion and update of packages with
a lot of files by a non-trivial amount.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
I needed this today to get the application working from scratch on
another host. Probably not all there yet, but we'll see how far it gets
us.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This allow importing signatures from a provided gpg keyring, such as
that produced by the generate_keyring management command that already
exists. These will eventually be used for producing stats involving
developer signing keys and their certification by master keys.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Way to fail at refactoring, Dan.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This removes a bunch of the conditional logic at a slight cost of some
code duplication. However, the methods and madness is now much easier to
follow.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This was broken after the select for update changes. We really should
split the whole filesonly update into another method instead of the
current shotgun approach with conditionals everywhere.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This adds a bunch of transaction magic and SELECT FOR UPDATE stuff to
reporead to cope with the now-concurrent runs of reporead we get when
invoked from our inotify-based updater. The collision occurs with 'any'
architecture packages as both repo databases contain the new version,
and the updates occur at exactly the same time.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This prevents the reporead job from taking over time from more important
processes; this is not a rush task.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This prevents an otherwise idle connection from sitting around and being
totally useless.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This prevents memory usage from ballooning to absolutely huge values,
such as when multiple threads kick off at the same time. The bulk of our
memory allocation obviously comes in these threads and not the main
threads, so being able to isolate them in processes helps a lot.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This is the new on-the-fly updates hotness. Rather than continue to
schedule reporead to run once an hour in cron or however else you ran
it, this command can be run once and left running, and will
automagically pick up on any database file changes and run an import.
It operates on the files databases only; this will keep both the
packages and files always in sync and remove the delay in updating,
especially helpful for new testing packages.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Ensure we can accept either a Arch object or an architecture name when
passed to read_repo() by moving the validation there and being a bit
more careful about typechecking and object lookup.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Fuck you too, Django.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This makes them totally unusable for any real purpose down the road.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This grabs all the PGP keys from the developer profiles and adds them to
the keyrings. Obviously we may want to do more in the future such as
filter by groups, active status, etc. but this is just a first
iteration.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
And add eventual display code for it to the details template, but don't
show it yet as no packages will have it.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This allows quick resolution of all unmatched packages, especially after
tweaking the way find_user works.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This moves the cache inside an instance. Also add a few more tests.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This could be handy elsewhere as well, and it is loosely coupled to
anything else in reporead.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
* Parse builddate when reading from repo database file
* Use defaultdict where it comes in handy
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
The real reason I originally added transactions to this code was to
prevent half-updates; e.g. a package gets in without the matching
depends values. We can safely commit between packages and resume
processing the database at a later time.
Take advantage of this fact and commit every so often in batch fashion
if we have a lot of updates piling up. In the case of updating the files
DB, this can really cut down on the need to hold open a long-running,
statement heavy transaction and get the information public faster.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Now that we aren't seeing odd segfaults and hung tasks, we can remove
the traceback stuff from the scripts. Also use the 'io' module only, it
has been long enough.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Rather than the twisted mix of local times and UTC times we currently have.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Ensure all our multivalued attributes already exist on the object
beforehand, and add some special sauce to handle the difference between
a package without files and a database without files entries.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This will come in more handy with our new models, but we can adapt groups
and licenses to use it first.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This comes with pacman 3.5, replacing the old "force" PKGBUILD option.
We parse it and store it for now, but don't display it anywhere just
yet. Also update a few queries relying on version differences in any of
the multiple parts.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
|
|
When importing over a million files, it makes sense to take the slightly
faster route and call the PackageFile() constructor directly rather than
going through the related manager's create method.
We can also get huge performance improvements, especially with files
databases, by using the 'io' rather than 'codecs' module. The former is
now implemented in C in 2.7 and results in a no-work import (so
measuring only the DB read speed) of extra.files.tar.gz from ~30 seconds
to ~5 seconds.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
Signed-off-by: Dan McGee <dan@archlinux.org>
|
|
This allows us to store multiple licenses per package in a more elegant
fashion, and will later allow us to search and filter on this information.
Signed-off-by: Dan McGee <dan@archlinux.org>
|
