summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authoreric <eric>2004-06-13 18:15:49 +0000
committereric <eric>2004-06-13 18:15:49 +0000
commit569216db792249ba470a2371fc3ad72c79a567cd (patch)
treef6889d321013508dc647f1febce953820a5d09d8
parent36d0b810ad8de99337e3993de84087ae5878b2ec (diff)
initial import
-rw-r--r--README.txt107
-rw-r--r--support/README.txt6
-rw-r--r--support/schema/aur-schema.sql139
-rw-r--r--tupkg/README.txt99
-rwxr-xr-xtupkg/client/tupkg19
-rwxr-xr-xtupkg/server/tupkgs22
-rw-r--r--web/README.txt94
7 files changed, 486 insertions, 0 deletions
diff --git a/README.txt b/README.txt
new file mode 100644
index 0000000..12f64c7
--- /dev/null
+++ b/README.txt
@@ -0,0 +1,107 @@
+This is the finalized draft of the project requirements for the
+new Arch package submittal process. AUR (Arch User-community Repo).
+The sub-directories contain more specific implementation notes
+for each component of the project.
+
+
+Requirements:
+-------------
+1) User accounts (users, TUs)
+ - Create account. (email address required)
+ - Update account (change password/email address)
+ - Log in/out
+
+2) Search for packages (public)
+ - needs knowledge of ALL pkgs (OfficalRepos/AUR/Unsupported). This
+ should be easy enough if this site lives on the same machine as
+ the current package database (dragon?), or is allowed to query
+ the db.
+ - Display official repo (current/extra) a package lives in.
+
+3) Manual voting (requires user acct)
+ - reset/clear all votes (for starting over, this can be added later
+ if there is any demand for it)
+
+4) Package Management
+ - A package can be submitted by anyone (as long as they create
+ an account with a valid email address on the web site). From
+ there, a TU inspects the package and works with the submitter
+ to resolve any bugs/issues. Once approved, the TU can place the
+ package in the AUR. From there, if the package is popular enough
+ (or deemed necessary), a developer can move the package from the
+ AUR to extra/current/etc. A developer can also downgrade a
+ package from extra/current/etc to the AUR.
+ - The person that uploaded the new package can make changes to
+ it before it has been added to the AUR.
+ - TUs need to be able to purge packages in "Unsupported" if the
+ package is no longer maintained and there is no interest in
+ keeping it.
+ - Packages in the AUR/Unsupported need some sort of 'flag out of
+ date' support.
+ - Interested users can download the package from "Unsupported"
+ and build/install it by hand.
+ - Provide a separate installation of flyspray for tracking bugs
+ for packages living in the AUR. All bugs should be resolved
+ in either flyspray (AUR/official) prior to a package being
+ promoted/demoted.
+
+5) Reports
+ - package popularity by number of votes
+
+6) Wiki Docs (UID/GID db, provides db, irc nicks/names TUs/devs)
+ - Move the appropriate dev wiki pages to the new system's
+ wiki area. The devs will just need to consult the UID/GID
+ list from the new system site rather than our own wiki.
+
+7) Submitting 'new' packages by users. Initially start with
+ a simple web upload page where users submit a tgz containing
+ the PKGBUILD, scriptlets, patches, etc. The script will
+ unpack the tgz in an appropriate location and update the
+ backend database to 'register' the pacakge.
+
+8) TU package management
+ - A TU adopts a package from "Unsupported" and that shows users
+ and other TUs that the package is being reviewed.
+ - When the TU is ready to move the package to the AUR, they
+ use a custom utility/script that allows them to upload the
+ pkg.tar.gz (web uploads are inadequate for this process).
+ The upload utility/script has a server counterpart that
+ performs TU authentication and updates the database.
+ - A cronjob on the server can handle the new AUR package,
+ update the database, and rebuild the AUR sync db, and send
+ email notices to the TU about errors if any occur.
+ - The TUs should also be able to demote a package from the
+ AUR via the web interface.
+ - TUs will use cvs/svn interface (just like devs) to pull
+ down the PKGBUILD tree. This tree should reflect the same
+ layout as extra for easier package migration. They make
+ changes to their local copy, test, and then commit. They
+ use the xfer utility to upload the binary package to the
+ server. No shell access is required.
+
+
+Automated Voting Tool (similar to ArchStats client)
+=====================
+
+Requirements:
+-------------
+ 1) Name of tool is 'pkgvote'
+
+ 2) Requires registered account on web - email address not required
+
+ 3) Casts 'yes' votes for all installed packages (except itself?)
+
+Implementation:
+---------------
+ A statically compiled C program that gathers the list of installed
+ packages and casts the vote to the web site. Very similar to the
+ way that ArchStats works now. When making the HTTP Post, it adds
+ a custom HEADER that the PHP script(s) can detect to know that it
+ is receiving a vote from a 'pkgvote' client. If the PHP script
+ does not see the special HEADER, it assumes it is a web browser
+ and gives the user HTML output.
+
+ Once installed, the user edits the config file and supplies their
+ username/password. If no username/password exists in the config
+ file when it starts, it spits out an error message and exits.
+
diff --git a/support/README.txt b/support/README.txt
new file mode 100644
index 0000000..8b01ccb
--- /dev/null
+++ b/support/README.txt
@@ -0,0 +1,6 @@
+The schema dir contains the AUR database schema file.
+
+The scripts dir will contain any supporting scripts for the AUR such
+as the script that handles moving newly uploaded packages via tupkg
+to the AUR.
+
diff --git a/support/schema/aur-schema.sql b/support/schema/aur-schema.sql
new file mode 100644
index 0000000..e147c47
--- /dev/null
+++ b/support/schema/aur-schema.sql
@@ -0,0 +1,139 @@
+-- The MySQL database layout for the AUR. Certain data
+-- is also included such as AccountTypes, PackageLocations, etc.
+--
+
+-- Define the Account Types for the AUR.
+--
+CREATE TABLE AccountTypes (
+ ID UNSIGNED TINYINT NOT NULL AUTO_INCREMENT,
+ AccountType char(32) NOT NULL DEFAULT '',
+ PRIMARY KEY (ID)
+);
+INSERT INTO TABLE (ID, AccountType) VALUES (1, 'User');
+INSERT INTO TABLE (ID, AccountType) VALUES (2, 'Trusted User');
+INSERT INTO TABLE (ID, AccountType) VALUES (3, 'Developer');
+
+
+-- User information for each user regardless of type.
+--
+CREATE TABLE Users (
+ ID UNSIGNED INTEGER NOT NULL AUTO_INCREMENT,
+ AccountTypeID UNSIGNED TINYINT NOT NULL DEFAULT 1,
+ Suspended UNSIGNED TINYINT NOT NULL DEFAULT 0,
+ Email CHAR(64) NOT NULL,
+ Passwd CHAR(32) NOT NULL,
+ RealName CHAR(64) NOT NULL DEFAULT '',
+ IRCNick CHAR(32) NOT NULL DEFAULT '',
+ LastVoted UNSIGNED BIGINT NOT NULL DEFAULT 0,
+ NewPkgNotify UNSIGNED TINYINT NOT NULL DEFAULT 0,
+ PRIMARY KEY (ID),
+ UNIQUE INDEX Emailx (Email),
+ INDEX AccountTypeIDx (AccountTypeID),
+ INDEX NewPkgNotifyx (NewPkgNotify),
+ FOREIGN KEY AccountTypeIDr REFERENCES AccountTypes (ID)
+);
+-- A default developer account for testing purposes
+INSERT INTO Users (ID, AccountTypeID, Email, Passwd) VALUES (
+ 1, 3, 'root@localhost', 'changeme');
+
+
+-- Track Users logging in/out of AUR web site.
+--
+CREATE TABLE Sessions (
+ UsersID UNSIGNED INTEGER NOT NULL,
+ SessionID CHAR(32) NOT NULL,
+ LastUpdateTS UNSIGNED BIGINT NOT NULL,
+ FOREIGN KEY UsersIDr REFERENCES Users (ID)
+);
+
+
+-- Categories for grouping packages when they reside in
+-- Unsupported or the AUR - based on the categories defined
+-- in 'extra'.
+--
+CREATE TABLE PackageCategories (
+ ID UNSIGNED TINYINT NOT NULL AUTO_INCREMENT,
+ Category CHAR(32) NOT NULL,
+ PRIMARY KEY (ID)
+);
+INSERT INTO PackageCategories (Category) VALUES ('daemons');
+INSERT INTO PackageCategories (Category) VALUES ('devel');
+INSERT INTO PackageCategories (Category) VALUES ('editors');
+INSERT INTO PackageCategories (Category) VALUES ('emulators');
+INSERT INTO PackageCategories (Category) VALUES ('games');
+INSERT INTO PackageCategories (Category) VALUES ('gnome');
+INSERT INTO PackageCategories (Category) VALUES ('i18n');
+INSERT INTO PackageCategories (Category) VALUES ('kde');
+INSERT INTO PackageCategories (Category) VALUES ('lib');
+INSERT INTO PackageCategories (Category) VALUES ('modules');
+INSERT INTO PackageCategories (Category) VALUES ('multimedia');
+INSERT INTO PackageCategories (Category) VALUES ('network');
+INSERT INTO PackageCategories (Category) VALUES ('office');
+INSERT INTO PackageCategories (Category) VALUES ('science');
+INSERT INTO PackageCategories (Category) VALUES ('system');
+INSERT INTO PackageCategories (Category) VALUES ('x11');
+INSERT INTO PackageCategories (Category) VALUES ('xfce');
+
+
+-- The various repositories that a package could live in.
+--
+CREATE TABLE PackageLocations (
+ ID UNSIGNED TINYINT NOT NULL AUTO_INCREMENT,
+ Location CHAR(32) NOT NULL,
+ PRIMARY KEY (ID)
+);
+INSERT INTO PackageLocations (ID, Location) VALUES (1, 'Unsupported');
+INSERT INTO PackageLocations (ID, Location) VALUES (2, 'AUR');
+INSERT INTO PackageLocations (ID, Location) VALUES (3, 'Current');
+INSERT INTO PackageLocations (ID, Location) VALUES (4, 'Extra');
+INSERT INTO PackageLocations (ID, Location) VALUES (5, 'Unstable');
+
+
+-- Information about the actual packages
+--
+CREATE TABLE Packages (
+ ID UNSIGNED INTEGER NOT NULL AUTO_INCREMENT,
+ Name CHAR(32) NOT NULL,
+ Version CHAR(32) NOT NULL DEFAULT '',
+ CategoryID UNSIGNED TINYINT NOT NULL,
+ Description CHAR(128) NOT NULL DEFAULT "An Arch Package",
+ URL CHAR(256) NOT NULL DEFAULT "http://www.archlinux.org",
+ Source CHAR(256) NOT NULL DEFAULT "/dev/null",
+ LocationID UNSIGNED TINYINT NOT NULL,
+ OutOfDate UNSIGNED TINYINT DEFAULT 0,
+ SubmittedTS UNSIGNED BIGINT NOT NULL,
+ SubmitterUID UNSIGNED INTEGER NOT NULL DEFAULT 0,
+ MaintainerUID UNSIGNED INTEGER NOT NULL DEFAULT 0,
+ PRIMARY KEY (ID),
+ UNIQUE INDEX Namex (Name),
+ INDEX CategoryIDx (CategoryID),
+ INDEX LocationIDx (LocationID),
+ INDEX OutOfDatex (OutOfDate),
+ INDEX SubmitterUIDx (SubmitterUID),
+ INDEX MaintainerUIDx (MaintainerUID),
+ FOREIGN KEY CategoryIDr REFERENCES PackageCategories (ID),
+ FOREIGN KEY LocationIDr REFERENCES PackageLocations (ID)
+ FOREIGN KEY SubmitterUIDr REFERENCES Users (ID)
+ FOREIGN KEY MaintainerUIDr REFERENCES Users (ID)
+);
+
+
+-- Track votes for packages
+--
+CREATE TABLE PackageVotes (
+ UsersID UNSIGNED INTEGER NOT NULL,
+ PackageID UNSIGNED INTEGER NOT NULL,
+ PRIMARY KEY (ID),
+ FOREIGN KEY UsersIDx REFERENCES Users (ID),
+ FOREIGN KEY PackageIDx REFERENCES Packages (ID)
+);
+
+
+-- The individual files and their file system location.
+--
+CREATE TABLE PackageContents (
+ PackageID UNSIGNED INTEGER NOT NULL,
+ Path CHAR(256) NOT NULL,
+ INDEX PackageIDx (PackageID)
+);
+
diff --git a/tupkg/README.txt b/tupkg/README.txt
new file mode 100644
index 0000000..47facc4
--- /dev/null
+++ b/tupkg/README.txt
@@ -0,0 +1,99 @@
+TU Packaging Tools (tupkg)
+--------------------------
+- client side (python for proof of concept, later re-write to C?)
+ The main purpose of this tool is to upload the compiled
+ pkg.tar.gz to the server. It can (should?) do some verification
+ on the package prior to uploading to the server. It will have
+ a config file to store run-time information such as username
+ (email), password, and server name.
+
+- server side (python for proof of concept, later re-write to C?)
+ The server side will handle incoming connections from its client
+ side counterpart. The server should bind to port 80 (maybe a
+ vhost such as tupkg.archlinux.org?) so that firewalls won't be
+ an issue. The server verifies the client authentication data,
+ and then accepts the package(s). If port 80 is not available,
+ perhaps 443, or are there other 'standard' ports that usually
+ do not get filtered?
+
+ I think the server should be multithreaded to handle simultaneous
+ uploads rather than queue up requests. The download should be
+ stored in a temp directory based on the username to prevent
+ directory, filename clashes.
+
+ Once the package(s) is uploaded, the server can either kick off
+ a gensync, or we can write a separate script to call gensync once
+ or twice a day. My preference would be a separate script to call
+ gensync (like the *NIX philosophy of one tool per task).
+
+- protocol (c: => client, s: => server)
+ Whenever the client/server exchange a message, it is always
+ preceeded by two-bytes representing the following message's
+ length. For example, when the client connects, it will send:
+
+ 0x0028username=bfinch@example.net&password=B0b
+
+ 0x0028 is the 40 byte strlen of the message in two-bytes. The
+ client and server always read two-bytes from the socket, and
+ then know how much data is coming and can read that amount of
+ bytes from the socket.
+
+ ==> authentication
+ c: username=emailaddy&password=mypassword
+ s: result=PASS|FAIL
+
+ NOTE: We can add encryption easily enough with the python
+ version using the socket.ssl method.
+
+ ==> uploading package data
+ if PASS:
+
+ c: numpkgs=2&name1=p1.pkg.tar.gz&size1=123&md5sum1=abcd\
+ name2=p2.pkg.tar.gz&size2=3&md5sum2=def1
+ s: numpkgs=2&name1=p1.pkg.tar.gz&size1=119&\
+ name2=p2.pkg.tar.gz&size2=0 (*)
+
+ (*) NOTE: The server will reply back to the client how many
+ packages it has already received and its local file size.
+ This way, the client can resume an upload. In the example
+ above, the server still needs the last four (123-119) bytes
+ for the first package, and that it has no part of the
+ second package. The client would then begin sending the
+ last four bytes from the first package (p1.pkg.tar.gz) and
+ then follow it with the full second package (p2.pkg.tar.gz).
+ The data would be sent as a continuous chunk of data. The
+ server will then need to track which bytes belong to which
+ package.
+
+ else FAIL:
+ c: -spits out error message on stderr to user-
+
+
+ ==> after upload completes
+ The server should verify the integrity of the uploaded packages
+ by doing an md5sum on each and sending the info back to the client
+ for comparison. After sending the message, the server waits for
+ the 'ack' message from the client and then closes the connection.
+
+ s: np=2&m1=PASS&m2=FAIL
+ c: ack
+
+ The client replies with the 'ack' and then closes its connection
+ to the server. It then reports the PASS/FAIL status of each
+ package's upload attempt.
+
+ NOTE: If the upload fails (client connection dies), the server
+ keeps any data it has received in order to support resuming an
+ upload. However, if the client uploads all data, and the server
+ successully reads all data and the final MD5 fails, the server
+ deletes the failed package.
+
+
+Terms/definitions:
+======================
+TU - No change (trusted by the community, if anyone asks what trust
+ means)
+TUR - renamed to Arch User-community Repo (AUR) (so we can use -u for
+ versions)
+Incoming - renamed to "Unsupported"
+
diff --git a/tupkg/client/tupkg b/tupkg/client/tupkg
new file mode 100755
index 0000000..ab42b43
--- /dev/null
+++ b/tupkg/client/tupkg
@@ -0,0 +1,19 @@
+#!/usr/bin/python -O
+#
+# Description:
+# ------------
+# This is the client-side portion of the Trusted User package
+# manager. The TUs will use this program to upload packages into
+# the AUR. For more information, see the ../README.txt file.
+#
+# Python Indentation:
+# -------------------
+# Use tabs not spaces. If you use vim, the following comment will
+# configure it to use tabs.
+# vim: ts=2 sw=2 noet ft=python
+#
+
+
+
+# TODO write the code
+#
diff --git a/tupkg/server/tupkgs b/tupkg/server/tupkgs
new file mode 100755
index 0000000..2c410a5
--- /dev/null
+++ b/tupkg/server/tupkgs
@@ -0,0 +1,22 @@
+#!/usr/bin/python -O
+#
+# Description:
+# ------------
+# This is the server-side portion of the Trusted User package
+# manager. This program will receive uploads from its client-side
+# couterpart, tupkg. Once a package is received and verified, it
+# is placed in a specified temporary incoming directory where
+# a separate script will handle migrating it to the AUR. For
+# more information, see the ../README.txt file.
+#
+# Python Indentation:
+# -------------------
+# Use tabs not spaces. If you use vim, the following comment will
+# configure it to use tabs.
+# vim: ts=2 sw=2 noet ft=python
+#
+
+
+
+# TODO write the code
+#
diff --git a/web/README.txt b/web/README.txt
new file mode 100644
index 0000000..5795705
--- /dev/null
+++ b/web/README.txt
@@ -0,0 +1,94 @@
+Web Interface:
+==============
+
+Directory Layout:
+-----------------
+./html - DocumentRoot for AUR, where the PHP scripts live.
+./html/css - CSS stylesheets
+./html/images - Any AUR images live here.
+./lib - Supporting PHP include files. Access denied to Apache.
+
+
+Scripts:
+--------
+- lib/funcs.inc
+ This is where we can stick functions that can be shared
+ between the various scripts. Also a good place to put the
+ MySQL authentication variables since it should live outside
+ the DocumentRoot.
+
+- html/login.php (probably index.php)
+ PHP script to handle logging users into the AUR web site. It
+ authenticates using the email address and a password against
+ the Users table. Once authenticated, a session id is generated
+ and stored in the Sessions table and sent as a cookie to the
+ user's browser.
+
+- html/logout.php
+ PHP script to logout. It clears the session id from the
+ Sessions table and unsets the cookie.
+
+- html/account.php
+ PHP script to handle registering for a new account. It prompts
+ the visitor for account information: Email, password, real name,
+ irc nick. The info is recorded in the Users table. Perhaps later,
+ we can add a preferences field that allows the user to request to
+ be notified when new packages are submitted so that they can cast
+ votes for them?
+
+ If a TU is logged into the system, they can edit accounts and set
+ the account type (regular user or TU). If a Dev is logged in, they
+ can also set the account type to Dev. TUs and Devs are able to
+ delete accounts. If an account is deleted, all "Unsupported"
+ packages are orphaned (the Users.ID field in the UnsupportedPackages
+ table is set to Null).
+
+- html/pkgsearch.php
+ PHP script to search the package database. It should support
+ searching by location ("unsupported", "AUR", "extra"), name,
+ category, maintainer, popularity, etc. It should resemble the
+ packages.php script on archlinux.org. A checkbox should be
+ included next to each package to allow users to flag a package
+ out of date.
+
+- html/pkgvote.php
+ The PHP script that handles voting for packages. It works like
+ the search script above to provide a list of packages (maybe by
+ location only?) with a checkbox for the user to register their
+ 'yes' vote. It should probably only list 50-ish packages per page
+ and allow the user to vote a page at a time. Each page contains a
+ 'Continue' button to advance to the next list of packages. At the
+ final page, a summary is presented with a 'Cast Vote' button. Once
+ the vote is cast, the PackageVotes table is first cleared for that
+ User and then all new entries are added for the new vote (this will
+ be easier than trying to figure out if the vote has changed for a
+ particular package).
+
+- html/pkgmgmnt.php
+ The PHP script for managing packages. It provides a list of
+ packages under the management of the logged in user (normal or
+ TU). The user can edit the package information stored in the
+ database such as the description, url, category, and location
+ (a TU can move it to/from "unsupported" and the "AUR"). This
+ is where TUs can adopt packages that are in the "unsupported"
+ area.
+
+- html/pkgsubmit.php
+ This is the PHP script that allows users to upload a new package.
+ The package format will be a tgz containing the PKGBUILD,
+ scriptlets, and patches necessary to build the package from
+ source. Initially, the user submitting the package can select
+ its category (network, devel, etc) but that can be modified
+ later by the adopting TU. The script makes appropriate entries
+ into the database (and perhaps notifies interested users of the
+ new package - see question above).
+
+
+New terms/definitions:
+======================
+TU - No change (trusted by the community, if anyone asks what trust
+ means)
+TUR - renamed to Arch User-community Repo (AUR) (so we can use -u for
+ versions)
+Incoming - renamed to "Unsupported"
+