From db125088520d739b19607057e8635a9ddb28fe6e Mon Sep 17 00:00:00 2001 From: Luke Shumaker Date: Fri, 24 May 2013 13:50:34 -0400 Subject: clean up doc/; normalize filenames, clean up createworkdir(.md) --- doc/createworkdir | 62 ----------------------- doc/createworkdir.md | 41 ++++++++++++++++ doc/fullpkg | 7 --- doc/fullpkg.md | 7 +++ doc/treepkg.markdown | 130 ------------------------------------------------- doc/treepkg.md | 130 +++++++++++++++++++++++++++++++++++++++++++++++++ doc/workflows.markdown | 60 ----------------------- doc/workflows.md | 60 +++++++++++++++++++++++ 8 files changed, 238 insertions(+), 259 deletions(-) delete mode 100644 doc/createworkdir create mode 100644 doc/createworkdir.md delete mode 100644 doc/fullpkg create mode 100644 doc/fullpkg.md delete mode 100644 doc/treepkg.markdown create mode 100644 doc/treepkg.md delete mode 100644 doc/workflows.markdown create mode 100644 doc/workflows.md diff --git a/doc/createworkdir b/doc/createworkdir deleted file mode 100644 index d74198a..0000000 --- a/doc/createworkdir +++ /dev/null @@ -1,62 +0,0 @@ -== CreateWorkDir - -This script recreates a proper dir tree for packaging. It's aim is to help -you be organized with the work you do as a packager, and stablish a -certain standard for packages' ubication, so you don't have to loose much -time with them. Just package and upload! - -It will create a dir tree like this: -workdir/ - abslibre/ - .git/ - libre/ - libre-testing/ - staging/ - libre/ - libre-testing/ - repos/ - libre/ - i686/ - x86_64/ - libre-testing/ - i686/ - x86_64/ - -*Related Variables* - - WORKDIR - - -=== staging/ - -This directory contains one dir for each repository, where the resulting -packages are in moved for syncing against the main repository on -Parabola's server. This dir is architecture independent. - -*Related Variables* - - REPOS - - -=== abslibre/ - -This is the git repo for Parabola's PKGBUILDs. Here you can find the ABS -tree for our packages, and also where you'll have to put new ones for -commit. - -(You'll need push access to Parabola's main server, but pulling is -public.) - -*Related Variables* - - ABSLIBREGIT - - -=== repos/ - -Contains the repo dir structure. Probably not useful for uploading -packages, but it will be for creating new repos for public access. - -It contains a dir for every repository, and inside them, a dir for every -supported architecture. - -*Related Variables* - - REPOS - - ARCHES diff --git a/doc/createworkdir.md b/doc/createworkdir.md new file mode 100644 index 0000000..2539920 --- /dev/null +++ b/doc/createworkdir.md @@ -0,0 +1,41 @@ +# CreateWorkDir + +This script recreates a proper directory tree for packaging. It's aim is to +help you be organized with the work you do as a packager, and establish a +certain standard for packages' publication, so you don't have to loose much +time with them. Just package and upload! + +It will create a directory tree like this: + + $WORKDIR/ + ├── abslibre/ + │ ├── .git/ + │ ├── libre/ + │ └── libre-testing/ + └── staging/ + ├── libre/ + └── libre-testing/ + +*Related Variables* + - WORKDIR + +## staging/ + +This directory contains one directory for each repository, where the +resulting packages are in moved for syncing against the main repository on +Parabola's server. This directory is architecture independent. + +*Related Variables* + - REPOS + +## abslibre/ + +This is the git repo for Parabola's PKGBUILDs. Here you can find the ABS +tree for our packages, and also where you'll have to put new ones for +commit. + +(You'll need push access to Parabola's main server, but pulling is +public.) + +*Related Variables* + - ABSLIBREGIT diff --git a/doc/fullpkg b/doc/fullpkg deleted file mode 100644 index 5645fae..0000000 --- a/doc/fullpkg +++ /dev/null @@ -1,7 +0,0 @@ -# FullPKG -FullPKG is a tool to build a package from ABS, find all needed dependencies -along the way and build them too if necessary. - -## Before running fullpkg -Update PKGBUILD path cache by running `toru -u ` on your ABS root. Once -for each repo you'll be using. diff --git a/doc/fullpkg.md b/doc/fullpkg.md new file mode 100644 index 0000000..5645fae --- /dev/null +++ b/doc/fullpkg.md @@ -0,0 +1,7 @@ +# FullPKG +FullPKG is a tool to build a package from ABS, find all needed dependencies +along the way and build them too if necessary. + +## Before running fullpkg +Update PKGBUILD path cache by running `toru -u ` on your ABS root. Once +for each repo you'll be using. diff --git a/doc/treepkg.markdown b/doc/treepkg.markdown deleted file mode 100644 index 7f7ece1..0000000 --- a/doc/treepkg.markdown +++ /dev/null @@ -1,130 +0,0 @@ -# TreePKG - -`treepkg` is a tool for recursively building packages from an ABS tree. It has -been tested while building packages for the mips64el port and it has proven to -be useful. - -It was written having in mind the experience of `fullpkg`, which converted to -`fullpkg-ng` and then splitted into `fullpkg-build` and `fullpkg-find`. It's -aim is to simplify algorithms implemented on the fullpkg family, while solving -some design issues that made fullpkg miss some packages sometimes. - - -## Requirements - -`treepkg` needs the help of `toru-path` for "indexing" an ABS tree. `toru-path` -stores a tokyocabinet database of "pkgname" => "path" pairs, where pkgname is -replaced by the "pkgbase", "pkgname", and "provides" fields of a PKGBUILD, -followed by the path of the current PKGBUILD. - -This information is then used by `treepkg` to know where to find the PKGBUILD -of a package. The fullpkg family needed to guess this by traversing the full -ABS tree, and therefore it was unable to find pkgnames that differ from -pkgbase. - -So, to use `treepkg` you need to run `toru-path` after the ABS tree update. - -> Split PKGBUILDs make it difficult to extract metadata if it's stored inside -> package() functions. This will happen with the provides field and `treepkg` -> won't find that linux-libre-headers provides linux-headers, for instance. - -## How does it work - -`treepkg` must be run from the path of the PKGBUILD you're going to build (this -may change over time). This will be DEPTH=0 and it will create a temporary -directory to work on in the format /tmp/pkgbase-treepkg-random-string. Inside -this directory, it will copy the needed PKGBUILDs prefixed with their depth -number. The first package will always be copied to 000\_pkgbase. - -From then on, treepkg sources the PKGBUILD and runs itself over all pkgnames on -the depends and makedepends array, only if it detects their versions aren't -already built or deprecated by newer ones, using the `is_built` utility. - -While processing this info, it will increase the depth of the packages. It'll -also write a CSV file with the knowledge it acquires from the ABS tree (useful -for debugging). This file is called BUILDORDER. - -When this process ends (no more needed dependencies are found), the temporary -work dir is traversed in inverse order (from higher depths to 0) running the -FULLBUILDCMD from libretools.conf and then the HOOKLOCALRELEASE variable, which -*must* provide a way to `repo-add` the packages to an available repository, so -the next build will find and install the newer version using pacman. - -For instance, having this as the first pacman repository (on /etc/pacman.conf): - - [stage3] - Server = /var/cache/pacman/pkg - -And this on /etc/makepkg.conf: - - PKGDEST=/var/cache/pacman/pkg - -Your HOOKLOCALRELEASE script should look like this: - - # Get needed vars - source /etc/makepkg.conf - source /etc/libretools.conf - source PKGBUILD - - unset build package check - - fullver=$(full_version ${epoch:-0} ${pkgver} ${pkgrel}) - pkgs=() - - # Generate canonical package paths - msg "Adding packages to [stage3]..." - for name in ${pkgname[@]}; do - msg2 "${name} ${fullver}" - pkgs+=("${PKGDEST}/${name}-${fullver}-*.pkg.tar.*") - done - - # Add the packages to a local - repo-add ${PKGDEST}/stage3.db.tar.gz ${pkgs[@]} - - # Stage the packages for later releasing - librestage $1 - -> Note the first HOOKLOCALRELEASE argument is the remote repository name (core, -> extra, etc.) - -There's a special case when a dependency depends on another that was put on -a depth level lower than itself. In this case the build order will be wrongly -assumed and you may end up with broken packages. - -To explain it with an example: - - ghostscript (0) - fontconfig (1) - \ cups (1) - fontconfig (ignored) - -The second time fontconfig appears, it will be ignored. In this case cups will -build against an fontconfig version that will be outdated by the fontconfig -version at depth 1. In this cases, `treepkg` will detect it cached the -dependency on a lower depth, and will "bury" it to a depth higher than the -current one. Thus this will become the build path: - - ghostscript (0) - fontconfig (buried) - \ cups (1) - fontconfig (2) - -> Note: currently, `treepkg` doesn't perform recursive burying, so if you hit -> a really long build tree with some circular dependencies you may find -> packages buried several times and queued to build before their actuals deps. - -## Tips - -`treepkg` accepts two arguments: 1) the temporary work dir and 2) the next -depth level it should use (if current equals 0, it'll pass 1). You don't need -to pass this arguments when running it manually, they're used internally to -automatically construct the build path. - -But if a build failed, `treepkg` will cancel itself immediately informing you -where the leftovers files were left. If you pass this path to `treepkg` as the -first argument, it will resume the build, skipping to the last package being -packaged. - -You can take the opportunity given by this to modify the build path or the -PKGBUILDs, without having to run `treepkg` twice. For instance you can remove -a package from the build order, or move it manually, or update the PKGBUILD -that made `treepkg` fail in the first place. You can force a skipped package -(after building it manually) by using `touch built_ok` on the PKGBUILD dir. - -You don't probably want to mess with the second argument though. diff --git a/doc/treepkg.md b/doc/treepkg.md new file mode 100644 index 0000000..7f7ece1 --- /dev/null +++ b/doc/treepkg.md @@ -0,0 +1,130 @@ +# TreePKG + +`treepkg` is a tool for recursively building packages from an ABS tree. It has +been tested while building packages for the mips64el port and it has proven to +be useful. + +It was written having in mind the experience of `fullpkg`, which converted to +`fullpkg-ng` and then splitted into `fullpkg-build` and `fullpkg-find`. It's +aim is to simplify algorithms implemented on the fullpkg family, while solving +some design issues that made fullpkg miss some packages sometimes. + + +## Requirements + +`treepkg` needs the help of `toru-path` for "indexing" an ABS tree. `toru-path` +stores a tokyocabinet database of "pkgname" => "path" pairs, where pkgname is +replaced by the "pkgbase", "pkgname", and "provides" fields of a PKGBUILD, +followed by the path of the current PKGBUILD. + +This information is then used by `treepkg` to know where to find the PKGBUILD +of a package. The fullpkg family needed to guess this by traversing the full +ABS tree, and therefore it was unable to find pkgnames that differ from +pkgbase. + +So, to use `treepkg` you need to run `toru-path` after the ABS tree update. + +> Split PKGBUILDs make it difficult to extract metadata if it's stored inside +> package() functions. This will happen with the provides field and `treepkg` +> won't find that linux-libre-headers provides linux-headers, for instance. + +## How does it work + +`treepkg` must be run from the path of the PKGBUILD you're going to build (this +may change over time). This will be DEPTH=0 and it will create a temporary +directory to work on in the format /tmp/pkgbase-treepkg-random-string. Inside +this directory, it will copy the needed PKGBUILDs prefixed with their depth +number. The first package will always be copied to 000\_pkgbase. + +From then on, treepkg sources the PKGBUILD and runs itself over all pkgnames on +the depends and makedepends array, only if it detects their versions aren't +already built or deprecated by newer ones, using the `is_built` utility. + +While processing this info, it will increase the depth of the packages. It'll +also write a CSV file with the knowledge it acquires from the ABS tree (useful +for debugging). This file is called BUILDORDER. + +When this process ends (no more needed dependencies are found), the temporary +work dir is traversed in inverse order (from higher depths to 0) running the +FULLBUILDCMD from libretools.conf and then the HOOKLOCALRELEASE variable, which +*must* provide a way to `repo-add` the packages to an available repository, so +the next build will find and install the newer version using pacman. + +For instance, having this as the first pacman repository (on /etc/pacman.conf): + + [stage3] + Server = /var/cache/pacman/pkg + +And this on /etc/makepkg.conf: + + PKGDEST=/var/cache/pacman/pkg + +Your HOOKLOCALRELEASE script should look like this: + + # Get needed vars + source /etc/makepkg.conf + source /etc/libretools.conf + source PKGBUILD + + unset build package check + + fullver=$(full_version ${epoch:-0} ${pkgver} ${pkgrel}) + pkgs=() + + # Generate canonical package paths + msg "Adding packages to [stage3]..." + for name in ${pkgname[@]}; do + msg2 "${name} ${fullver}" + pkgs+=("${PKGDEST}/${name}-${fullver}-*.pkg.tar.*") + done + + # Add the packages to a local + repo-add ${PKGDEST}/stage3.db.tar.gz ${pkgs[@]} + + # Stage the packages for later releasing + librestage $1 + +> Note the first HOOKLOCALRELEASE argument is the remote repository name (core, +> extra, etc.) + +There's a special case when a dependency depends on another that was put on +a depth level lower than itself. In this case the build order will be wrongly +assumed and you may end up with broken packages. + +To explain it with an example: + + ghostscript (0) - fontconfig (1) + \ cups (1) - fontconfig (ignored) + +The second time fontconfig appears, it will be ignored. In this case cups will +build against an fontconfig version that will be outdated by the fontconfig +version at depth 1. In this cases, `treepkg` will detect it cached the +dependency on a lower depth, and will "bury" it to a depth higher than the +current one. Thus this will become the build path: + + ghostscript (0) - fontconfig (buried) + \ cups (1) - fontconfig (2) + +> Note: currently, `treepkg` doesn't perform recursive burying, so if you hit +> a really long build tree with some circular dependencies you may find +> packages buried several times and queued to build before their actuals deps. + +## Tips + +`treepkg` accepts two arguments: 1) the temporary work dir and 2) the next +depth level it should use (if current equals 0, it'll pass 1). You don't need +to pass this arguments when running it manually, they're used internally to +automatically construct the build path. + +But if a build failed, `treepkg` will cancel itself immediately informing you +where the leftovers files were left. If you pass this path to `treepkg` as the +first argument, it will resume the build, skipping to the last package being +packaged. + +You can take the opportunity given by this to modify the build path or the +PKGBUILDs, without having to run `treepkg` twice. For instance you can remove +a package from the build order, or move it manually, or update the PKGBUILD +that made `treepkg` fail in the first place. You can force a skipped package +(after building it manually) by using `touch built_ok` on the PKGBUILD dir. + +You don't probably want to mess with the second argument though. diff --git a/doc/workflows.markdown b/doc/workflows.markdown deleted file mode 100644 index f55ae7e..0000000 --- a/doc/workflows.markdown +++ /dev/null @@ -1,60 +0,0 @@ -# Workflows - -Describe your packaging workflow here! - - -## fauno's way - -During packaging, I don't usually restart a build from scratch if I have to -make changes to the PKGBUILD. I use a lot of commenting out commands already -ran, `makepkg -R`, etc. When I used `libremakepkg` I ended up using a lot more -`librechroot` and working from inside the unconfigured chroot, because -`makechrootpkg` (the underlying technology for `libremakepkg`) tries to be too -smart. - -When I started writing `treepkg` I found that mounting what I need directly on -the chroot and working from inside it was much more comfortable and simple than -having a makepkg wrapper doing funny stuff (for instance, mangling makepkg.conf -and breaking everything.) - -This is how the chroot is configured: - -* Create the same user (with same uid) on the chroot that the one I use regularly. - -* Give it password-less sudo on the chroot. - -* Bind mount /home to /chroot/home, where I have the abslibre-mips64el clone. - -* Bind mount /var/cache/pacman/pkg to /chroot/var/cache/pacman/pkg - -* Put these on system's fstab so I don't have to do it everytime - -* Configure makepkg.conf to PKGDEST=CacheDir and SRCDEST to something on my home. - -Workflow: - -* Enter the chroot with `systemd-nspawn -D/chroot` and `su - fauno`. - -* From another shell (I use tmux) edit the abslibre or search for updates with - `git log --no-merges --numstat`. - -* Pick a package and run `treepkg` from its dir on the chroot, or retake - a build with `treepkg /tmp/package-treepkg-xxxx`. (Refer to doc/treepkg - here). - -What this allows: - -* Not having to worry about the state of the chroot. `chcleanup` removes and - adds packages in a smart way so shared dependencies stay and others move - along (think of installing and removing qt for a complete kde rebuild). - -* Building many packages in a row without recreating a chroot for every one of - them. - -* Knowing that any change you made to the chroot stays as you want (no one - touches your makepkg.conf) - -* Hability to run regular commands, not through a chroot wrapper. I can `cd` to - a dir and use `makepkg -whatever` on it and nothing breaks. - -* No extra code spent on wrappers. diff --git a/doc/workflows.md b/doc/workflows.md new file mode 100644 index 0000000..f55ae7e --- /dev/null +++ b/doc/workflows.md @@ -0,0 +1,60 @@ +# Workflows + +Describe your packaging workflow here! + + +## fauno's way + +During packaging, I don't usually restart a build from scratch if I have to +make changes to the PKGBUILD. I use a lot of commenting out commands already +ran, `makepkg -R`, etc. When I used `libremakepkg` I ended up using a lot more +`librechroot` and working from inside the unconfigured chroot, because +`makechrootpkg` (the underlying technology for `libremakepkg`) tries to be too +smart. + +When I started writing `treepkg` I found that mounting what I need directly on +the chroot and working from inside it was much more comfortable and simple than +having a makepkg wrapper doing funny stuff (for instance, mangling makepkg.conf +and breaking everything.) + +This is how the chroot is configured: + +* Create the same user (with same uid) on the chroot that the one I use regularly. + +* Give it password-less sudo on the chroot. + +* Bind mount /home to /chroot/home, where I have the abslibre-mips64el clone. + +* Bind mount /var/cache/pacman/pkg to /chroot/var/cache/pacman/pkg + +* Put these on system's fstab so I don't have to do it everytime + +* Configure makepkg.conf to PKGDEST=CacheDir and SRCDEST to something on my home. + +Workflow: + +* Enter the chroot with `systemd-nspawn -D/chroot` and `su - fauno`. + +* From another shell (I use tmux) edit the abslibre or search for updates with + `git log --no-merges --numstat`. + +* Pick a package and run `treepkg` from its dir on the chroot, or retake + a build with `treepkg /tmp/package-treepkg-xxxx`. (Refer to doc/treepkg + here). + +What this allows: + +* Not having to worry about the state of the chroot. `chcleanup` removes and + adds packages in a smart way so shared dependencies stay and others move + along (think of installing and removing qt for a complete kde rebuild). + +* Building many packages in a row without recreating a chroot for every one of + them. + +* Knowing that any change you made to the chroot stays as you want (no one + touches your makepkg.conf) + +* Hability to run regular commands, not through a chroot wrapper. I can `cd` to + a dir and use `makepkg -whatever` on it and nothing breaks. + +* No extra code spent on wrappers. -- cgit v1.2.3-54-g00ecf