diff options
Diffstat (limited to 'posts/2022')
| -rw-r--r-- | posts/2022/packaging-for-arch-linux.rst | 1144 |
1 files changed, 1144 insertions, 0 deletions
diff --git a/posts/2022/packaging-for-arch-linux.rst b/posts/2022/packaging-for-arch-linux.rst new file mode 100644 index 0000000..4975454 --- /dev/null +++ b/posts/2022/packaging-for-arch-linux.rst @@ -0,0 +1,1144 @@ +.. title: Packaging for Arch Linux +.. slug: packaging-for-arch-linux +.. date: 2022-04-06 13:22:53 UTC+02:00 +.. tags: arch linux, packaging, reproducible builds, arch-repo-management, dbscripts +.. category: archlinux +.. link: +.. description: +.. type: text + +In `Arch, a recap <https://sleepmap.de/2022/arch-a-recap>`_ I elaborated a bit +on my reasons for getting involved with Arch Linux. In this post I would like +to highlight a few technical details and give a "behind the scenes" when it +comes to packaging on and for Arch Linux. +This post is written from the viewpoint of a distribution packager, but it +is likely to contain information also useful to people packaging on different +distributions or for private purposes. + +.. TEASER_END + +|arch linux| is a |linux distribution|, that offers binary packages in +|software repositories| (aka. repos). To achieve this, packages are built from +source files using tooling that is developed by the distribution and various +volunteers. The resulting binary packages are then provided to users on mirrors +of the distribution (i.e. package files and their cryptographic signatures are +provided by |web servers|) and are downloaded, verified, validated and +installed using a |package manager|. + +.. note:: + + Other distributions may use different concepts. E.g. |gentoo linux| offers + installation media that is used to install a base system. From then on users + rebuild the software on their systems themselves based on distribution + provided source files. There are no binary packages for users to install. + +The upsides of a central software package system facilitating binary repos are + +- users do not have to build the software on their systems themselves, which + e.g. for web browsers can take a very long time and eat a lot of energy +- software for the entire system can be updated with one command and only takes + as long as download and extraction of a given set of packages + +Packages +======== + +When looking at the concept of binary software packages it probably helps to +consider the point of view from e.g. Windows and macOS, which both provide +software to users in different ways and give a good case for comparison. In +case you already know how binary packages function and compare, skip this +section. + +For brevity I will skip the proprietary app stores in the below examples as +they abstract the concept of software installation to the point where this is +opaque to the user and delivers no direct comparison in the context of packages +(while under the hood most app stores use the below mentioned technologies). + +Windows +------- + +On Windows software is usually provided by the means of an installer (e.g. +shipped as a ``.exe`` or ``.msi`` file). An installer usually needs to be +downloaded from thirdparty websites (often without verification) and then +executed one-by-one. The installer often already contains the (prebuilt) files +to be installed (sometimes files are also downloaded by the installer +application on-the-fly), offers some form of modification (e.g. the +installation location), installs the bundled or downloaded files and modifies +the system's registry (e.g. for auto-start or other features). Although +Microsoft has attempted to consolidate its installation backends, the user +experience is usually still a mixed bag. +System updates (those modifying the operating system) are handled by the OS +itself and the user usually has not much of a say when/ how that happens (this +can be modified to some extent). Additionally, some hardware may not use the +latest version of Windows due to software-based |planned obsolescence|. + +MacOS +----- + +On macOS software can be installed using images or installers (shipped as e.g. +``.dmg`` and ``.pkg`` respectively). The download of the files in question +usually functions in the same way as it does on Windows (unverified downloads +from thirdparty websites). Where with images the user experience is usually +*"drag and drop"* from a mounted image to the list of applications, installers +on the other hand offer similar functionality to how installers work on Windows +(e.g. setup auto-starting). +System updates, similar to those on Windows are handled by macOS itself and +also here software enforced |planned obsolescence| is a thing. + +Looking at the above examples it becomes clear, that automation on both +platforms is quite terrible: The distinction between OS updates and *"other +software"* leads to a mix and match approach towards updates, that is (if at +all) only partly remedied by externally developed and provided package managers +for some of the *"other software"* (e.g. |homebrew| or |chocolatey|), but at +best remains a fragmented experience for the user. + +Linux +----- + +On distributions that offer binary package repositories, users use a package +manager to install packages and to upgrade **all software** [1]_ on their +system. +Packages are essentially |archive files|, that are downloaded, verified and +extracted by the package manager. As the files contained in (distribution) +packages follow a well-defined location schema (e.g. |filesystem hierarchy +standard| or |file-hierarchy|), the system can check for file conflicts and +users can usually have reasonable assumptions about where files of a package +are located (package managers usually also track the files of all packages). +Additional functionality, such as post install scripts (e.g. to create users or +to change ownership on files) are usually +contained in package files and executed after installation. However, on systemd +based distributions, much of the post installation tasks have been streamlined +with the help of |sysusers.d| and |tmpfiles.d| (more on that later). Some +distributions also make use of non-standardized hooks (see |alpm-hooks| for how +this is implemented for |pacman|), that are used by the package manager for +certain tasks on files that are not owned by one specific package (e.g. update +font cache). + +Build tooling +============= + +The most basic build tooling for Arch Linux - |makepkg| - is bundled with +|pacman_website| (the package manager used to install all software packages on the +distribution). It is used in conjunction with a |PKGBUILD|, which as a package +source file describes where/ how to get a package's source files (and in which +version), how to build and test it (if applicable) and how/ where to install +it. + +In case you have experience with |bash|: Both ``makepkg`` and ``PKGBUILDs`` are +written in it. + +When building packages with plain ``makepkg``, the built package will be +created in the context of the user's system and as such will make use of the +software available on the user's system. While this works (given all +dependencies are met) it is not recommended to do so, as the user's +system may use custom packages or settings to ``makepkg`` (see |makepkg.conf|), +that can alter the outcome of the build, which may make the package unusable on +another system. + +Clean chroots +============= + +To enable builds, that are done in a clean environment (i.e. one that only has +official distribution packages installed and does not depend on configuration +or custom packages on a local system), Arch Linux and various contributors have +created special build tooling, which is contained in the |devtools| project. + +With the help of |makechrootpkg| one can run ``makepkg`` in a |systemd-nspawn| +based |chroot|, which will only have the packages installed, that are required +for building, testing and running a given package. + +Using |makechrootpkg| and its various repository-specific symlinks is how Arch +Linux packagers build all packages in the official repositories. + +.. note:: + + It is generally advisable to build **all packages** in a clean environment. + +An implict upside of using ``makechrootpkg`` to build packages is, that +|checkpkg| and |namcap| are being run on the resulting package, which can give +valuable hints at possible improvements of the package. + +Building packages +================= + +First off: There are sometimes a lot of subtleties involved with packaging and +especially producing packages that are of good quality. In the following +sections I will discuss a few tools and packaging specifics, that may seem +quite overwhelming or complicated at first. Luckily, a lot of the tooling is +fairly well documented and it is probably always good to remember, that +everyone is a learner and that as the tooling and the best practices evolve, +this is an open-ended topic. + +A good starting point is always to use ``makechrootpkg`` and to adhere to the +|arch package guidelines|. + +.. note:: + + There is the |arch package guidelines category| of more specific guidelines + for various programming languages and special use-cases. + +Getting package build sources +----------------------------- + +The act of getting the sources for a binary package is described in the context +of the |arch build system| (ABS). While users without write access to the Arch +Linux source repositories can rely upon |asp| to get to the package build +sources, the official packagers rely on a rather organically grown packaging +workflow, that is described in |howto be a packager|. + +At the time of writing, Arch Linux still relies on two monolithic |svn| +repositories for the package build sources (one for the ``[core]`` and +``[extra]`` repositories and one for the ``[community]`` and ``[multilib]`` +repositories) which are exported to |git| via |git-svn| on the official Arch +Linux Github organization (|svntogit-packages| and |svntogit-community|, +respectively). + +.. note:: + + Work is underway to switch to |git| for the package build sources. However, + this has implications for maintaining state of resulting binary repositories + (this is currently done in the svn repositories). While |dbscripts| has been + used for managing the state for many years, it is likely to be replaced by + |arch-repo-management| in the future. + Additionally, it should be mentioned, that such a switch is not trivial after + 20 years, if one is also concerned with the technical feasibility and + maintainability of the tooling in use. + +PKGBUILDs +--------- + +As mentioned earlier, |PKGBUILD| files are really just |bash| scripts, that are +being evaluated by |makepkg|. As such they define a few variables and functions +(some of which are required, others only being optional). + +The below example shows a bare minimum example, derived from the prototype +files, that can be found in ``/usr/share/pacman/``: + +.. code:: sh + + # Maintainer: Your Name <youremail@domain.com> + pkgname=dummy-package + pkgver=0.1.0 + pkgrel=1 + pkgdesc="A dummy package" + arch=(any) + url="https://my-upstream.link/to/dummy-package" + license=(GPL3) + depends=(another-package) + optdepends=('some-additional: for additional feature X') + source=(https://my-upstream.link/to/$pkgname-$pkgver.tar.gz) + b2sums=('THISISADUMMYCHECKSUM') + + install() { + make DESTDIR="$pgkdir" install -C $pkgname-$pkgver + } + +To go through the essentials of this very minimalistic example, which assumes +that we have a project using |make| to install a few files: + +- While the ``Maintainer`` comment is technically not required, it is always + helpful for others trying to contact the author of a given package build + source +- ``pkgname``: The name of the package. Refer to the wiki section + |pkgbuild#pkgname| for further info (e.g. restrictions) +- ``pkgver``: The (upstream) version of the package. Refer to the wiki section + |pkgbuild#pkgver| for further info (e.g. restrictions) +- ``pkgrel``: The release version of the package, which identifies the build of + the particular package in version ``pkgver``. This is a string specific to + Arch Linux (see |pkgbuild#pkgrel|) and *is not related to the upstream + version* of the software. +- ``pkgdesc``: A short description of what this package provides +- ``arch``: The architecture of the resulting package. As this is an array, it + can contain several entries (``makepkg`` will envoke a build for each + architecture). At the time of writing Arch Linux only supports the ``x86_64`` + and ``any`` architectures. +- ``url``: The URL of the upstream project (e.g. a website or a link to the + |version control| sources) +- ``license``: The licenses that apply to the project. This again is an array + and may contain several licenses. In case licenses that are not covered by + the |licenses package| are encountered, their license files must be installed + in the ``install()`` function (refer to the wiki section |pkgbuild#license| + for further information). +- ``depends``: An array of runtime dependencies for the package. They will be + installed automatically during build when building with ``makechrootpkg`` or + ``makepkg -s``. +- ``optdepends``: An array of optional dependencies and a short description + about their purpose. These packages will not be installed during build time + (for this ``makedepends`` needs to be used). +- ``source``: An array of resources for ``makepkg`` to retrieve. As |makepkg| + is able to handle various |version control| systems, local and remote files, + as well as to rename files, it is advisable to read the relevant man page + section for ``makepkg``. +- ``b2sums``: An array of checksums for all resources in the ``source`` array. + It is advisable to use either (or all of) ``sh256sums``, ``sha512sums`` and + ``b2sums`` as older hashing mechanisms are by now unsafe (see + |pkgbuild#integrity|). The checksums are used to guard against changing (and + potentially malicious) upstream resources. The resources and checksums for a + new version of a given package may be retrieved and updated using + |updpkgsums| (contained in the |pacman-contrib| package). +- ``install()``: This function defines all steps necessary to install the files + of the upstream project to an empty location (represented by the *magic + variable* ``"$pkgdir"``), that will contain all installable files of the + package. This function is called using |fakeroot|, which means that to the + installing processes it looks like they are being executed by ``root``. + +PGP validation +-------------- + +Upstream project resources (e.g. signed source tarballs or git tags/ commits) +can be validated using |pgp|. + +.. note:: + + While other mechanisms are theoretically possible (e.g. |signify| or + |cosign|), they are currently not implemented in the context of |makepkg|. + +Technically all that is required for this is, +that the ``validpgpkeys`` array in the PKGBUILD contains at least one +retrievable PGP key ID and that the ``source`` array contains either a ``.sig`` +or ``.asc`` file valid for one of the resources, or that a git object to be +checked is targetted using the ``?signed`` identifier (see +|makepkg#signature_checking| and |pkgbuild#using_vcs_sources|). + +Although it is advisable to have cryptographic signature validation (e.g. +using |pgp|) for releases, this should only be considered under the following +circumstances in regards to an upstream project: + +* there is a track record of signing releases with the same key ID and the + project specifically provides the expectable key ID publicly (e.g. on the + website) +* keeps a |chain of trust| between multiple and/or successive key IDs +* no key easily used by multiple users is used (e.g. Github's PGP key, which + can be used by multiple users of a given Github project and is not handled by + the users themselves) + +The first point is usually easy to check up on, while the 2nd might require +getting in touch with the project developers if it happens (or happened in the +past) - this is the case more often than you would think and does block package +updates, as a new key ID must not be trusted without investigating the cause +for a missing |chain of trust| to prevent a potential |supply chain attack|! + +.. note:: + + I am contemplating to prepare an |arch linux rfc| for this, as the handling + of PGP signed sources is not well defined for the distribution and often + leads to mishandling (e.g. using new PGP key IDs without having a |chain of + trust|). + +The 3rd point practically provides a false sense of security: A PGP key signed +a release of a project, but in actuality multiple members of a project may have +access to this functionality. From the outside it is impossible to tell who +triggered a release and signed off on it (it could easily be malicious because +someone's Github account has been hacked). + +Reproducibility +--------------- + +Arch Linux as a distribution is committed to packages becoming bit-for-bit +reproducible (have a look at the overarching |reproducible builds| project for +more background information on the general topic). The status of the current +packages in the official repositories is tracked on +https://reproducible.archlinux.org, which is backed by |rebuilderd|. + +After building a package it can be rebuilt using |makerepropkg|, which may use +|diffoscope| on the resulting package in case it is not reproducible. + +As the use of ``makerepropkg`` requires the ``PKGBUILD`` used to build the +initial package, it can not be used when only a package file is available. +However, for that use-case |repro| may be used. + +.. note:: + + Both tools make use of the |buildinfo| files contained in each binary + package. + +Dealing with the strange +======================== + +In `building packages <#building-packages>`_ we have looked at some of the more +basic use-cases. The following sub-sections will deal with more uncommon or +very specific ones as well as problems at the intersection of build tooling and +binary repository management. + +Split packages +-------------- + +There are situations, in which one wants to build several packages from a +single ``PKGBUILD``. Those are usually: + +- the documentation of the project is very large +- certain features (e.g. language bindings) are not required by the main + application or use-case of the project +- specific functionality would require a large tree of dependencies but is not + required for the main application or use-case + +In all three cases this can be handled using a split package setup in which the +extra functionality (as a package) is declared an optional dependency of the +main package. + +To create a split package, the ``pkgname`` variable of the ``PKGBUILD`` is +turned into an array, containing multiple package names, while the ``pkgbase`` +variable (see |pkgbuild#pkgbase|) should be set. Additionally, the generic +``install()`` function needs to be split up into specific functions for each +package (``prepare()``, ``build()`` and ``check()`` are shared). + +Using the example from `PKGBUILDs <#pkgbuilds>`_, this is how it would look +like when e.g. splitting out documentation (assuming that the upstream project +provides separate install targets for the components). + +.. code:: sh + + # Maintainer: Your Name <youremail@domain.com> + pkgbase=dummy-package + pkgname=(dummy-package dummy-package-docs) + pkgver=0.1.0 + pkgrel=1 + pkgdesc="A dummy package" + arch=(any) + url="https://my-upstream.link/to/dummy-package" + license=(GPL3) + makedepends=(another-package) + source=(https://my-upstream.link/to/$pkgname-$pkgver.tar.gz) + b2sums=('THISISADUMMYCHECKSUM') + + install_dummy-package() { + depends=(another-package) + optdepends=( + 'dummy-package-docs: for documentation' + 'some-additional: for additional feature X' + ) + + make DESTDIR="$pgkdir" install-scripts -C $pkgname-$pkgver + } + + install_dummy-package-docs() { + make DESTDIR="$pgkdir" install-docs -C $pkgname-$pkgver + } + +Binary repository management +---------------------------- + +The resulting packages of a build process can be installed on a local machine, +but are often of course more useful, if more systems can install them as well. +For this purpose the repository sync databases exist, which |pacman| uses (see +|libalpm_databases|) to retrieve the difference between a remote package +repository and a local machine's state and to figure out which packages to +upgrade. + +The most rudimentary actions (adding and removing packages, optionally signing +a database) on a binary repository can be done using |repo-add| and +|repo-remove|, which are shipped with |pacman|. As the tooling is very basic, +it does not offer any form of state tracking (i.e. a log of actions, such as +additions or removals done to a sync database by a specific user). + +At the time of writing Arch Linux packagers make use of |dbscripts| for the +binary repository management, which also does (a form of) state tracking by +interacting with and using the the two |svn|-based monorepos for package build +sources for this purpose. +The tooling consists of a set of shell scripts (making use of |repo-add| and +|repo-remove| internally), that are being called by authorized users on a +specific host over |ssh|. The user authentification is therefore done using +|ssh| while the user authorization is implemented using plain unix groups +(different sets of packagers have access to ``[core]`` and ``[extra]`` vs. +``[community]`` and ``[multilib]`` - often only for historical reasons). +However, this setup is showing its age and comes with its own set of pitfalls: + +- changes to repositories are not externally auditable +- package data is only checked rudimentarily +- integrity of repository sync databases can not be guaranteed +- repository sync databases can not be rebuilt to a specific state +- setting the target binary repository for a package is a manual operation +- due to the blocking nature of dbscripts, it is possible to brick the state of + a repository if e.g. connection to the host running dbscripts is lost during + the move of packages between two + repositories +- it is not possible to setup rebuild-specific staging repositories on the fly +- many users need |ssh| access to a machine + +This all being said, work is underway with |arch-repo-management| to provide a +more manageable and easy to configure solution that runs as a service and does +not rely on multiple users to have direct access on a target system. +One of the project's main focusses is to be able to verify incoming package +data and to fully decouple the state from the repository sync databases (to be +able to rebuild them whenever needed). +Going forward it should become more easy to setup ephemeral staging +repositories to build against and safer to move data due to more atomic +repository operations, while allowing externals to audit each repository's +history. Currently the project is still far from being usable though and there +are quite a few things left to be implemented. Switching from the current setup +in which both package build sources and binary repository state are handled by +one |version control| system, to one where these concerns are separated is a +hard problem, especially when one wants to get this right. +I hope that going forward we will end up with a solution that can be easily +contributed to and reused also outside of Arch Linux. I will write another post +in the coming months, that highlights work and concepts of +|arch-repo-management|. + +Distributing trust +------------------ + +Packagers use the |makepkg.conf| variables ``PACKAGER`` and ``GPGKEY`` to set +the packager user ID (i.e. name and e-mail address) and the PGP key ID used for +signing created packages. + +Other users that wish to use packages signed by someone else need to import +that other user's PGP public key using |pacman-key|. + +Arch Linux maintains a |web of trust| between a set of main signing keys and +all packagers and between all packagers amongst themselves (see the |main keys| +page for an extensive overview). This setup allows for user systems to evaluate +whether a given package signature done by a packager is considered trusted (see +|pacman.conf#package_and_database_signature_checking| for further info). +These constructs are system-wide PGP keyrings for the use with |pacman| and can +be handled with |pacman-key|. + +.. note:: + + The main signing keys are considered **fully trusted** on a user system. They + define the root trust for the distribution which is handed down to the + packagers. + +In the |archlinux-keyring| project the distribution trust of Arch Linux is +maintained as a set of decomposed PGP public keys and the signatures on them. +The custom tooling ``keyringctl`` (which uses |sequoia|'s ``sq`` under the +hood) is used to maintain (e.g. import public keys and signatures) a PGP +keyring that is packaged in the |archlinux-keyring package| and which is +automatically added and updated upon install. + +More than or equal to three main signing key holders are required to uphold +the web of trust. More than or equal to three valid main key signatures are +required for a packager key (if it is itself still valid) to be allowed for +distributing packages in the official Arch Linux repositories. + +.. note:: + + Writing the new tooling ``keyringctl`` to manage the distribution trust has + been a huge topic of the past year, that |Levente Polyak| and I have been + working on, as the previous setup was very brittle. I will elaborate a bit + more on that topic in an upcoming post. + + +Sonames +------- + +Linux distributions mostly build C and C++ libraries and executables using +|dynamic linking|. This implies, that shared libraries usually provide a +|soname| (e.g. ``libexample.so.1``), which is in turn used (i.e. linked +against) by one or more other libraries or executables. +If the |application binary interface| (ABI) of the library in question changes, +its |soname| should be increased as well (e.g. ``libexample.so.2``). If a +package with an updated soname is released and installed, without rebuilding +any of the packages depending on it, those will fail to load (the now +non-existent) ``libexample.so.1`` shared object. + +A common task as a packager is therefore to do rebuilds for libraries and +executables when a soname change is introduced. Depending on the library +introducing the soname change or the library/executable being affected by it, +this is sometimes a bit of a painful and time consuming experience. +While it is not unheard of that projects either forget to introduce a soname +change (silently breaking consumers) or accidentally downgrade their soname, +consumers are more likely to run into trouble because of not yet implementing +changes introduced by the ABI change (requiring patches not yet included in a +stable release). + +To safeguard against cases in which soname changes went unnoticed and packages +are pushed to the repositories, it is possible to make use of |makepkg|'s +builtin dependency resolution. Extending upon the example in `PKGBUILDs +<#pkgbuilds>`_ and assuming that ``libexample`` is the package providing +``libexample.so``: + +.. code:: sh + + # Maintainer: Your Name <youremail@domain.com> + pkgname=libexample + pkgver=1.0.0 + pkgrel=1 + pkgdesc="A dummy library" + arch=(any) + url="https://my-upstream.link/to/libexample" + license=(GPL3) + depends=(glibc) + provides=(libexample.so) + source=(https://my-upstream.link/to/$pkgname-$pkgver.tar.gz) + b2sums=('THISISADUMMYCHECKSUM') + + build() { + make -C $pkgname-$pkgver + } + + install() { + make DESTDIR="$pgkdir" install -C $pkgname-$pkgver + } + +.. code:: sh + + # Maintainer: Your Name <youremail@domain.com> + pkgname=dummy-package + pkgver=0.1.0 + pkgrel=1 + pkgdesc="A dummy package" + arch=(any) + url="https://my-upstream.link/to/dummy-package" + license=(GPL3) + depends=(another-package libexample libexample.so) + optdepends=('some-additional: for additional feature X') + source=(https://my-upstream.link/to/$pkgname-$pkgver.tar.gz) + b2sums=('THISISADUMMYCHECKSUM') + + build() { + make -C $pkgname-$pkgver + } + + install() { + make DESTDIR="$pgkdir" install -C $pkgname-$pkgver + } + +If during build time ``libexample`` provided ``libexample.so.1``, the resulting +``dummy-package`` will now depend on ``libexample`` and ``libexample.so=1-64``, +which ``libexample`` provides. + +If the ``libexample`` package is then updated while accidentally including a +soname bump to ``libexample.so.2``, |pacman| will prevent this package from +being upgraded on a user's system, because it can no longer provide +``libexample.so.1``, which is required by its consumers (i.e. +``dummy-package``). +This only helps against immediate breakage on already installed systems. On +systems that are about to be installed it would lead to pacman not being able +to resolve the dependencies and bailing out. It is therefore to be considered a +stop-gap solution which allows for fixing the package(s) in question, while not +immediately breaking consumers of ``libexample``. + +In the future this feature will be directly built into |makepkg|, removing the +manual process of identifying shared libraries (and their sonames) which are +provided by packages. + +.. note:: + + To identify the sonames provided by a package, |find-libprovides| can be + used. Reversely, to identify the sonames required by a specific package + |find-libdeps| can be used. + +Debug packages +-------------- + +The ability to debug software using e.g. |gdb| is very powerful, as it allows +users to provide vital information about failing software to the packagers and +upstream projects. For this to work, a package's debug symbols need to be +provided to the debugger. +In February 2022 Arch Linux has started using |debug packages and debuginfod|, +which allows just that. + +Creating a package and additionally also building its debug symbols has now +become as easy as adding ``debug`` to the ``options`` array in a |pkgbuild| +(until this option eventually is added to the default for packagers of the +distribution). + +.. note:: + + At the time of writing not all issues with non-|file-hierarchy| compliant + files and directories have been solved yet, but a large set of debug packages + have already been built across all official repositories. + +Creating users +-------------- + +Historically, system users and groups for packages have been created +using ``.install`` scripts (see |pkgbuild#install|). This had the downside of +requiring a specific |user identifier| (UID) and/or |group identifier| (GID) +(see |UID/ GID database| for specific assignments) if file ownerships also +needed to be handled in the context of a package. +Additionally, the user and group creation was not standardized and required a +script (run as ``root``), which was only run *after* creating and installing +the package and therefore not easily testable. + +With the adoption of |systemd| and specifically |sysusers.d| the workflow has +changed to installing a single file in the context of a package to the vendor +location ``/usr/lib/sysusers.d/``. Based on the |systemd| |alpm-hooks| setup +the configuration is applied using |systemd-sysusers|. + +Changing files after package installation +----------------------------------------- + +Similar to how system users and groups have been created in the past, file +modifications (e.g. ownership, |extended file attributes| or |setuid|) have +been done using ``.install`` scripts or directly in PKGBUILDs. The problem with +this approach was, that it required the specific assignment and pinning of UIDs +and GIDs when creating the required users and groups, *before* doing the file +modifications (e.g. using |chown|). + +This task has been made less complex with |tmpfiles.d|, which allows for +packaging a single file in a package to the vendor location +``/usr/lib/tmpfiles.d/``. Due to the ordering of |alpm-hooks| first users +and groups are created and only afterwards the file configuration is applied +using |systemd-tmpfiles|, allowing for diverse scenarios. + +Packaging +========= + +Working on packages for software written in different languages (e.g. |php|, +|python|, |ruby|, |c|, |c++| or |rust|) using various build systems surely +makes for a very interesting Github profile eventually (due to providing issue +reports and fixes to many projects). + +You can find the |list of my packages| amongst the official repositories. +Moreover I currently maintain two |unofficial user repositories|: |[realtime]| +and |[pro-audio-legacy]| + +Packaging can be a fun but also a very time consuming and frustrating pastime. +As such there are many many more examples and specifics I could list (but this +article is already quite dense I am afraid). + +.. note:: + + None of the Arch Linux package maintainers are payed for their work, which + they do solely in their free time and the project is not a commercial + endeavor. Some dependency chains are fairly complex and require time and + care. The reasons as to why a package is not updated is not always obvious to + the outside and it could be due to various technical problems or the packager + just lacking the time. Please be mindful about this and instead of getting + upset try to lend a helping hand packaging (it is usually much appreciated) + or consider |getting involved|. + +At any rate, I hope I could spark your curiosity! If you are interested in +finding out more about packaging for specific languages or best practices, +the following are some good starting points: + +* |arch package guidelines| and |arch package guidelines category| +* |#archlinux| and |#archlinux-aur| on |libera.chat| + +.. [1] I am excluding `flatpak <https://flatpak.org/>`_ and `snap + <https://snapcraft.io/>`_ in this article as they follow an app-store/ + per-user installation paradigm. However, they relate to system's packaging + in that they provide precompiled binaries in a predefined format. + +.. |arch linux| raw:: html + + <a target="blank" href="https://archlinux.org">Arch Linux</a> + +.. |linux distribution| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Linux_distribution">Linux distribution</a> + +.. |software repositories| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Software_repository">software repositories</a> + +.. |web servers| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Web_server">web servers</a> + +.. |package manager| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Package_manager">package manager</a> + +.. |gentoo linux| raw:: html + + <a target="blank" href="https://www.gentoo.org/">Gentoo Linux</a> + +.. |planned obsolescence| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Planned_obsolescence">planned obsolescence</a> + +.. |homebrew| raw:: html + + <a target="blank" href="https://brew.sh/">Homebrew</a> + +.. |chocolatey| raw:: html + + <a target="blank" href="https://chocolatey.org/">Chocolatey</a> + +.. |archive files| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Archive_file">archive files</a> + +.. |filesystem hierarchy standard| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard">filesystem hierarchy standard</a> + +.. |file-hierarchy| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/core/systemd/file-hierarchy.7.en">file-hierarchy</a> + +.. |tmpfiles.d| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/tmpfiles.d.5.en">tmpfiles.d</a> + +.. |sysusers.d| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/sysusers.d.5.en">sysusers.d</a> + +.. |alpm-hooks| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/alpm-hooks.5">alpm-hooks</a> + +.. |makepkg| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/makepkg.8">makepkg</a> + +.. |pacman| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/pacman.8">pacman</a> + +.. |pacman_website| raw:: html + + <a target="blank" href="https://archlinux.org/pacman/">Pacman</a> + +.. |PKGBUILD| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/core/pacman/PKGBUILD.5.en">PKGBUILD</a> + +.. |bash| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Bash_(Unix_shell)">Bash</a> + +.. |makepkg.conf| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/makepkg.conf.5">makepkg.conf</a> + +.. |devtools| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/devtools">devtools</a> + +.. |makechrootpkg| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/makechrootpkg.1">makechrootpkg</a> + +.. |systemd-nspawn| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/systemd-nspawn.1">systemd-nspawn</a> + +.. |chroot| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/chroot.1.en">chroot</a> + +.. |checkpkg| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/checkpkg.1">checkpkg</a> + +.. |namcap| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/namcap.1">namcap</a> + +.. |arch package guidelines| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/Arch_package_guidelines">Arch package guidelines</a> + +.. |arch package guidelines category| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/Category:Arch_package_guidelines">Arch package guidelines category</a> + +.. |arch build system| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/Arch_Build_System">Arch Build System</a> + +.. |asp| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/asp.1">asp</a> + +.. |howto be a packager| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/DeveloperWiki:HOWTO_Be_A_Packager">HOWTO be a packager</a> + +.. |svn| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/svn.1">svn</a> + +.. |git| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/git.1">git</a> + +.. |git-svn| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/extra/git/git-svn.1.en">git-svn</a> + +.. |svntogit-packages| raw:: html + + <a target="blank" href="https://github.com/archlinux/svntogit-packages">svntogit-packages</a> + +.. |svntogit-community| raw:: html + + <a target="blank" href="https://github.com/archlinux/svntogit-community">svntogit-community</a> + +.. |dbscripts| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/dbscripts">dbscripts</a> + +.. |arch-repo-management| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/arch-repo-management">arch-repo-management</a> + +.. |make| raw:: html + + <a target="blank" href="https://www.gnu.org/software/make/">make</a> + +.. |pkgbuild#pkgname| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#pkgname">PKGBUILD#pkgname</a> + +.. |pkgbuild#pkgver| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#pkgver">PKGBUILD#pkgver</a> + +.. |pkgbuild#pkgrel| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#pkgrel">PKGBUILD#pkgrel</a> + +.. |version control| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Version_control">version control</a> + +.. |licenses package| raw:: html + + <a target="blank" href="https://archlinux.org/packages/core/any/licenses/">licenses package</a> + +.. |pkgbuild#license| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#license">PKGBUILD#licenses</a> + +.. |pkgbuild#integrity| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#Integrity">PKGBUILD#Integrity</a> + +.. |updpkgsums| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/updpkgsums.8">updpkgsums</a> + +.. |pacman-contrib| raw:: html + + <a target="blank" href="https://archlinux.org/packages/?q=pacman-contrib">pacman-contrib</a> + +.. |fakeroot| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/fakeroot.1">fakeroot</a> + +.. |pgp| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Pretty_Good_Privacy">PGP</a> + +.. |signify| raw:: html + + <a target="blank" href="https://github.com/aperezdc/signify">signify</a> + +.. |cosign| raw:: html + + <a target="blank" href="https://github.com/sigstore/cosign">cosign</a> + +.. |makepkg#signature_checking| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/Makepkg#Signature_checking">makepkg#signature_checking</a> + +.. |pkgbuild#using_vcs_sources| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/PKGBUILD.5.en#USING_VCS_SOURCES">PKGBUILD#USING_VCS_SOURCES</a> + +.. |chain of trust| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Chain_of_trust">chain of trust</a> + +.. |supply chain attack| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Supply_chain_attack">supply chain attack</a> + +.. |arch linux rfc| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/rfcs">Arch Linux RFC</a> + +.. |reproducible builds| raw:: html + + <a target="blank" href="https://reproducible-builds.org/">Reproducible Builds</a> + +.. |rebuilderd| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/rebuilderd.1">rebuilderd</a> + +.. |makerepropkg| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/makerepropkg.1">makerepropkg</a> + +.. |diffoscope| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/diffoscope.1">diffoscope</a> + +.. |repro| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/repro.8">repro</a> + +.. |buildinfo| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/core/pacman/BUILDINFO.5.en">.BUILDINFO</a> + +.. |pkgbuild#pkgbase| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#pkgbase">PKGBUILD#pkgbase</a> + +.. |libalpm_databases| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/libalpm_databases.3">libalpm_databases</a> + +.. |repo-add| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/repo-add.8">repo-add</a> + +.. |repo-remove| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/repo-add.8">repo-remove</a> + +.. |ssh| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/ssh.1">ssh</a> + +.. |web of trust| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Web_of_trust">web of trust</a> + +.. |main keys| raw:: html + + <a target="blank" href="https://archlinux.org/master-keys/">main keys</a> + +.. |pacman.conf#package_and_database_signature_checking| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/pacman.conf.5#PACKAGE_AND_DATABASE_SIGNATURE_CHECKING">pacman.conf#PACKAGE_AND_DATABASE_SIGNATURE_CHECKING</a> + +.. |pacman-key| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/pacman-key.8">pacman-key</a> + +.. |archlinux-keyring| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/archlinux-keyring/">archlinux-keyring</a> + +.. |sequoia| raw:: html + + <a target="blank" href="https://sequoia-pgp.org/">sequoia</a> + +.. |archlinux-keyring package| raw:: html + + <a target="blank" href="https://archlinux.org/packages/core/any/archlinux-keyring/">archlinux-keyring package</a> + +.. |Levente Polyak| raw:: html + + <a target="blank" href="https://leventepolyak.net/">Levente Polyak</a> + +.. |dynamic linking| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Dynamic_linker">dynamic linking</a> + +.. |soname| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Soname">soname</a> + +.. |application binary interface| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Application_binary_interface">application binary interface</a> + +.. |find-libprovides| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/find-libprovides.1">find-libprovides</a> + +.. |find-libdeps| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/find-libdeps.1">find-libdeps</a> + +.. |gdb| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/gdb.1">gdb</a> + +.. |debug packages and debuginfod| raw:: html + + <a target="blank" href="https://archlinux.org/news/debug-packages-and-debuginfod/">debug packages and debuginfod</a> + +.. |pkgbuild#install| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#install">PKGBUILD#install</a> + +.. |user identifier| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/User_identifier">user identifier</a> + +.. |group identifier| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Group_identifier">group identifier</a> + +.. |UID/ GID database| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/DeveloperWiki:UID_/_GID_Database">UID/ GID database</a> + +.. |systemd| raw:: html + + <a target="blank" href="https://systemd.io/">systemd</a> + +.. |systemd-sysusers| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/systemd-sysusers.8">systemd-sysusers</a> + +.. |extended file attributes| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Extended_file_attributes#Linux">extended file attributes</a> + +.. |setuid| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Setuid">setuid</a> + +.. |chown| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/chown.1">chown</a> + +.. |systemd-tmpfiles| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/systemd-tmpfiles.8">systemd-tmpfiles</a> + +.. |php| raw:: html + + <a target="blank" href="https://www.php.net/">PHP</a> + +.. |python| raw:: html + + <a target="blank" href="https://www.python.org/">Python</a> + +.. |ruby| raw:: html + + <a target="blank" href="https://www.ruby-lang.org/en/">Ruby</a> + +.. |c| raw:: html + + <a target="blank" href="https://www.iso.org/standard/74528.html">C</a> + +.. |c++| raw:: html + + <a target="blank" href="https://isocpp.org/">C++</a> + +.. |rust| raw:: html + + <a target="blank" href="https://www.rust-lang.org/">Rust</a> + +.. |#archlinux| raw:: html + + <a target="blank" href="ircs://irc.libera.chat/archlinux">#archlinux</a> + +.. |#archlinux-aur| raw:: html + + <a target="blank" href="ircs://irc.libera.chat/archlinux-aur">#archlinux-aur</a> + +.. |libera.chat| raw:: html + + <a target="blank" href="https://libera.chat/">libera.chat</a> + +.. |list of my packages| raw:: html + + <a target="blank" href="https://archlinux.org/packages/?sort=&q=&maintainer=dvzrv&flagged=">list of my packages</a> + +.. |unofficial user repositories| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/Unofficial_user_repositories">unofficial user repositories</a> + +.. |[realtime]| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/Unofficial_user_repositories#realtime">[realtime]</a> + +.. |[pro-audio-legacy]| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/Unofficial_user_repositories#pro-audio-legacy">[pro-audio-legacy]</a> + +.. |getting involved| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/Getting_involved">getting involved</a> + |