Add post about packaging on Arch Linux

posts/2022/packaging-for-arch-linux.rst:
Add post about packaging on Arch Linux, that highlights many high and
low level features and topics.

1 files changed, 1144 insertions, 0 deletions

diff --git a/posts/2022/packaging-for-arch-linux.rst b/posts/2022/packaging-for-arch-linux.rst
new file mode 100644
index 0000000..4975454
--- /dev/null
+++ b/posts/2022/packaging-for-arch-linux.rst
@@ -0,0 +1,1144 @@
+.. title: Packaging for Arch Linux
+.. slug: packaging-for-arch-linux
+.. date: 2022-04-06 13:22:53 UTC+02:00
+.. tags: arch linux, packaging, reproducible builds, arch-repo-management, dbscripts
+.. category: archlinux
+.. link: 
+.. description: 
+.. type: text
+
+In `Arch, a recap <https://sleepmap.de/2022/arch-a-recap>`_ I elaborated a bit
+on my reasons for getting involved with Arch Linux. In this post I would like
+to highlight a few technical details and give a "behind the scenes" when it
+comes to packaging on and for Arch Linux.
+This post is written from the viewpoint of a distribution packager, but it
+is likely to contain information also useful to people packaging on different
+distributions or for private purposes.
+
+.. TEASER_END
+
+|arch linux| is a |linux distribution|, that offers binary packages in
+|software repositories| (aka. repos). To achieve this, packages are built from
+source files using tooling that is developed by the distribution and various
+volunteers. The resulting binary packages are then provided to users on mirrors
+of the distribution (i.e. package files and their cryptographic signatures are
+provided by |web servers|) and are downloaded, verified, validated and
+installed using a |package manager|.
+
+.. note::
+
+  Other distributions may use different concepts. E.g. |gentoo linux| offers
+  installation media that is used to install a base system. From then on users
+  rebuild the software on their systems themselves based on distribution
+  provided source files. There are no binary packages for users to install.
+
+The upsides of a central software package system facilitating binary repos are
+
+- users do not have to build the software on their systems themselves, which
+  e.g. for web browsers can take a very long time and eat a lot of energy
+- software for the entire system can be updated with one command and only takes
+  as long as download and extraction of a given set of packages
+
+Packages
+========
+
+When looking at the concept of binary software packages it probably helps to
+consider the point of view from e.g. Windows and macOS, which both provide
+software to users in different ways and give a good case for comparison. In
+case you already know how binary packages function and compare, skip this
+section.
+
+For brevity I will skip the proprietary app stores in the below examples as
+they abstract the concept of software installation to the point where this is
+opaque to the user and delivers no direct comparison in the context of packages
+(while under the hood most app stores use the below mentioned technologies).
+
+Windows
+-------
+
+On Windows software is usually provided by the means of an installer (e.g.
+shipped as a ``.exe`` or ``.msi`` file). An installer usually needs to be
+downloaded from thirdparty websites (often without verification) and then
+executed one-by-one. The installer often already contains the (prebuilt) files
+to be installed (sometimes files are also downloaded by the installer
+application on-the-fly), offers some form of modification (e.g. the
+installation location), installs the bundled or downloaded files and modifies
+the system's registry (e.g. for auto-start or other features). Although
+Microsoft has attempted to consolidate its installation backends, the user
+experience is usually still a mixed bag.
+System updates (those modifying the operating system) are handled by the OS
+itself and the user usually has not much of a say when/ how that happens (this
+can be modified to some extent). Additionally, some hardware may not use the
+latest version of Windows due to software-based |planned obsolescence|.
+
+MacOS
+-----
+
+On macOS software can be installed using images or installers (shipped as e.g.
+``.dmg`` and ``.pkg`` respectively). The download of the files in question
+usually functions in the same way as it does on Windows (unverified downloads
+from thirdparty websites). Where with images the user experience is usually
+*"drag and drop"* from a mounted image to the list of applications, installers
+on the other hand offer similar functionality to how installers work on Windows
+(e.g. setup auto-starting).
+System updates, similar to those on Windows are handled by macOS itself and
+also here software enforced |planned obsolescence| is a thing.
+
+Looking at the above examples it becomes clear, that automation on both
+platforms is quite terrible: The distinction between OS updates and *"other
+software"* leads to a mix and match approach towards updates, that is (if at
+all) only partly remedied by externally developed and provided package managers
+for some of the *"other software"* (e.g. |homebrew| or |chocolatey|), but at
+best remains a fragmented experience for the user.
+
+Linux
+-----
+
+On distributions that offer binary package repositories, users use a package
+manager to install packages and to upgrade **all software** [1]_ on their
+system.
+Packages are essentially |archive files|, that are downloaded, verified and
+extracted by the package manager. As the files contained in (distribution)
+packages follow a well-defined location schema (e.g. |filesystem hierarchy
+standard| or |file-hierarchy|), the system can check for file conflicts and
+users can usually have reasonable assumptions about where files of a package
+are located (package managers usually also track the files of all packages).
+Additional functionality, such as post install scripts (e.g. to create users or
+to change ownership on files) are usually
+contained in package files and executed after installation. However, on systemd
+based distributions, much of the post installation tasks have been streamlined
+with the help of |sysusers.d| and |tmpfiles.d| (more on that later). Some
+distributions also make use of non-standardized hooks (see |alpm-hooks| for how
+this is implemented for |pacman|), that are used by the package manager for
+certain tasks on files that are not owned by one specific package (e.g. update
+font cache).
+
+Build tooling
+=============
+
+The most basic build tooling for Arch Linux - |makepkg| - is bundled with
+|pacman_website| (the package manager used to install all software packages on the
+distribution). It is used in conjunction with a |PKGBUILD|, which as a package
+source file describes where/ how to get a package's source files (and in which
+version), how to build and test it (if applicable) and how/ where to install
+it.
+
+In case you have experience with |bash|: Both ``makepkg`` and ``PKGBUILDs`` are
+written in it.
+
+When building packages with plain ``makepkg``, the built package will be
+created in the context of the user's system and as such will make use of the
+software available on the user's system. While this works (given all
+dependencies are met) it is not recommended to do so, as the user's
+system may use custom packages or settings to ``makepkg`` (see |makepkg.conf|),
+that can alter the outcome of the build, which may make the package unusable on
+another system.
+
+Clean chroots
+=============
+
+To enable builds, that are done in a clean environment (i.e. one that only has
+official distribution packages installed and does not depend on configuration
+or custom packages on a local system), Arch Linux and various contributors have
+created special build tooling, which is contained in the |devtools| project.
+
+With the help of |makechrootpkg| one can run ``makepkg`` in a |systemd-nspawn|
+based |chroot|, which will only have the packages installed, that are required
+for building, testing and running a given package.
+
+Using |makechrootpkg| and its various repository-specific symlinks is how Arch
+Linux packagers build all packages in the official repositories.
+
+.. note::
+
+  It is generally advisable to build **all packages** in a clean environment.
+
+An implict upside of using ``makechrootpkg`` to build packages is, that
+|checkpkg| and |namcap| are being run on the resulting package, which can give
+valuable hints at possible improvements of the package.
+
+Building packages
+=================
+
+First off: There are sometimes a lot of subtleties involved with packaging and
+especially producing packages that are of good quality. In the following
+sections I will discuss a few tools and packaging specifics, that may seem
+quite overwhelming or complicated at first. Luckily, a lot of the tooling is
+fairly well documented and it is probably always good to remember, that
+everyone is a learner and that as the tooling and the best practices evolve,
+this is an open-ended topic.
+
+A good starting point is always to use ``makechrootpkg`` and to adhere to the
+|arch package guidelines|.
+
+.. note::
+
+  There is the |arch package guidelines category| of more specific guidelines
+  for various programming languages and special use-cases.
+
+Getting package build sources
+-----------------------------
+
+The act of getting the sources for a binary package is described in the context
+of the |arch build system| (ABS). While users without write access to the Arch
+Linux source repositories can rely upon |asp| to get to the package build
+sources, the official packagers rely on a rather organically grown packaging
+workflow, that is described in |howto be a packager|.
+
+At the time of writing, Arch Linux still relies on two monolithic |svn|
+repositories for the package build sources (one for the ``[core]`` and
+``[extra]`` repositories and one for the ``[community]`` and ``[multilib]``
+repositories) which are exported to |git| via |git-svn| on the official Arch
+Linux Github organization (|svntogit-packages| and |svntogit-community|,
+respectively).
+
+.. note::
+
+  Work is underway to switch to |git| for the package build sources. However,
+  this has implications for maintaining state of resulting binary repositories
+  (this is currently done in the svn repositories). While |dbscripts| has been
+  used for managing the state for many years, it is likely to be replaced by
+  |arch-repo-management| in the future.
+  Additionally, it should be mentioned, that such a switch is not trivial after
+  20 years, if one is also concerned with the technical feasibility and
+  maintainability of the tooling in use.
+
+PKGBUILDs
+---------
+
+As mentioned earlier, |PKGBUILD| files are really just |bash| scripts, that are
+being evaluated by |makepkg|. As such they define a few variables and functions
+(some of which are required, others only being optional).
+
+The below example shows a bare minimum example, derived from the prototype
+files, that can be found in ``/usr/share/pacman/``:
+
+.. code:: sh
+
+  # Maintainer: Your Name <youremail@domain.com>
+  pkgname=dummy-package
+  pkgver=0.1.0
+  pkgrel=1
+  pkgdesc="A dummy package"
+  arch=(any)
+  url="https://my-upstream.link/to/dummy-package"
+  license=(GPL3)
+  depends=(another-package)
+  optdepends=('some-additional: for additional feature X')
+  source=(https://my-upstream.link/to/$pkgname-$pkgver.tar.gz)
+  b2sums=('THISISADUMMYCHECKSUM')
+
+  install() {
+    make DESTDIR="$pgkdir" install -C $pkgname-$pkgver
+  }
+
+To go through the essentials of this very minimalistic example, which assumes
+that we have a project using |make| to install a few files:
+
+- While the ``Maintainer`` comment is technically not required, it is always
+  helpful for others trying to contact the author of a given package build
+  source
+- ``pkgname``: The name of the package. Refer to the wiki section
+  |pkgbuild#pkgname| for further info (e.g. restrictions)
+- ``pkgver``: The (upstream) version of the package. Refer to the wiki section
+  |pkgbuild#pkgver| for further info (e.g. restrictions)
+- ``pkgrel``: The release version of the package, which identifies the build of
+  the particular package in version ``pkgver``. This is a string specific to
+  Arch Linux (see |pkgbuild#pkgrel|) and *is not related to the upstream
+  version* of the software.
+- ``pkgdesc``: A short description of what this package provides
+- ``arch``: The architecture of the resulting package. As this is an array, it
+  can contain several entries (``makepkg`` will envoke a build for each
+  architecture). At the time of writing Arch Linux only supports the ``x86_64``
+  and ``any`` architectures.
+- ``url``: The URL of the upstream project (e.g. a website or a link to the
+  |version control| sources)
+- ``license``: The licenses that apply to the project. This again is an array
+  and may contain several licenses. In case licenses that are not covered by
+  the |licenses package| are encountered, their license files must be installed
+  in the ``install()`` function (refer to the wiki section |pkgbuild#license|
+  for further information).
+- ``depends``: An array of runtime dependencies for the package. They will be
+  installed automatically during build when building with ``makechrootpkg`` or
+  ``makepkg -s``.
+- ``optdepends``: An array of optional dependencies and a short description
+  about their purpose. These packages will not be installed during build time
+  (for this ``makedepends`` needs to be used).
+- ``source``: An array of resources for ``makepkg`` to retrieve. As |makepkg|
+  is able to handle various |version control| systems, local and remote files,
+  as well as to rename files, it is advisable to read the relevant man page
+  section for ``makepkg``.
+- ``b2sums``: An array of checksums for all resources in the ``source`` array.
+  It is advisable to use either (or all of) ``sh256sums``, ``sha512sums`` and
+  ``b2sums`` as older hashing mechanisms are by now unsafe (see
+  |pkgbuild#integrity|). The checksums are used to guard against changing (and
+  potentially malicious) upstream resources. The resources and checksums for a
+  new version of a given package may be retrieved and updated using
+  |updpkgsums| (contained in the |pacman-contrib| package).
+- ``install()``: This function defines all steps necessary to install the files
+  of the upstream project to an empty location (represented by the *magic
+  variable* ``"$pkgdir"``), that will contain all installable files of the
+  package. This function is called using |fakeroot|, which means that to the
+  installing processes it looks like they are being executed by ``root``.
+
+PGP validation
+--------------
+
+Upstream project resources (e.g. signed source tarballs or git tags/ commits)
+can be validated using |pgp|.
+
+.. note::
+
+  While other mechanisms are theoretically possible (e.g. |signify| or
+  |cosign|), they are currently not implemented in the context of |makepkg|.
+
+Technically all that is required for this is,
+that the ``validpgpkeys`` array in the PKGBUILD contains at least one
+retrievable PGP key ID and that the ``source`` array contains either a ``.sig``
+or ``.asc`` file valid for one of the resources, or that a git object to be
+checked is targetted using the ``?signed`` identifier (see
+|makepkg#signature_checking| and |pkgbuild#using_vcs_sources|).
+
+Although it is advisable to have cryptographic signature validation (e.g.
+using |pgp|) for releases, this should only be considered under the following
+circumstances in regards to an upstream project:
+
+* there is a track record of signing releases with the same key ID and the
+  project specifically provides the expectable key ID publicly (e.g. on the
+  website)
+* keeps a |chain of trust| between multiple and/or successive key IDs
+* no key easily used by multiple users is used (e.g. Github's PGP key, which
+  can be used by multiple users of a given Github project and is not handled by
+  the users themselves)
+
+The first point is usually easy to check up on, while the 2nd might require
+getting in touch with the project developers if it happens (or happened in the
+past) - this is the case more often than you would think and does block package
+updates, as a new key ID must not be trusted without investigating the cause
+for a missing |chain of trust| to prevent a potential |supply chain attack|!
+
+.. note::
+
+  I am contemplating to prepare an |arch linux rfc| for this, as the handling
+  of PGP signed sources is not well defined for the distribution and often
+  leads to mishandling (e.g. using new PGP key IDs without having a |chain of
+  trust|).
+
+The 3rd point practically provides a false sense of security: A PGP key signed
+a release of a project, but in actuality multiple members of a project may have
+access to this functionality. From the outside it is impossible to tell who
+triggered a release and signed off on it (it could easily be malicious because
+someone's Github account has been hacked).
+
+Reproducibility
+---------------
+
+Arch Linux as a distribution is committed to packages becoming bit-for-bit
+reproducible (have a look at the overarching |reproducible builds| project for
+more background information on the general topic). The status of the current
+packages in the official repositories is tracked on
+https://reproducible.archlinux.org, which is backed by |rebuilderd|.
+
+After building a package it can be rebuilt using |makerepropkg|, which may use
+|diffoscope| on the resulting package in case it is not reproducible.
+
+As the use of ``makerepropkg`` requires the ``PKGBUILD`` used to build the
+initial package, it can not be used when only a package file is available.
+However, for that use-case |repro| may be used.
+
+.. note::
+
+  Both tools make use of the |buildinfo| files contained in each binary
+  package.
+
+Dealing with the strange
+========================
+
+In `building packages <#building-packages>`_ we have looked at some of the more
+basic use-cases. The following sub-sections will deal with more uncommon or
+very specific ones as well as problems at the intersection of build tooling and
+binary repository management.
+
+Split packages
+--------------
+
+There are situations, in which one wants to build several packages from a
+single ``PKGBUILD``. Those are usually:
+
+- the documentation of the project is very large
+- certain features (e.g. language bindings) are not required by the main
+  application or use-case of the project
+- specific functionality would require a large tree of dependencies but is not
+  required for the main application or use-case
+
+In all three cases this can be handled using a split package setup in which the
+extra functionality (as a package) is declared an optional dependency of the
+main package.
+
+To create a split package, the ``pkgname`` variable of the ``PKGBUILD`` is
+turned into an array, containing multiple package names, while the ``pkgbase``
+variable (see |pkgbuild#pkgbase|) should be set. Additionally, the generic
+``install()`` function needs to be split up into specific functions for each
+package (``prepare()``, ``build()`` and ``check()`` are shared).
+
+Using the example from `PKGBUILDs <#pkgbuilds>`_, this is how it would look
+like when e.g. splitting out documentation (assuming that the upstream project
+provides separate install targets for the components).
+
+.. code:: sh
+
+  # Maintainer: Your Name <youremail@domain.com>
+  pkgbase=dummy-package
+  pkgname=(dummy-package dummy-package-docs)
+  pkgver=0.1.0
+  pkgrel=1
+  pkgdesc="A dummy package"
+  arch=(any)
+  url="https://my-upstream.link/to/dummy-package"
+  license=(GPL3)
+  makedepends=(another-package)
+  source=(https://my-upstream.link/to/$pkgname-$pkgver.tar.gz)
+  b2sums=('THISISADUMMYCHECKSUM')
+
+  install_dummy-package() {
+    depends=(another-package)
+    optdepends=(
+      'dummy-package-docs: for documentation'
+      'some-additional: for additional feature X'
+    )
+
+    make DESTDIR="$pgkdir" install-scripts -C $pkgname-$pkgver
+  }
+
+  install_dummy-package-docs() {
+    make DESTDIR="$pgkdir" install-docs -C $pkgname-$pkgver
+  }
+
+Binary repository management
+----------------------------
+
+The resulting packages of a build process can be installed on a local machine,
+but are often of course more useful, if more systems can install them as well.
+For this purpose the repository sync databases exist, which |pacman| uses (see
+|libalpm_databases|) to retrieve the difference between a remote package
+repository and a local machine's state and to figure out which packages to
+upgrade.
+
+The most rudimentary actions (adding and removing packages, optionally signing
+a database) on a binary repository can be done using |repo-add| and
+|repo-remove|, which are shipped with |pacman|. As the tooling is very basic,
+it does not offer any form of state tracking (i.e. a log of actions, such as
+additions or removals done to a sync database by a specific user).
+
+At the time of writing Arch Linux packagers make use of |dbscripts| for the
+binary repository management, which also does (a form of) state tracking by
+interacting with and using the the two |svn|-based monorepos for package build
+sources for this purpose.
+The tooling consists of a set of shell scripts (making use of |repo-add| and
+|repo-remove| internally), that are being called by authorized users on a
+specific host over |ssh|. The user authentification is therefore done using
+|ssh| while the user authorization is implemented using plain unix groups
+(different sets of packagers have access to ``[core]`` and ``[extra]`` vs.
+``[community]`` and ``[multilib]`` - often only for historical reasons).
+However, this setup is showing its age and comes with its own set of pitfalls:
+
+- changes to repositories are not externally auditable
+- package data is only checked rudimentarily
+- integrity of repository sync databases can not be guaranteed
+- repository sync databases can not be rebuilt to a specific state
+- setting the target binary repository for a package is a manual operation
+- due to the blocking nature of dbscripts, it is possible to brick the state of
+  a repository if e.g. connection to the host running dbscripts is lost during
+  the move of packages between two
+  repositories
+- it is not possible to setup rebuild-specific staging repositories on the fly
+- many users need |ssh| access to a machine
+
+This all being said, work is underway with |arch-repo-management| to provide a
+more manageable and easy to configure solution that runs as a service and does
+not rely on multiple users to have direct access on a target system.
+One of the project's main focusses is to be able to verify incoming package
+data and to fully decouple the state from the repository sync databases (to be
+able to rebuild them whenever needed).
+Going forward it should become more easy to setup ephemeral staging
+repositories to build against and safer to move data due to more atomic
+repository operations, while allowing externals to audit each repository's
+history. Currently the project is still far from being usable though and there
+are quite a few things left to be implemented. Switching from the current setup
+in which both package build sources and binary repository state are handled by
+one |version control| system, to one where these concerns are separated is a
+hard problem, especially when one wants to get this right.
+I hope that going forward we will end up with a solution that can be easily
+contributed to and reused also outside of Arch Linux. I will write another post
+in the coming months, that highlights work and concepts of
+|arch-repo-management|.
+
+Distributing trust
+------------------
+
+Packagers use the |makepkg.conf| variables ``PACKAGER`` and ``GPGKEY`` to set
+the packager user ID (i.e. name and e-mail address) and the PGP key ID used for
+signing created packages.
+
+Other users that wish to use packages signed by someone else need to import
+that other user's PGP public key using |pacman-key|.
+
+Arch Linux maintains a |web of trust| between a set of main signing keys and
+all packagers and between all packagers amongst themselves (see the |main keys|
+page for an extensive overview). This setup allows for user systems to evaluate
+whether a given package signature done by a packager is considered trusted (see
+|pacman.conf#package_and_database_signature_checking| for further info).
+These constructs are system-wide PGP keyrings for the use with |pacman| and can
+be handled with |pacman-key|.
+
+.. note::
+
+  The main signing keys are considered **fully trusted** on a user system. They
+  define the root trust for the distribution which is handed down to the
+  packagers.
+
+In the |archlinux-keyring| project the distribution trust of Arch Linux is
+maintained as a set of decomposed PGP public keys and the signatures on them.
+The custom tooling ``keyringctl`` (which uses |sequoia|'s ``sq`` under the
+hood) is used to maintain (e.g. import public keys and signatures) a PGP
+keyring that is packaged in the |archlinux-keyring package| and which is
+automatically added and updated upon install.
+
+More than or equal to three main signing key holders are required to uphold
+the web of trust. More than or equal to three valid main key signatures are
+required for a packager key (if it is itself still valid) to be allowed for
+distributing packages in the official Arch Linux repositories.
+
+.. note::
+
+  Writing the new tooling ``keyringctl`` to manage the distribution trust has
+  been a huge topic of the past year, that |Levente Polyak| and I have been
+  working on, as the previous setup was very brittle. I will elaborate a bit
+  more on that topic in an upcoming post.
+
+
+Sonames
+-------
+
+Linux distributions mostly build C and C++ libraries and executables using
+|dynamic linking|. This implies, that shared libraries usually provide a
+|soname| (e.g. ``libexample.so.1``), which is in turn used (i.e. linked
+against) by one or more other libraries or executables.
+If the |application binary interface| (ABI) of the library in question changes,
+its |soname| should be increased as well (e.g. ``libexample.so.2``). If a
+package with an updated soname is released and installed, without rebuilding
+any of the packages depending on it, those will fail to load (the now
+non-existent) ``libexample.so.1`` shared object.
+
+A common task as a packager is therefore to do rebuilds for libraries and
+executables when a soname change is introduced. Depending on the library
+introducing the soname change or the library/executable being affected by it,
+this is sometimes a bit of a painful and time consuming experience.
+While it is not unheard of that projects either forget to introduce a soname
+change (silently breaking consumers) or accidentally downgrade their soname,
+consumers are more likely to run into trouble because of not yet implementing
+changes introduced by the ABI change (requiring patches not yet included in a
+stable release).
+
+To safeguard against cases in which soname changes went unnoticed and packages
+are pushed to the repositories, it is possible to make use of |makepkg|'s
+builtin dependency resolution. Extending upon the example in `PKGBUILDs
+<#pkgbuilds>`_ and assuming that ``libexample`` is the package providing
+``libexample.so``:
+
+.. code:: sh
+
+  # Maintainer: Your Name <youremail@domain.com>
+  pkgname=libexample
+  pkgver=1.0.0
+  pkgrel=1
+  pkgdesc="A dummy library"
+  arch=(any)
+  url="https://my-upstream.link/to/libexample"
+  license=(GPL3)
+  depends=(glibc)
+  provides=(libexample.so)
+  source=(https://my-upstream.link/to/$pkgname-$pkgver.tar.gz)
+  b2sums=('THISISADUMMYCHECKSUM')
+
+  build() {
+    make -C $pkgname-$pkgver
+  }
+
+  install() {
+    make DESTDIR="$pgkdir" install -C $pkgname-$pkgver
+  }
+
+.. code:: sh
+
+  # Maintainer: Your Name <youremail@domain.com>
+  pkgname=dummy-package
+  pkgver=0.1.0
+  pkgrel=1
+  pkgdesc="A dummy package"
+  arch=(any)
+  url="https://my-upstream.link/to/dummy-package"
+  license=(GPL3)
+  depends=(another-package libexample libexample.so)
+  optdepends=('some-additional: for additional feature X')
+  source=(https://my-upstream.link/to/$pkgname-$pkgver.tar.gz)
+  b2sums=('THISISADUMMYCHECKSUM')
+
+  build() {
+    make -C $pkgname-$pkgver
+  }
+
+  install() {
+    make DESTDIR="$pgkdir" install -C $pkgname-$pkgver
+  }
+
+If during build time ``libexample`` provided ``libexample.so.1``, the resulting
+``dummy-package`` will now depend on ``libexample`` and ``libexample.so=1-64``,
+which ``libexample`` provides.
+
+If the ``libexample`` package is then updated while accidentally including a
+soname bump to ``libexample.so.2``, |pacman| will prevent this package from
+being upgraded on a user's system, because it can no longer provide
+``libexample.so.1``, which is required by its consumers (i.e.
+``dummy-package``).
+This only helps against immediate breakage on already installed systems. On
+systems that are about to be installed it would lead to pacman not being able
+to resolve the dependencies and bailing out. It is therefore to be considered a
+stop-gap solution which allows for fixing the package(s) in question, while not
+immediately breaking consumers of ``libexample``.
+
+In the future this feature will be directly built into |makepkg|, removing the
+manual process of identifying shared libraries (and their sonames) which are
+provided by packages.
+
+.. note::
+
+  To identify the sonames provided by a package, |find-libprovides| can be
+  used. Reversely, to identify the sonames required by a specific package
+  |find-libdeps| can be used.
+
+Debug packages
+--------------
+
+The ability to debug software using e.g. |gdb| is very powerful, as it allows
+users to provide vital information about failing software to the packagers and
+upstream projects. For this to work, a package's debug symbols need to be
+provided to the debugger.
+In February 2022 Arch Linux has started using |debug packages and debuginfod|,
+which allows just that.
+
+Creating a package and additionally also building its debug symbols has now
+become as easy as adding ``debug`` to the ``options`` array in a |pkgbuild|
+(until this option eventually is added to the default for packagers of the
+distribution).
+
+.. note::
+
+  At the time of writing not all issues with non-|file-hierarchy| compliant
+  files and directories have been solved yet, but a large set of debug packages
+  have already been built across all official repositories.
+
+Creating users
+--------------
+
+Historically, system users and groups for packages have been created
+using ``.install`` scripts (see |pkgbuild#install|). This had the downside of
+requiring a specific |user identifier| (UID) and/or |group identifier| (GID)
+(see |UID/ GID database| for specific assignments) if file ownerships also
+needed to be handled in the context of a package.
+Additionally, the user and group creation was not standardized and required a
+script (run as ``root``), which was only run *after* creating and installing
+the package and therefore not easily testable.
+
+With the adoption of |systemd| and specifically |sysusers.d| the workflow has
+changed to installing a single file in the context of a package to the vendor
+location ``/usr/lib/sysusers.d/``. Based on the |systemd| |alpm-hooks| setup
+the configuration is applied using |systemd-sysusers|.
+
+Changing files after package installation
+-----------------------------------------
+
+Similar to how system users and groups have been created in the past, file
+modifications (e.g. ownership, |extended file attributes| or |setuid|) have
+been done using ``.install`` scripts or directly in PKGBUILDs. The problem with
+this approach was, that it required the specific assignment and pinning of UIDs
+and GIDs when creating the required users and groups, *before* doing the file
+modifications (e.g. using |chown|).
+
+This task has been made less complex with |tmpfiles.d|, which allows for
+packaging a single file in a package to the vendor location
+``/usr/lib/tmpfiles.d/``. Due to the ordering of |alpm-hooks| first users
+and groups are created and only afterwards the file configuration is applied
+using |systemd-tmpfiles|, allowing for diverse scenarios.
+
+Packaging
+=========
+
+Working on packages for software written in different languages (e.g. |php|,
+|python|, |ruby|, |c|, |c++| or |rust|) using various build systems surely
+makes for a very interesting Github profile eventually (due to providing issue
+reports and fixes to many projects).
+
+You can find the |list of my packages| amongst the official repositories.
+Moreover I currently maintain two |unofficial user repositories|: |[realtime]|
+and |[pro-audio-legacy]|
+
+Packaging can be a fun but also a very time consuming and frustrating pastime.
+As such there are many many more examples and specifics I could list (but this
+article is already quite dense I am afraid).
+
+.. note::
+
+  None of the Arch Linux package maintainers are payed for their work, which
+  they do solely in their free time and the project is not a commercial
+  endeavor. Some dependency chains are fairly complex and require time and
+  care. The reasons as to why a package is not updated is not always obvious to
+  the outside and it could be due to various technical problems or the packager
+  just lacking the time. Please be mindful about this and instead of getting
+  upset try to lend a helping hand packaging (it is usually much appreciated)
+  or consider |getting involved|.
+
+At any rate, I hope I could spark your curiosity! If you are interested in
+finding out more about packaging for specific languages or best practices,
+the following are some good starting points:
+
+* |arch package guidelines| and |arch package guidelines category|
+* |#archlinux| and |#archlinux-aur| on |libera.chat|
+
+.. [1] I am excluding `flatpak <https://flatpak.org/>`_ and `snap
+   <https://snapcraft.io/>`_ in this article as they follow an app-store/
+   per-user installation paradigm. However, they relate to system's packaging
+   in that they provide precompiled binaries in a predefined format.
+
+.. |arch linux| raw:: html
+
+  <a target="blank" href="https://archlinux.org">Arch Linux</a>
+
+.. |linux distribution| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Linux_distribution">Linux distribution</a>
+
+.. |software repositories| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Software_repository">software repositories</a>
+
+.. |web servers| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Web_server">web servers</a>
+
+.. |package manager| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Package_manager">package manager</a>
+
+.. |gentoo linux| raw:: html
+
+  <a target="blank" href="https://www.gentoo.org/">Gentoo Linux</a>
+
+.. |planned obsolescence| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Planned_obsolescence">planned obsolescence</a>
+
+.. |homebrew| raw:: html
+
+  <a target="blank" href="https://brew.sh/">Homebrew</a>
+
+.. |chocolatey| raw:: html
+
+  <a target="blank" href="https://chocolatey.org/">Chocolatey</a>
+
+.. |archive files| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Archive_file">archive files</a>
+
+.. |filesystem hierarchy standard| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard">filesystem hierarchy standard</a>
+
+.. |file-hierarchy| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/core/systemd/file-hierarchy.7.en">file-hierarchy</a>
+
+.. |tmpfiles.d| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/tmpfiles.d.5.en">tmpfiles.d</a>
+
+.. |sysusers.d| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/sysusers.d.5.en">sysusers.d</a>
+
+.. |alpm-hooks| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/alpm-hooks.5">alpm-hooks</a>
+
+.. |makepkg| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/makepkg.8">makepkg</a>
+
+.. |pacman| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/pacman.8">pacman</a>
+
+.. |pacman_website| raw:: html
+
+  <a target="blank" href="https://archlinux.org/pacman/">Pacman</a>
+
+.. |PKGBUILD| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/core/pacman/PKGBUILD.5.en">PKGBUILD</a>
+
+.. |bash| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Bash_(Unix_shell)">Bash</a>
+
+.. |makepkg.conf| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/makepkg.conf.5">makepkg.conf</a>
+
+.. |devtools| raw:: html
+
+  <a target="blank" href="https://gitlab.archlinux.org/archlinux/devtools">devtools</a>
+
+.. |makechrootpkg| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/makechrootpkg.1">makechrootpkg</a>
+
+.. |systemd-nspawn| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/systemd-nspawn.1">systemd-nspawn</a>
+
+.. |chroot| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/chroot.1.en">chroot</a>
+
+.. |checkpkg| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/checkpkg.1">checkpkg</a>
+
+.. |namcap| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/namcap.1">namcap</a>
+
+.. |arch package guidelines| raw:: html
+
+  <a target="blank" href="https://wiki.archlinux.org/title/Arch_package_guidelines">Arch package guidelines</a>
+
+.. |arch package guidelines category| raw:: html
+
+  <a target="blank" href="https://wiki.archlinux.org/title/Category:Arch_package_guidelines">Arch package guidelines category</a>
+
+.. |arch build system| raw:: html
+
+  <a target="blank" href="https://wiki.archlinux.org/title/Arch_Build_System">Arch Build System</a>
+
+.. |asp| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/asp.1">asp</a>
+
+.. |howto be a packager| raw:: html
+
+  <a target="blank" href="https://wiki.archlinux.org/title/DeveloperWiki:HOWTO_Be_A_Packager">HOWTO be a packager</a>
+
+.. |svn| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/svn.1">svn</a>
+
+.. |git| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/git.1">git</a>
+
+.. |git-svn| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/extra/git/git-svn.1.en">git-svn</a>
+
+.. |svntogit-packages| raw:: html
+
+  <a target="blank" href="https://github.com/archlinux/svntogit-packages">svntogit-packages</a>
+
+.. |svntogit-community| raw:: html
+
+  <a target="blank" href="https://github.com/archlinux/svntogit-community">svntogit-community</a>
+
+.. |dbscripts| raw:: html
+
+  <a target="blank" href="https://gitlab.archlinux.org/archlinux/dbscripts">dbscripts</a>
+
+.. |arch-repo-management| raw:: html
+
+  <a target="blank" href="https://gitlab.archlinux.org/archlinux/arch-repo-management">arch-repo-management</a>
+
+.. |make| raw:: html
+
+  <a target="blank" href="https://www.gnu.org/software/make/">make</a>
+
+.. |pkgbuild#pkgname| raw:: html
+
+  <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#pkgname">PKGBUILD#pkgname</a>
+
+.. |pkgbuild#pkgver| raw:: html
+
+  <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#pkgver">PKGBUILD#pkgver</a>
+
+.. |pkgbuild#pkgrel| raw:: html
+
+  <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#pkgrel">PKGBUILD#pkgrel</a>
+
+.. |version control| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Version_control">version control</a>
+
+.. |licenses package| raw:: html
+
+  <a target="blank" href="https://archlinux.org/packages/core/any/licenses/">licenses package</a>
+
+.. |pkgbuild#license| raw:: html
+
+  <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#license">PKGBUILD#licenses</a>
+
+.. |pkgbuild#integrity| raw:: html
+
+  <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#Integrity">PKGBUILD#Integrity</a>
+
+.. |updpkgsums| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/updpkgsums.8">updpkgsums</a>
+
+.. |pacman-contrib| raw:: html
+
+  <a target="blank" href="https://archlinux.org/packages/?q=pacman-contrib">pacman-contrib</a>
+
+.. |fakeroot| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/fakeroot.1">fakeroot</a>
+
+.. |pgp| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Pretty_Good_Privacy">PGP</a>
+
+.. |signify| raw:: html
+
+  <a target="blank" href="https://github.com/aperezdc/signify">signify</a>
+
+.. |cosign| raw:: html
+
+  <a target="blank" href="https://github.com/sigstore/cosign">cosign</a>
+
+.. |makepkg#signature_checking| raw:: html
+
+  <a target="blank" href="https://wiki.archlinux.org/title/Makepkg#Signature_checking">makepkg#signature_checking</a>
+
+.. |pkgbuild#using_vcs_sources| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/PKGBUILD.5.en#USING_VCS_SOURCES">PKGBUILD#USING_VCS_SOURCES</a>
+
+.. |chain of trust| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Chain_of_trust">chain of trust</a>
+
+.. |supply chain attack| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Supply_chain_attack">supply chain attack</a>
+
+.. |arch linux rfc| raw:: html
+
+  <a target="blank" href="https://gitlab.archlinux.org/archlinux/rfcs">Arch Linux RFC</a>
+
+.. |reproducible builds| raw:: html
+
+  <a target="blank" href="https://reproducible-builds.org/">Reproducible Builds</a>
+
+.. |rebuilderd| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/rebuilderd.1">rebuilderd</a>
+
+.. |makerepropkg| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/makerepropkg.1">makerepropkg</a>
+
+.. |diffoscope| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/diffoscope.1">diffoscope</a>
+
+.. |repro| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/repro.8">repro</a>
+
+.. |buildinfo| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/core/pacman/BUILDINFO.5.en">.BUILDINFO</a>
+
+.. |pkgbuild#pkgbase| raw:: html
+
+  <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#pkgbase">PKGBUILD#pkgbase</a>
+
+.. |libalpm_databases| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/libalpm_databases.3">libalpm_databases</a>
+
+.. |repo-add| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/repo-add.8">repo-add</a>
+
+.. |repo-remove| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/repo-add.8">repo-remove</a>
+
+.. |ssh| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/ssh.1">ssh</a>
+
+.. |web of trust| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Web_of_trust">web of trust</a>
+
+.. |main keys| raw:: html
+
+  <a target="blank" href="https://archlinux.org/master-keys/">main keys</a>
+
+.. |pacman.conf#package_and_database_signature_checking| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/pacman.conf.5#PACKAGE_AND_DATABASE_SIGNATURE_CHECKING">pacman.conf#PACKAGE_AND_DATABASE_SIGNATURE_CHECKING</a>
+
+.. |pacman-key| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/pacman-key.8">pacman-key</a>
+
+.. |archlinux-keyring| raw:: html
+
+  <a target="blank" href="https://gitlab.archlinux.org/archlinux/archlinux-keyring/">archlinux-keyring</a>
+
+.. |sequoia| raw:: html
+
+  <a target="blank" href="https://sequoia-pgp.org/">sequoia</a>
+
+.. |archlinux-keyring package| raw:: html
+
+  <a target="blank" href="https://archlinux.org/packages/core/any/archlinux-keyring/">archlinux-keyring package</a>
+
+.. |Levente Polyak| raw:: html
+
+  <a target="blank" href="https://leventepolyak.net/">Levente Polyak</a>
+
+.. |dynamic linking| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Dynamic_linker">dynamic linking</a>
+
+.. |soname| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Soname">soname</a>
+
+.. |application binary interface| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Application_binary_interface">application binary interface</a>
+
+.. |find-libprovides| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/find-libprovides.1">find-libprovides</a>
+
+.. |find-libdeps| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/find-libdeps.1">find-libdeps</a>
+
+.. |gdb| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/gdb.1">gdb</a>
+
+.. |debug packages and debuginfod| raw:: html
+
+  <a target="blank" href="https://archlinux.org/news/debug-packages-and-debuginfod/">debug packages and debuginfod</a>
+
+.. |pkgbuild#install| raw:: html
+
+  <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#install">PKGBUILD#install</a>
+
+.. |user identifier| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/User_identifier">user identifier</a>
+
+.. |group identifier| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Group_identifier">group identifier</a>
+
+.. |UID/ GID database| raw:: html
+
+  <a target="blank" href="https://wiki.archlinux.org/title/DeveloperWiki:UID_/_GID_Database">UID/ GID database</a>
+
+.. |systemd| raw:: html
+
+  <a target="blank" href="https://systemd.io/">systemd</a>
+
+.. |systemd-sysusers| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/systemd-sysusers.8">systemd-sysusers</a>
+
+.. |extended file attributes| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Extended_file_attributes#Linux">extended file attributes</a>
+
+.. |setuid| raw:: html
+
+  <a target="blank" href="https://en.wikipedia.org/wiki/Setuid">setuid</a>
+
+.. |chown| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/chown.1">chown</a>
+
+.. |systemd-tmpfiles| raw:: html
+
+  <a target="blank" href="https://man.archlinux.org/man/systemd-tmpfiles.8">systemd-tmpfiles</a>
+
+.. |php| raw:: html
+
+  <a target="blank" href="https://www.php.net/">PHP</a>
+
+.. |python| raw:: html
+
+  <a target="blank" href="https://www.python.org/">Python</a>
+
+.. |ruby| raw:: html
+
+  <a target="blank" href="https://www.ruby-lang.org/en/">Ruby</a>
+
+.. |c| raw:: html
+
+  <a target="blank" href="https://www.iso.org/standard/74528.html">C</a>
+
+.. |c++| raw:: html
+
+  <a target="blank" href="https://isocpp.org/">C++</a>
+
+.. |rust| raw:: html
+
+  <a target="blank" href="https://www.rust-lang.org/">Rust</a>
+
+.. |#archlinux| raw:: html
+
+  <a target="blank" href="ircs://irc.libera.chat/archlinux">#archlinux</a>
+
+.. |#archlinux-aur| raw:: html
+
+  <a target="blank" href="ircs://irc.libera.chat/archlinux-aur">#archlinux-aur</a>
+
+.. |libera.chat| raw:: html
+
+  <a target="blank" href="https://libera.chat/">libera.chat</a>
+
+.. |list of my packages| raw:: html
+
+  <a target="blank" href="https://archlinux.org/packages/?sort=&q=&maintainer=dvzrv&flagged=">list of my packages</a>
+
+.. |unofficial user repositories| raw:: html
+
+  <a target="blank" href="https://wiki.archlinux.org/title/Unofficial_user_repositories">unofficial user repositories</a>
+
+.. |[realtime]| raw:: html
+
+  <a target="blank" href="https://wiki.archlinux.org/title/Unofficial_user_repositories#realtime">[realtime]</a>
+
+.. |[pro-audio-legacy]| raw:: html
+
+  <a target="blank" href="https://wiki.archlinux.org/title/Unofficial_user_repositories#pro-audio-legacy">[pro-audio-legacy]</a>
+
+.. |getting involved| raw:: html
+
+  <a target="blank" href="https://wiki.archlinux.org/title/Getting_involved">getting involved</a>
+