diff options
Diffstat (limited to 'posts/2022')
| -rw-r--r-- | posts/2022/arch-a-recap.rst | 294 | ||||
| -rw-r--r-- | posts/2022/managing-binary-package-repositories.rst | 193 | ||||
| -rw-r--r-- | posts/2022/new-pgp-key-id-1793dad5d803a8ffd7451697bb992f9864fad168.rst | 182 | ||||
| -rw-r--r-- | posts/2022/packaging-for-arch-linux.rst | 1152 |
4 files changed, 1821 insertions, 0 deletions
diff --git a/posts/2022/arch-a-recap.rst b/posts/2022/arch-a-recap.rst new file mode 100644 index 0000000..9ac0b19 --- /dev/null +++ b/posts/2022/arch-a-recap.rst @@ -0,0 +1,294 @@ +.. title: Arch, a recap +.. slug: arch-a-recap +.. date: 2022-01-30 18:0:00 UTC+02:00 +.. tags: arch linux, pro-audio, packaging +.. category: archlinux +.. link: +.. description: +.. type: text + +One of the things, that has kept me (increasingly) busy over the past few years +is my involvement with the |linux distribution| |arch linux|. +While I have been using |linux| for probably about 14 years it is frankly hard to +pinpoint when exactly I went down the rabbit hole that this operating system/ +ecosystem/ community is (relevant |xkcd|). However, I can elaborate on my +motivation and where that got me. + +.. TEASER_END + +As a musician of a varying background (from band-based rock music to solo +performances on guitar or with a modular synthesizer) I have found myself +evaluating the available pieces of software that are commonly used in music +production (e.g. |daws| and |audio plug-ins|). Most of them are |proprietary| +and only available for Windows or macOS (both proprietary as well). During my +studies a lot of the software in use has also not been |free|, but was provided +by the university with a *student discount* (e.g. Operating Systems (OSes), +|ides| or certain types of |vpls|). +I got increasingly annoyed by dealing with intransparent proprietary OSes, +|vendor lock-in| schemes, paying for software updates and being driven into +software piracy for working with so-called *industry standards*. + +Some time during the studies for my B.Sc. I decided to "try out Linux", not +knowing what that would mean actually. So there I was, booting an |ubuntu| Live +CD and clicking around in an interface, that would install the OS alongside a +still existing Windows 7. At that point I did not yet know about the joys of +missing or failing |device drivers|. Many hard fights with the |x window +system| later I settled on |ubuntu studio| for a while, as it had many nice +audio related pieces of software available out-of-the-box. + +I have always been a person that is interested in *"how things work"*. Soon I +realized, that the Linux ecosystem consisted of people that thought quite +similarly. Through various (often distribution specific) online documentation, +forums, mailing lists and the documentation of software projects for the first +time I felt as if I could actually learn something that mattered, because it +was not sold in a box and instead had a community of like-minded people +gathering around it. +I found quite appealing that a lot of things were *not polished* and that some +things were *not easy*, because it provided the sense of achieving something or +*getting good at something*. Where in Windows land I would have reinstalled the +OS upon getting intermittent |bluescreens|, in Linux land I just kept reading +about a certain topic until I was able to fix it myself. + +Probably a year after dipping my toes into Linux I bought a new laptop (my +previous one decided to commit suicide by desoldering its GPU). At that time it +seemed like a good idea to not dual-boot anymore, as apart from an occasional +game session I was not using Windows anymore. +Due to a friend of one of my fellow students I got introduced to Arch Linux, +which that person used in a (to me) bizarrely minimal fashion by booting from a +USB stick, as the hard drive in his laptop was broken. + +After installing Arch Linux for the first time (probably around 2008) I +realized quite early on that I would rather continue using it instead of Ubuntu +Studio, which by that time had given me many issues due to unclear +configuration management, documentation and graphics driver handling. +The |arch wiki| already back then was an excellent source of knowledge (and +still is to this day). Although some articles lack structure or are sometimes +outdated, it is this trove of knowledge that I had been missing with less +hands-on distributions up to then. +The |aur| offered an entrypoint for me to understand how distribution |package +management| works and how to build my own packages, that I could install using +the |pacman| package manager. This new found knowledge made me realize how +inefficient my work on Windows had been in the past (a realization that had not +fully dawned upon me on Ubuntu Studio as their update model follows a certain +cadence in which new versions of softwares are updated and otherwise only gain +bugfix releases). I was suddenly enabled to build packages myself of software +that I liked and used and I was able to alter existing packages to e.g. fix +problems or to extend their functionality. I was enabled to upgrade my entire +system with a single command, while on Windows I had to scavenge websites to +download installers and run those without knowing whether they were compatible. + +As my main focus shifted even further towards pro-audio applications, |dsp| +languages and recording practices with my M.Sc., I of course do have to mention +the accumulation of knowledge that |linux audio| represents (e.g. with the +|linux audio wiki|). It is an invaluable, although by now partially outdated +and not actively maintained point of - often distribution agnostic - +information on all things audio on Linux. This ecosystem is what drove and/ or +documented the development and use of many of the projects at the center of +what it means to achieve low-latency audio on Linux such as |jack|. + +After 2011 I started to conceptualize the Arch Linux distribution also as a +social construct, that relies on individuals to develop and change it (by +|getting involved|). While Arch is a distribution that can be used for many +purposes, the lack of structure or features in e.g. the pro-audio section was +apparent (although partially covered by packages in the AUR). I was able to do +most of the projects I was working on at that time using |supercollider| but +sometimes ran into issues that e.g. Ubuntu Studio had solved in a distribution +specific way (e.g. by providing packages for the |realtime kernel| to achieve +very low latencies while recording). + +Around 2017 it became apparent that many of the packages that I was using on a +day-to-day basis were starting to fall behind on updates or had not been +updated in a long time. It turned out that two packagers that had been active +packaging audio related software were missing in action or were extremely +unresponsive. I realized, that unlike a company supported endeavor, Arch Linux +was a community driven effort and as such required help. + +At the time of writing, Arch Linux consists of |arch linux developers|, |arch +linux trusted users| and |arch linux support staff|. While the first group +somewhat defines the direction of the distribution itself and takes care of the +most centric package repositories (``[core]`` and ``[extra]``), the second +group historically has been an additional group of packagers that takes care of +the ``[community]`` repository and the AUR and the last group is keeping a lot +of the infrastructure (e.g. wiki, forums, mailing lists, IRC channels, servers, +hosted applications) alive. + +For outsiders it is possible to join the Support Staff by starting to help +maintain infrastructure together with the existing staff. +When looking at the Trusted Users group the barrier of entry is a bit higher, +as the person gains access to package repositories that are used by thousands +of users. As such newcomers have to follow a vetting process and be sponsored +by existing group members (this is similar, but much less clearly defined, for +Developers). + +At the end of 2017 I was able to apply as Trusted User with the help of Ray +Rashif, who had been maintaining the packages I wanted to help improve and +update. + +Admittedly, the application process has been a bit daunting at the time, but it +also made me realize, how much I could still improve my package scripts and the +amount of work that is sometimes required to get things right. + +Since becoming a Trusted User in 2017 I have added many new packages in the +context of a pro-audio package group and spent time on various other packaging +areas as well. +In 2019 I have been promoted to Developer and spent some more time on +non-packaging topics on the side, such as infrastructure improvements +(|releng|, |arch-release-promotion| and |arch-repo-management|), installation +medium (|archiso|) and various community related topics (setting up an |rfc +process| and help holding |arch linux conferences|). + +I will follow up on packaging and best practices in an upcoming article +(hopefully in the not so distant future) and shed some light on the +non-packaging topics in further articles as well. + +That's it for now! I hope you enjoyed reading a user story, that eventually led +to getting more involved with an international community of (very talented) +developers working on a challenging project with many facets. As you can +imagine there are always many more stories to tell, but I tried to give a +general overview. Maybe I could even get someone interested in reaching out and +lending a hand in further developing this amazing beast that is Arch Linux :-) + +.. |linux distribution| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Linux_distribution">Linux distribution</a> + +.. |arch linux| raw:: html + + <a target="blank" href="https://www.archlinux.org">Arch Linux</a> + +.. |linux| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Linux">Linux</a> + +.. |xkcd| raw:: html + + <a target="blank" href="https://xkcd.com/456/">XKCD</a> + +.. |proprietary| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Proprietary_software">proprietary</a> + +.. |daws| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Digital_audio_workstation">Digital Audio Workstations (DAWs)</a> + +.. |audio plug-ins| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Audio_plug-in">audio plugin-ins</a> + +.. |free| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Free-software_license">free</a> + +.. |ides| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Integrated_development_environment">Integrated Development Environments (IDEs)</a> + +.. |vpls| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Visual_programming_language">Visual Programming Languages (VPLs)</a> + +.. |vendor lock-in| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Vendor_lock-in">vendor lock-in</a> + +.. |ubuntu| raw:: html + + <a target="blank" href="https://ubuntu.com/">Ubuntu</a> + +.. |device drivers| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Device_driver">device drivers</a> + +.. |x window system| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/X_Window_System">X Window System</a> + +.. |ubuntu studio| raw:: html + + <a target="blank" href="https://ubuntustudio.org/">Ubuntu Studio</a> + +.. |bluescreens| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Blue_Screen_of_Death">bluescreens</a> + +.. |arch wiki| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/">Arch Wiki</a> + +.. |aur| raw:: html + + <a target="blank" href="https://aur.archlinux.org">AUR</a> + +.. |package management| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Package_manager">package management</a> + +.. |pacman| raw:: html + + <a target="blank" href="https://archlinux.org/pacman/">pacman</a> + +.. |dsp| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Digital_signal_processing">DSP</a> + +.. |linux audio| raw:: html + + <a target="blank" href="https://linuxaudio.org/">Linux Audio</a> + +.. |linux audio wiki| raw:: html + + <a target="blank" href="https://wiki.linuxaudio.org/">Linux Audio Wiki</a> + +.. |jack| raw:: html + + <a target="blank" href="https://jackaudio.org/">JACK</a> + +.. |getting involved| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/Getting_involved">getting involved</a> + +.. |supercollider| raw:: html + + <a target="blank" href="https://supercollider.github.io/">SuperCollider</a> + +.. |realtime kernel| raw:: html + + <a target="blank" href="https://wiki.linuxfoundation.org/realtime/start">realtime kernel</a> + +.. |arch linux developers| raw:: html + + <a target="blank" href="https://archlinux.org/people/developers/">Developers</a> + +.. |arch linux trusted users| raw:: html + + <a target="blank" href="https://archlinux.org/people/trusted-users/">Trusted Users</a> + +.. |arch linux support staff| raw:: html + + <a target="blank" href="https://archlinux.org/people/support-staff/">Support Staff</a> + +.. |releng| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/releng/">releng</a> + +.. |arch-release-promotion| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/arch-release-promotion/">arch-release-promotion</a> + +.. |arch-repo-management| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/arch-repo-management/">arch-repo-management</a> + +.. |archiso| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/archiso/">archiso</a> + +.. |rfc process| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/rfcs/">RFC process</a> + +.. |arch linux conferences| raw:: html + + <a target="blank" href="https://conf.archlinux.org/">Arch Linux Conferences</a> + diff --git a/posts/2022/managing-binary-package-repositories.rst b/posts/2022/managing-binary-package-repositories.rst new file mode 100644 index 0000000..89f203f --- /dev/null +++ b/posts/2022/managing-binary-package-repositories.rst @@ -0,0 +1,193 @@ +.. title: Managing binary package repositories +.. slug: managing-binary-package-repositories +.. date: 2022-07-02 13:19:59 UTC+02:00 +.. tags: arch linux, packaging, repod, dbscripts +.. category: archlinux +.. link: +.. description: +.. type: text + +In `Packaging for Arch Linux +<https://sleepmap.de/2022/packaging-for-arch-linux>`_ I described the ins and +outs of binary repository management and some of the issues that come with the +tooling currently used by |arch linux|. + +In this article I will highlight the work on new tooling and its features. + +Since my last write-up on this topic, the project formerly known as +``arch-repo-management`` has been renamed to ``repod`` (as in *repo-d*) and has +just seen its first minor release. 🎉 + +You can find its documentation at https://repod.archlinux.page. + +.. TEASER_END + +.. note:: + + The https://archlinux.page domain has recently been acquired by Arch Linux to + serve as the common platform for documentation of projects driven by the + distribution. + +Please note, that |repod| 0.1.0 is still alpha grade software and **should +not** be used to actually manage binary package repositories at this point in +time! + +However, it is already possible to do a few things with the software and if you +are able to test it or are interested in helping develop it, that is very much +welcomed! + +On Arch Linux you can install it using |pacman|: + +.. code:: sh + + pacman -S repod + +On other distributions (even on macOS!) you may install it using |pip| (|repod| +is |available on pypi|) until your respective package management system makes +the software available to you on a system level: + +.. code:: sh + + pip install --user repod + +Working on repod +---------------- + +Since 2021 I have been on and off working on what is now |repod|. The project +is written in typed |python| and is extensively tested using |pytest|. + +Work first began after |arch conf| 2019, at which a working group had looked +into improving the workflows currently employed by the distribution and pushing +for tooling that would allow moving away from an |svn| monorepo based approach +to a deconstructed |git| setup. + +A proof of concept (PoC) to mimic the behavior of |dbscripts| had been created, +but after 2019 this work laid dormant. + +Over 2021 I have spent time to transform parts of the PoC into a Python project +following best practices for development (e.g. type hints following |pep0484|, +100% test coverage, data validation using |pydantic| models), exposing first +features in scripts. + +In 2022 more work has been done to extend validation and transform the project +into a package based setup for easier handling and extension in the future. + +Concepts of repod +----------------- + +Contrary to |dbscripts|, |repod| follows a paradigm in which it is largely +decoupled from the |source repository| of the binary packages it maintains and +aims at becoming a self-contained service. + +Package files and their signatures are consumed, relevant metadata is extracted +and transferred to a |management repository|, which is where the state of +each |binary repository| is kept. + +Available packages in a given |binary repository| are exposed to |pacman| via +|sync database| files. The |management repository| contents can be +(transparently and reproducibly) transformed into |sync database| files and +vice versa. + +The above functionality is exposed on the command line via |repod-file|. + +Upcoming work +------------- + +As mentioned above, |repod| is not yet stable and still misses quite a few +features. +The following topics (and more) will be worked on in the next milestones (not +necessarily in this order): + +* file handling (moving package files from staging areas to actual repositories) +* signature validation (PGP signature validation of package files) +* handling of debug packages +* consolidate schema of |management repository| +* improve logging throughout the project +* git backend to transparently expose changes to the |management repository| +* caching for |management repository| state (e.g. to allow fast searches) +* API to interact with a |repod| instance over the wire in an authenticated fashion +* client-side tooling to interact with |repod|'s API + +There is still a lot of work to be done, so if you have a background in +|Python| development and are interested in working on a project that is very +close to the distribution and will likely improve the workflow of many people +while making binary repository management more transparent to the user, have a +look at the project's |contribution guidelines| and do not hesitate to reach +out! + +.. |arch linux| raw:: html + + <a target="blank" href="https://archlinux.org">Arch Linux</a> + +.. |repod| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/repod">repod</a> + +.. |pacman| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/pacman.8">pacman</a> + +.. |pip| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/pip.1">pip</a> + +.. |available on pypi| raw:: html + + <a target="blank" href="https://pypi.org/project/repod/">available on pypi</a> + +.. |python| raw:: html + + <a target="blank" href="https://www.python.org/">Python</a> + +.. |pytest| raw:: html + + <a target="blank" href="https://docs.pytest.org/en/latest/">pytest</a> + +.. |arch conf| raw:: html + + <a target="blank" href="https://conf.archlinux.org/">Arch Conf</a> + +.. |svn| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/svn.1">svn</a> + +.. |git| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/git.1">git</a> + +.. |dbscripts| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/dbscripts">dbscripts</a> + +.. |pep0484| raw:: html + + <a target="blank" href="https://peps.python.org/pep-0484/">PEP 0484</a> + +.. |pydantic| raw:: html + + <a target="blank" href="https://pydantic-docs.helpmanual.io/">pydantic</a> + +.. |source repository| raw:: html + + <a target="blank" href="https://repod.archlinux.page/repositories/source_repository.html">source repository</a> + +.. |management repository| raw:: html + + <a target="blank" href="https://repod.archlinux.page/repositories/management_repository.html">management repository</a> + +.. |binary repository| raw:: html + + <a target="blank" href="https://repod.archlinux.page/repositories/binary_repository.html">binary repository</a> + +.. |sync database| raw:: html + + <a target="blank" href="https://repod.archlinux.page/repositories/sync_database.html">sync database</a> + +.. |repod-file| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/repod-file.1">repod-file</a> + +.. |contribution guidelines| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/repod/-/blob/79a0fa9aa4a4d9def6dad24805f9bbff0388f734/CONTRIBUTING.md">contribution guidelines</a> + diff --git a/posts/2022/new-pgp-key-id-1793dad5d803a8ffd7451697bb992f9864fad168.rst b/posts/2022/new-pgp-key-id-1793dad5d803a8ffd7451697bb992f9864fad168.rst new file mode 100644 index 0000000..15ce8fb --- /dev/null +++ b/posts/2022/new-pgp-key-id-1793dad5d803a8ffd7451697bb992f9864fad168.rst @@ -0,0 +1,182 @@ +.. title: New PGP key ID 1793DAD5D803A8FFD7451697BB992F9864FAD168 +.. slug: new-pgp-key-id-1793dad5d803a8ffd7451697bb992f9864fad168 +.. date: 2022-04-30 10:35:57 UTC+02:00 +.. tags: chain of trust, gnupg, gpg, infrastructure, openpgp, sequoia, sq, web key directory, web of trust +.. category: admin +.. link: +.. description: +.. type: text + +As my current |PGP| key ``91BD8815FE0040FA7FF5D68754C28F4FF5A1A949`` will be +expired soon, I have created a new one to replace it. + +You can get my new key ``1793DAD5D803A8FFD7451697BB992F9864FAD168`` as well as +the old one and the cross-signatures required to establish the |chain of trust| +between the two via Web Key Directory (|WKD|) (which should be used +automatically by ``gpg >= 2.1.23``). + +To not deal with the rather convoluted |gnupg| tooling I have created a +deployment method for this using |sequoia-pgp|'s |sq|, about which you can read +in the rest of this article. + +.. TEASER_END + +Key servers +=========== + +Historically, there has been a set of |key servers|, which have been used to +distribute the public keys of users centrally and/ or in a synchronized +fashion. These key servers have been widely relied upon, but they suffer(ed) +from a lot of issues in regards to privacy, stability and speed. Most notably +the Synchronized Key Server (|SKS|) system collapsed under its technical debt +and had to be shut down. + +To this day there are still other large, non-synchronized keyserver systems +around (e.g. |hockeypuck|, which drives https://keyserver.ubuntu.com), but they +all suffer from the fact, that a large centralized setup, in which keys are +only ever appended, does not scale. + +Additionally, not all keyserver systems support all key types or signatures on +keys, which is problematic, as they can not reflect upon |chain of trust| +between two or more keys. This is very problematic for the |web of trust| of +|PGP|. + +Web Key Directory +================= + +At the time of writing the |WKD| system is formalized in +|draft-koch-openpgp-webkey-service-11|. It describes a decentralized way of +providing public key material via a given domain's webserver, by exposing +specially crafted files. + +WKD Deployment +============== + +Personally, I believe that the |gpg| commandline interface is incredibly +convoluted and very complex. I therefore used |sequoia-pgp|'s |sq| instead to +combine PGP public key material and prepare the required directory structure. + +In my personal |wkd| project I assembled a simple system consisting of easy to +prepare directories in which to place PGP public keys, which are converted to a +keyring using |sq keyring join|, converted into a |WKD| structure using |sq wkd +generate| and synchronized using |rsync|. + +As I do not provide an ``openpgpkey`` subdomain, I am using a direct domain +directory structure (see |WKDHosting| for further details). + +Using keys from WKD +=================== + +In theory all that is required for |gpg| to make use of |WKD| is a version +``>=2.1.23``. However, its use can be somewhat confusing: + +.. code:: sh + + gpg --locate-keys dave@sleepmap.de + +The above only returns the new key +``1793DAD5D803A8FFD7451697BB992F9864FAD168``, but not the old +``91BD8815FE0040FA7FF5D68754C28F4FF5A1A949``. It is entirely opaque to the user +as to why. + +Meanwhile with |sq wkd get| it is possible to return the public key material of +both keys: + +.. code:: sh + + sq wkd get dave@sleepmap.de + +The certificates can be inspected directly using |sq inspect|: + +.. code:: sh + + sq inspect --certifications <(sq wkd get dave@sleepmap.de) + +Additionally, the certificates can be imported into an existing |gnupg| based +keyring: + +.. code:: sh + + gpg --import <(sq wkd get dave@sleepmap.de) + + +Using |WKD|, the need for providing a separate PGP public key file in the +context of this website has been made obsolete and I have therefore instead +replaced it with plain mentions of the PGP key IDs on the `about page +<https://sleepmap.de/about/>`_. + +.. |PGP| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Pretty_Good_Privacy">PGP</a> + +.. |chain of trust| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Chain_of_trust">chain of trust</a> + +.. |web of trust| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Web_of_trust">web of trust</a> + +.. |WKD| raw:: html + + <a target="blank" href="https://wiki.gnupg.org/WKD">WKD</a> + +.. |gnupg| raw:: html + + <a target="blank" href="https://gnupg.org/">gnupg</a> + +.. |sequoia-pgp| raw:: html + + <a target="blank" href="https://sequoia-pgp.org/">sequoia-pgp</a> + +.. |sq| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/sq.1">sq</a> + +.. |key servers| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Key_server_(cryptographic)">key servers</a> + +.. |SKS| raw:: html + + <a target="blank" href="https://github.com/SKS-Keyserver/sks-keyserver">SKS</a> + +.. |hockeypuck| raw:: html + + <a target="blank" href="https://github.com/hockeypuck/hockeypuck">hockeypuck</a> + +.. |draft-koch-openpgp-webkey-service-11| raw:: html + + <a target="blank" href="https://tools.ietf.org/id/draft-koch-openpgp-webkey-service-11.html">draft-koch-openpgp-webkey-service-11</a> + +.. |gpg| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/gpg.1">gpg</a> + +.. |wkd| raw:: html + + <a target="blank" href="https://git.sleepmap.de/dave/wkd.git/">wkd</a> + +.. |sq keyring join| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/sq-keyring-join.1">sq keyring join</a> + +.. |sq wkd generate| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/sq-wkd-generate.1">sq wkd generate</a> + +.. |rsync| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/rsync.1">rsync</a> + +.. |WKDHosting| raw:: html + + <a target="blank" href="https://wiki.gnupg.org/WKDHosting">WKDHosting</a> + +.. |sq wkd get| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/sq-wkd-get.1">sq wkd get</a> + +.. |sq inspect| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/sq-inspect.1">sq inspect</a> diff --git a/posts/2022/packaging-for-arch-linux.rst b/posts/2022/packaging-for-arch-linux.rst new file mode 100644 index 0000000..0b27eeb --- /dev/null +++ b/posts/2022/packaging-for-arch-linux.rst @@ -0,0 +1,1152 @@ +.. title: Packaging for Arch Linux +.. slug: packaging-for-arch-linux +.. date: 2022-04-06 13:22:53 UTC+02:00 +.. tags: arch linux, packaging, reproducible builds, arch-repo-management, dbscripts +.. category: archlinux +.. link: +.. description: +.. type: text + +In `Arch, a recap <https://sleepmap.de/2022/arch-a-recap>`_ I elaborated a bit +on my reasons for getting involved with Arch Linux. In this post I would like +to highlight a few technical details and give a "behind the scenes" when it +comes to packaging on and for Arch Linux. +This post is written from the viewpoint of a distribution packager, but it +is likely to contain information also useful to people packaging on different +distributions or for private purposes. + +.. TEASER_END + +|arch linux| is a |linux distribution|, that offers binary packages in +|software repositories| (aka. repos). To achieve this, packages are built from +source files using tooling that is developed by the distribution and various +volunteers. The resulting binary packages are then provided to users on mirrors +of the distribution (i.e. package files and their cryptographic signatures are +provided by |web servers|) and are downloaded, verified, validated and +installed using a |package manager|. + +.. note:: + + Other distributions may use different concepts. E.g. |gentoo linux| offers + installation media that is used to install a base system. From then on users + rebuild the software on their systems themselves based on distribution + provided source files. There are no binary packages for users to install. + +The upsides of a central software package system facilitating binary repos are + +- users do not have to build the software on their systems themselves, which + e.g. for web browsers can take a very long time and eat a lot of energy +- software for the entire system can be updated with one command and only takes + as long as download and extraction of a given set of packages + +Packages +======== + +When looking at the concept of binary software packages it probably helps to +consider the point of view from e.g. Windows and macOS, which both provide +software to users in different ways and give a good case for comparison. In +case you already know how binary packages function and compare, skip this +section. + +For brevity I will skip the proprietary app stores in the below examples as +they abstract the concept of software installation to the point where this is +opaque to the user and delivers no direct comparison in the context of packages +(while under the hood most app stores use the below mentioned technologies). + +Windows +------- + +On Windows software is usually provided by the means of an installer (e.g. +shipped as a ``.exe`` or ``.msi`` file). An installer usually needs to be +downloaded from thirdparty websites (often without verification) and then +executed one-by-one. The installer often already contains the (prebuilt) files +to be installed (sometimes files are also downloaded by the installer +application on-the-fly), offers some form of modification (e.g. the +installation location), installs the bundled or downloaded files and modifies +the system's registry (e.g. for auto-start or other features). Although +Microsoft has attempted to consolidate its installation backends, the user +experience is usually still a mixed bag. +System updates (those modifying the operating system) are handled by the OS +itself and the user usually has not much of a say when/ how that happens (this +can be modified to some extent). Additionally, some hardware may not use the +latest version of Windows due to software-based |planned obsolescence|. + +MacOS +----- + +On macOS software can be installed using images or installers (shipped as e.g. +``.dmg`` and ``.pkg`` respectively). The download of the files in question +usually functions in the same way as it does on Windows (unverified downloads +from thirdparty websites). Where with images the user experience is usually +*"drag and drop"* from a mounted image to the list of applications, installers +on the other hand offer similar functionality to how installers work on Windows +(e.g. setup auto-starting). +System updates, similar to those on Windows are handled by macOS itself and +also here software enforced |planned obsolescence| is a thing. + +Looking at the above examples it becomes clear, that automation on both +platforms is quite terrible: The distinction between OS updates and *"other +software"* leads to a mix and match approach towards updates, that is (if at +all) only partly remedied by externally developed and provided package managers +for some of the *"other software"* (e.g. |homebrew| or |chocolatey|), but at +best remains a fragmented experience for the user. + +Linux +----- + +On distributions that offer binary package repositories, users use a package +manager to install packages and to upgrade **all software** [1]_ on their +system. +Packages are essentially |archive files|, that are downloaded, verified and +extracted by the package manager. As the files contained in (distribution) +packages follow a well-defined location schema (e.g. |filesystem hierarchy +standard| or |file-hierarchy|), the system can check for file conflicts and +users can usually have reasonable assumptions about where files of a package +are located (package managers usually also track the files of all packages). +Additional functionality, such as post install scripts (e.g. to create users or +to change ownership on files) are usually +contained in package files and executed after installation. However, on systemd +based distributions, much of the post installation tasks have been streamlined +with the help of |sysusers.d| and |tmpfiles.d| (more on that later). Some +distributions also make use of non-standardized hooks (see |alpm-hooks| for how +this is implemented for |pacman|), that are used by the package manager for +certain tasks on files that are not owned by one specific package (e.g. update +font cache). + +Build tooling +============= + +The most basic build tooling for Arch Linux - |makepkg| - is bundled with +|pacman_website| (the package manager used to install all software packages on the +distribution). It is used in conjunction with a |PKGBUILD|, which as a package +source file describes where/ how to get a package's source files (and in which +version), how to build and test it (if applicable) and how/ where to install +it. + +In case you have experience with |bash|: Both ``makepkg`` and ``PKGBUILDs`` are +written in it. + +When building packages with plain ``makepkg``, the built package will be +created in the context of the user's system and as such will make use of the +software available on the user's system. While this works (given all +dependencies are met) it is not recommended to do so, as the user's +system may use custom packages or settings to ``makepkg`` (see |makepkg.conf|), +that can alter the outcome of the build, which may make the package unusable on +another system. + +Clean chroots +============= + +To enable builds, that are done in a clean environment (i.e. one that only has +official distribution packages installed and does not depend on configuration +or custom packages on a local system), Arch Linux and various contributors have +created special build tooling, which is contained in the |devtools| project. + +With the help of |makechrootpkg| one can run ``makepkg`` in a |systemd-nspawn| +based |chroot|, which will only have the packages installed, that are required +for building, testing and running a given package. + +Using |makechrootpkg| and its various repository-specific symlinks is how Arch +Linux packagers build all packages in the official repositories. + +.. note:: + + It is generally advisable to build **all packages** in a clean environment. + +An implict upside of using ``makechrootpkg`` to build packages is, that +|checkpkg| and |namcap| are being run on the resulting package, which can give +valuable hints at possible improvements of the package. + +Building packages +================= + +First off: There are sometimes a lot of subtleties involved with packaging and +especially producing packages that are of good quality. In the following +sections I will discuss a few tools and packaging specifics, that may seem +quite overwhelming or complicated at first. Luckily, a lot of the tooling is +fairly well documented and it is probably always good to remember, that +everyone is a learner and that as the tooling and the best practices evolve, +this is an open-ended topic. + +A good starting point is always to use ``makechrootpkg`` and to adhere to the +|arch package guidelines|. + +.. note:: + + There is the |arch package guidelines category| of more specific guidelines + for various programming languages and special use-cases. + +Getting package build sources +----------------------------- + +The act of getting the sources for a binary package is described in the context +of the |arch build system| (ABS). While users without write access to the Arch +Linux source repositories can rely upon |asp| to get to the package build +sources, the official packagers rely on a rather organically grown packaging +workflow, that is described in |howto be a packager|. + +At the time of writing, Arch Linux still relies on two monolithic |svn| +repositories for the package build sources (one for the ``[core]`` and +``[extra]`` repositories and one for the ``[community]`` and ``[multilib]`` +repositories) which are exported to |git| via |git-svn| on the official Arch +Linux Github organization (|svntogit-packages| and |svntogit-community|, +respectively). + +.. note:: + + Work is underway to switch to |git| for the package build sources. However, + this has implications for maintaining state of resulting binary repositories + (this is currently done in the svn repositories). While |dbscripts| has been + used for managing the state for many years, it is likely to be replaced by + |arch-repo-management| in the future. + Additionally, it should be mentioned, that such a switch is not trivial after + 20 years, if one is also concerned with the technical feasibility and + maintainability of the tooling in use. + +PKGBUILDs +--------- + +As mentioned earlier, |PKGBUILD| files are really just |bash| scripts, that are +being evaluated by |makepkg|. As such they define a few variables and functions +(some of which are required, others only being optional). + +The below example shows a bare minimum example, derived from the prototype +files, that can be found in ``/usr/share/pacman/``: + +.. code:: sh + + # Maintainer: Your Name <youremail@domain.com> + pkgname=dummy-package + pkgver=0.1.0 + pkgrel=1 + pkgdesc="A dummy package" + arch=(any) + url="https://my-upstream.link/to/dummy-package" + license=(GPL3) + depends=(another-package) + optdepends=('some-additional: for additional feature X') + source=(https://my-upstream.link/to/$pkgname-$pkgver.tar.gz) + b2sums=('THISISADUMMYCHECKSUM') + + package() { + make DESTDIR="$pkgdir" install -C $pkgname-$pkgver + } + +To go through the essentials of this very minimalistic example, which assumes +that we have a project using |make| to install a few files: + +- While the ``Maintainer`` comment is technically not required, it is always + helpful for others trying to contact the author of a given package build + source +- ``pkgname``: The name of the package. Refer to the wiki section + |pkgbuild#pkgname| for further info (e.g. restrictions) +- ``pkgver``: The (upstream) version of the package. Refer to the wiki section + |pkgbuild#pkgver| for further info (e.g. restrictions) +- ``pkgrel``: The release version of the package, which identifies the build of + the particular package in version ``pkgver``. This is a string specific to + Arch Linux (see |pkgbuild#pkgrel|) and *is not related to the upstream + version* of the software. +- ``pkgdesc``: A short description of what this package provides +- ``arch``: The architecture of the resulting package. As this is an array, it + can contain several entries (``makepkg`` will envoke a build for each + architecture). At the time of writing Arch Linux only supports the ``x86_64`` + and ``any`` architectures. +- ``url``: The URL of the upstream project (e.g. a website or a link to the + |version control| sources) +- ``license``: The licenses that apply to the project. This again is an array + and may contain several licenses. In case licenses that are not covered by + the |licenses package| are encountered, their license files must be installed + in the ``package()`` function (refer to the wiki section |pkgbuild#license| + for further information). +- ``depends``: An array of runtime dependencies for the package. They will be + installed automatically during build when building with ``makechrootpkg`` or + ``makepkg -s``. +- ``optdepends``: An array of optional dependencies and a short description + about their purpose. These packages will not be installed during build time + (for this ``makedepends`` needs to be used). +- ``source``: An array of resources for ``makepkg`` to retrieve. As |makepkg| + is able to handle various |version control| systems, local and remote files, + as well as to rename files, it is advisable to read the relevant man page + section for ``makepkg``. +- ``b2sums``: An array of checksums for all resources in the ``source`` array. + It is advisable to use either (or all of) ``sh256sums``, ``sha512sums`` or + ``b2sums`` as older hashing mechanisms are by now unsafe (see + |pkgbuild#integrity|). The checksums are used to guard against changing (and + potentially malicious) upstream resources. The resources and checksums for a + new version of a given package may be retrieved and updated using + |updpkgsums| (contained in the |pacman-contrib| package). +- ``package()``: This function defines all steps necessary to install the files + of the upstream project to an empty location (represented by the *magic + variable* ``"$pkgdir"``), that will contain all installable files of the + package. This function is called using |fakeroot|, which means that to the + installing processes it looks like they are being executed by ``root``. + +PGP validation +-------------- + +Upstream project resources (e.g. signed source tarballs or git tags/ commits) +can be validated using |pgp|. + +.. note:: + + While other mechanisms are theoretically possible (e.g. |signify| or + |cosign|), they are currently not implemented in the context of |makepkg|. + +Technically all that is required for this is, +that the ``validpgpkeys`` array in the PKGBUILD contains at least one +retrievable PGP key ID and that the ``source`` array contains either a ``.sig`` +or ``.asc`` file valid for one of the resources, or that a git object to be +checked is targetted using the ``?signed`` identifier (see +|makepkg#signature_checking| and |pkgbuild#using_vcs_sources|). + +Although it is advisable to have cryptographic signature validation (e.g. +using |pgp|) for releases, this should only be considered under the following +circumstances in regards to an upstream project: + +* there is a track record of signing releases with the same key ID and the + project specifically provides the expectable key ID publicly (e.g. on the + website) +* keeps a |chain of trust| between multiple and/or successive key IDs +* no key easily used by multiple users is used (e.g. Github's PGP key, which + can be used by multiple users of a given Github project and is not handled by + the users themselves) + +The first point is usually easy to check up on, while the 2nd might require +getting in touch with the project developers if it happens (or happened in the +past) - this is the case more often than you would think and does block package +updates, as a new key ID must not be trusted without investigating the cause +for a missing |chain of trust| to prevent a potential |supply chain attack|! + +.. note:: + + I am contemplating to prepare an |arch linux rfc| for this, as the handling + of PGP signed sources is not well defined for the distribution and often + leads to mishandling (e.g. using new PGP key IDs without having a |chain of + trust|). + +The 3rd point practically provides a false sense of security: A PGP key signed +a release of a project, but in actuality multiple members of a project may have +access to this functionality. From the outside it is impossible to tell who +triggered a release and signed off on it (it could easily be malicious because +someone's Github account has been hacked). + +Reproducibility +--------------- + +Arch Linux as a distribution is committed to packages becoming bit-for-bit +reproducible (have a look at the overarching |reproducible builds| project for +more background information on the general topic). The status of the current +packages in the official repositories is tracked on +https://reproducible.archlinux.org, which is backed by |rebuilderd|. + +After building a package it can be rebuilt using |makerepropkg|, which may use +|diffoscope| on the resulting package in case it is not reproducible. + +As the use of ``makerepropkg`` requires the ``PKGBUILD`` used to build the +initial package, it can not be used when only a package file is available. +However, for that use-case |repro| may be used. + +.. note:: + + Both tools make use of the |buildinfo| files contained in each binary + package. + +Dealing with the strange +======================== + +In `building packages <#building-packages>`_ we have looked at some of the more +basic use-cases. The following sub-sections will deal with more uncommon or +very specific ones as well as problems at the intersection of build tooling and +binary repository management. + +Split packages +-------------- + +There are situations, in which one wants to build several packages from a +single ``PKGBUILD``. Those are usually: + +- the documentation of the project is very large +- certain features (e.g. language bindings) are not required by the main + application or use-case of the project +- specific functionality would require a large tree of dependencies but is not + required for the main application or use-case + +In all three cases this can be handled using a split package setup in which the +extra functionality (as a package) is declared an optional dependency of the +main package. + +To create a split package, the ``pkgname`` variable of the ``PKGBUILD`` is +turned into an array, containing multiple package names, while the ``pkgbase`` +variable (see |pkgbuild#pkgbase|) should be set. Additionally, the generic +``package()`` function needs to be split up into specific functions for each +package (``prepare()``, ``build()`` and ``check()`` are shared). + +Using the example from `PKGBUILDs <#pkgbuilds>`_, this is how it would look +like when e.g. splitting out documentation (assuming that the upstream project +provides separate install targets for the components). + +.. code:: sh + + # Maintainer: Your Name <youremail@domain.com> + pkgbase=dummy-package + pkgname=(dummy-package dummy-package-docs) + pkgver=0.1.0 + pkgrel=1 + pkgdesc="A dummy package" + arch=(any) + url="https://my-upstream.link/to/dummy-package" + license=(GPL3) + makedepends=(another-package) + source=(https://my-upstream.link/to/$pkgname-$pkgver.tar.gz) + b2sums=('THISISADUMMYCHECKSUM') + + package_dummy-package() { + depends=(another-package) + optdepends=( + 'dummy-package-docs: for documentation' + 'some-additional: for additional feature X' + ) + + make DESTDIR="$pkgdir" install-scripts -C $pkgname-$pkgver + } + + package_dummy-package-docs() { + make DESTDIR="$pkgdir" install-docs -C $pkgname-$pkgver + } + +Binary repository management +---------------------------- + +The resulting packages of a build process can be installed on a local machine, +but are often of course more useful, if more systems can install them as well. +For this purpose the repository sync databases exist, which |pacman| uses (see +|libalpm_databases|) to retrieve the difference between a remote package +repository and a local machine's state and to figure out which packages to +upgrade. + +The most rudimentary actions (adding and removing packages, optionally signing +a database) on a binary repository can be done using |repo-add| and +|repo-remove|, which are shipped with |pacman|. As the tooling is very basic, +it does not offer any form of state tracking (i.e. a log of actions, such as +additions or removals done to a sync database by a specific user). + +At the time of writing Arch Linux packagers make use of |dbscripts| for the +binary repository management, which also does (a form of) state tracking by +interacting with and using the the two |svn|-based monorepos for package build +sources for this purpose. +The tooling consists of a set of shell scripts (making use of |repo-add| and +|repo-remove| internally), that are being called by authorized users on a +specific host over |ssh|. The user authentification is therefore done using +|ssh| while the user authorization is implemented using plain unix groups +(different sets of packagers have access to ``[core]`` and ``[extra]`` vs. +``[community]`` and ``[multilib]`` - often only for historical reasons). +However, this setup is showing its age and comes with its own set of pitfalls: + +- changes to repositories are not externally auditable +- package data is only checked rudimentarily +- integrity of repository sync databases can not be guaranteed +- repository sync databases can not be rebuilt to a specific state +- setting the target binary repository for a package is a manual operation +- due to the blocking nature of dbscripts, it is possible to brick the state of + a repository if e.g. connection to the host running dbscripts is lost during + the move of packages between two + repositories +- it is not possible to setup rebuild-specific staging repositories on the fly +- many users need |ssh| access to a machine + +This all being said, work is underway with |arch-repo-management| to provide a +more manageable and easy to configure solution that runs as a service and does +not rely on multiple users to have direct access on a target system. +One of the project's main focusses is to be able to verify incoming package +data and to fully decouple the state from the repository sync databases (to be +able to rebuild them whenever needed). +Going forward it should become more easy to setup ephemeral staging +repositories to build against and safer to move data due to more atomic +repository operations, while allowing externals to audit each repository's +history. Currently the project is still far from being usable though and there +are quite a few things left to be implemented. Switching from the current setup +in which both package build sources and binary repository state are handled by +one |version control| system, to one where these concerns are separated is a +hard problem, especially when one wants to get this right. +I hope that going forward we will end up with a solution that can be easily +contributed to and reused also outside of Arch Linux. I will write another post +in the coming months, that highlights work and concepts of +|arch-repo-management|. + +Distributing trust +------------------ + +Packagers use the |makepkg.conf| variables ``PACKAGER`` and ``GPGKEY`` to set +the packager user ID (i.e. name and e-mail address) and the PGP key ID used for +signing created packages. + +Other users that wish to use packages signed by someone else need to import +that other user's PGP public key using |pacman-key|. + +Arch Linux maintains a |web of trust| between a set of main signing keys and +all packagers and between all packagers amongst themselves (see the |main keys| +page for an extensive overview). This setup allows for user systems to evaluate +whether a given package signature done by a packager is considered trusted (see +|pacman.conf#package_and_database_signature_checking| for further info). +These constructs are system-wide PGP keyrings for the use with |pacman| and can +be handled with |pacman-key|. + +.. note:: + + The main signing keys are considered **fully trusted** on a user system. They + define the root trust for the distribution which is handed down to the + packagers. + +In the |archlinux-keyring| project the distribution trust of Arch Linux is +maintained as a set of decomposed PGP public keys and the signatures on them. +The custom tooling ``keyringctl`` (which uses |sequoia|'s ``sq`` under the +hood) is used to maintain (e.g. import public keys and signatures) a PGP +keyring that is packaged in the |archlinux-keyring package| and which is +automatically added and updated upon install. + +More than or equal to three main signing key holders are required to uphold +the web of trust. More than or equal to three valid main key signatures are +required for a packager key (if it is itself still valid) to be allowed for +distributing packages in the official Arch Linux repositories. + +.. note:: + + Writing the new tooling ``keyringctl`` to manage the distribution trust has + been a huge topic of the past year, that |Levente Polyak| and I have been + working on, as the previous setup was very brittle. I will elaborate a bit + more on that topic in an upcoming post. + + +Sonames +------- + +Linux distributions mostly build C and C++ libraries and executables using +|dynamic linking|. This implies, that shared libraries usually provide a +|soname| (e.g. ``libexample.so.1``), which is in turn used (i.e. linked +against) by one or more other libraries or executables. +If the |application binary interface| (ABI) of the library in question changes, +its |soname| should be increased as well (e.g. ``libexample.so.2``). If a +package with an updated soname is released and installed, without rebuilding +any of the packages depending on it, those will fail to load (the now +non-existent) ``libexample.so.1`` shared object. + +A common task as a packager is therefore to do rebuilds for libraries and +executables when a soname change is introduced. Depending on the library +introducing the soname change or the library/executable being affected by it, +this is sometimes a bit of a painful and time consuming experience. +While it is not unheard of that projects either forget to introduce a soname +change (silently breaking consumers) or accidentally downgrade their soname, +consumers are more likely to run into trouble because of not yet implementing +changes introduced by the ABI change (requiring patches not yet included in a +stable release). + +To safeguard against cases in which soname changes went unnoticed and packages +are pushed to the repositories, it is possible to make use of |makepkg|'s +builtin dependency resolution. Extending upon the example in `PKGBUILDs +<#pkgbuilds>`_ and assuming that ``libexample`` is the package providing +``libexample.so``: + +.. code:: sh + + # Maintainer: Your Name <youremail@domain.com> + pkgname=libexample + pkgver=1.0.0 + pkgrel=1 + pkgdesc="A dummy library" + arch=(any) + url="https://my-upstream.link/to/libexample" + license=(GPL3) + depends=(glibc) + provides=(libexample.so) + source=(https://my-upstream.link/to/$pkgname-$pkgver.tar.gz) + b2sums=('THISISADUMMYCHECKSUM') + + build() { + make -C $pkgname-$pkgver + } + + package() { + make DESTDIR="$pkgdir" install -C $pkgname-$pkgver + } + +.. code:: sh + + # Maintainer: Your Name <youremail@domain.com> + pkgname=dummy-package + pkgver=0.1.0 + pkgrel=1 + pkgdesc="A dummy package" + arch=(any) + url="https://my-upstream.link/to/dummy-package" + license=(GPL3) + depends=(another-package libexample libexample.so) + optdepends=('some-additional: for additional feature X') + source=(https://my-upstream.link/to/$pkgname-$pkgver.tar.gz) + b2sums=('THISISADUMMYCHECKSUM') + + build() { + make -C $pkgname-$pkgver + } + + package() { + make DESTDIR="$pkgdir" install -C $pkgname-$pkgver + } + +If during build time ``libexample`` provided ``libexample.so.1``, the resulting +``dummy-package`` will now depend on ``libexample`` and ``libexample.so=1-64``, +which ``libexample`` provides. + +If the ``libexample`` package is then updated while accidentally including a +soname bump to ``libexample.so.2``, |pacman| will prevent this package from +being upgraded on a user's system, because it can no longer provide +``libexample.so.1``, which is required by its consumers (i.e. +``dummy-package``). +This only helps against immediate breakage on already installed systems. On +systems that are about to be installed it would lead to pacman not being able +to resolve the dependencies and bailing out. It is therefore to be considered a +stop-gap solution which allows for fixing the package(s) in question, while not +immediately breaking consumers of ``libexample``. + +In the future this feature will be directly built into |makepkg|, removing the +manual process of identifying shared libraries (and their sonames) which are +provided by packages. + +.. note:: + + To identify the sonames provided by a package, |find-libprovides| can be + used. Reversely, to identify the sonames required by a specific package + |find-libdeps| can be used. + +Debug packages +-------------- + +The ability to debug software using e.g. |gdb| is very powerful, as it allows +users to provide vital information about failing software to the packagers and +upstream projects. For this to work, a package's debug symbols need to be +provided to the debugger. +In February 2022 Arch Linux has started using |debug packages and debuginfod|, +which allows just that. + +Creating a package and additionally also building its debug symbols has now +become as easy as adding ``debug`` to the ``options`` array in a |pkgbuild| +(until this option eventually is added to the default for packagers of the +distribution). + +.. note:: + + At the time of writing not all issues with non-|file-hierarchy| compliant + files and directories have been solved yet, but a large set of debug packages + have already been built across all official repositories. + +Creating users +-------------- + +Historically, system users and groups for packages have been created +using ``.install`` scripts (see |pkgbuild#install|). This had the downside of +requiring a specific |user identifier| (UID) and/or |group identifier| (GID) +(see |UID/ GID database| for specific assignments) if file ownerships also +needed to be handled in the context of a package. +Additionally, the user and group creation was not standardized and required a +script (run as ``root``), which was only run *after* creating and installing +the package and therefore not easily testable. + +With the adoption of |systemd| and specifically |sysusers.d| the workflow has +changed to installing a single file in the context of a package to the vendor +location ``/usr/lib/sysusers.d/``. Based on the |systemd| |alpm-hooks| setup +the configuration is applied using |systemd-sysusers|. + +Changing files after package installation +----------------------------------------- + +Similar to how system users and groups have been created in the past, file +modifications (e.g. ownership, |extended file attributes| or |setuid|) have +been done using ``.install`` scripts or directly in PKGBUILDs. The problem with +this approach was, that it required the specific assignment and pinning of UIDs +and GIDs when creating the required users and groups, *before* doing the file +modifications (e.g. using |chown|). + +This task has been made less complex with |tmpfiles.d|, which allows for +packaging a single file in a package to the vendor location +``/usr/lib/tmpfiles.d/``. Due to the ordering of |alpm-hooks| first users +and groups are created and only afterwards the file configuration is applied +using |systemd-tmpfiles|, allowing for diverse scenarios. + +Packaging +========= + +Working on packages for software written in different languages (e.g. |php|, +|python|, |ruby|, |c|, |c++| or |rust|) using various build systems surely +makes for a very interesting Github profile eventually (due to providing issue +reports and fixes to many projects). + +You can find the |list of my packages| amongst the official repositories. +Moreover I currently maintain two |unofficial user repositories|: |[realtime]| +and |[pro-audio-legacy]| + +Packaging can be a fun but also a very time consuming and frustrating pastime. +As such there are many many more examples and specifics I could list (but this +article is already quite dense I am afraid). + +.. note:: + + None of the Arch Linux package maintainers are payed for their work, which + they do solely in their free time and the project is not a commercial + endeavor. Some dependency chains are fairly complex and require time and + care. The reasons as to why a package is not updated is not always obvious to + the outside and it could be due to various technical problems or the packager + just lacking the time. Please be mindful about this and instead of getting + upset try to lend a helping hand packaging (it is usually much appreciated) + or consider |getting involved|. + +At any rate, I hope I could spark your curiosity! If you are interested in +finding out more about packaging for specific languages or best practices, +the following are some good starting points: + +* |arch package guidelines| and |arch package guidelines category| +* |#archlinux| and |#archlinux-aur| on |libera.chat| + +.. note:: + + Update: I have fixed some typos. Thanks to Andreas Schleifer (Segaja) for + noticing them! + + Update: I have fixed more typos. Thanks to `doesntthinkmuch + <https://www.reddit.com/user/doesntthinkmuch/>`_ for noticing them. + +.. [1] I am excluding `flatpak <https://flatpak.org/>`_ and `snap + <https://snapcraft.io/>`_ in this article as they follow an app-store/ + per-user installation paradigm. However, they relate to system's packaging + in that they provide precompiled binaries in a predefined format. + +.. |arch linux| raw:: html + + <a target="blank" href="https://archlinux.org">Arch Linux</a> + +.. |linux distribution| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Linux_distribution">Linux distribution</a> + +.. |software repositories| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Software_repository">software repositories</a> + +.. |web servers| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Web_server">web servers</a> + +.. |package manager| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Package_manager">package manager</a> + +.. |gentoo linux| raw:: html + + <a target="blank" href="https://www.gentoo.org/">Gentoo Linux</a> + +.. |planned obsolescence| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Planned_obsolescence">planned obsolescence</a> + +.. |homebrew| raw:: html + + <a target="blank" href="https://brew.sh/">Homebrew</a> + +.. |chocolatey| raw:: html + + <a target="blank" href="https://chocolatey.org/">Chocolatey</a> + +.. |archive files| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Archive_file">archive files</a> + +.. |filesystem hierarchy standard| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard">filesystem hierarchy standard</a> + +.. |file-hierarchy| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/core/systemd/file-hierarchy.7.en">file-hierarchy</a> + +.. |tmpfiles.d| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/tmpfiles.d.5.en">tmpfiles.d</a> + +.. |sysusers.d| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/sysusers.d.5.en">sysusers.d</a> + +.. |alpm-hooks| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/alpm-hooks.5">alpm-hooks</a> + +.. |makepkg| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/makepkg.8">makepkg</a> + +.. |pacman| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/pacman.8">pacman</a> + +.. |pacman_website| raw:: html + + <a target="blank" href="https://archlinux.org/pacman/">Pacman</a> + +.. |PKGBUILD| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/core/pacman/PKGBUILD.5.en">PKGBUILD</a> + +.. |bash| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Bash_(Unix_shell)">Bash</a> + +.. |makepkg.conf| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/makepkg.conf.5">makepkg.conf</a> + +.. |devtools| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/devtools">devtools</a> + +.. |makechrootpkg| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/makechrootpkg.1">makechrootpkg</a> + +.. |systemd-nspawn| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/systemd-nspawn.1">systemd-nspawn</a> + +.. |chroot| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/chroot.1.en">chroot</a> + +.. |checkpkg| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/checkpkg.1">checkpkg</a> + +.. |namcap| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/namcap.1">namcap</a> + +.. |arch package guidelines| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/Arch_package_guidelines">Arch package guidelines</a> + +.. |arch package guidelines category| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/Category:Arch_package_guidelines">Arch package guidelines category</a> + +.. |arch build system| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/Arch_Build_System">Arch Build System</a> + +.. |asp| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/asp.1">asp</a> + +.. |howto be a packager| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/DeveloperWiki:HOWTO_Be_A_Packager">HOWTO be a packager</a> + +.. |svn| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/svn.1">svn</a> + +.. |git| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/git.1">git</a> + +.. |git-svn| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/extra/git/git-svn.1.en">git-svn</a> + +.. |svntogit-packages| raw:: html + + <a target="blank" href="https://github.com/archlinux/svntogit-packages">svntogit-packages</a> + +.. |svntogit-community| raw:: html + + <a target="blank" href="https://github.com/archlinux/svntogit-community">svntogit-community</a> + +.. |dbscripts| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/dbscripts">dbscripts</a> + +.. |arch-repo-management| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/arch-repo-management">arch-repo-management</a> + +.. |make| raw:: html + + <a target="blank" href="https://www.gnu.org/software/make/">make</a> + +.. |pkgbuild#pkgname| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#pkgname">PKGBUILD#pkgname</a> + +.. |pkgbuild#pkgver| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#pkgver">PKGBUILD#pkgver</a> + +.. |pkgbuild#pkgrel| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#pkgrel">PKGBUILD#pkgrel</a> + +.. |version control| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Version_control">version control</a> + +.. |licenses package| raw:: html + + <a target="blank" href="https://archlinux.org/packages/core/any/licenses/">licenses package</a> + +.. |pkgbuild#license| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#license">PKGBUILD#licenses</a> + +.. |pkgbuild#integrity| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#Integrity">PKGBUILD#Integrity</a> + +.. |updpkgsums| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/updpkgsums.8">updpkgsums</a> + +.. |pacman-contrib| raw:: html + + <a target="blank" href="https://archlinux.org/packages/?q=pacman-contrib">pacman-contrib</a> + +.. |fakeroot| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/fakeroot.1">fakeroot</a> + +.. |pgp| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Pretty_Good_Privacy">PGP</a> + +.. |signify| raw:: html + + <a target="blank" href="https://github.com/aperezdc/signify">signify</a> + +.. |cosign| raw:: html + + <a target="blank" href="https://github.com/sigstore/cosign">cosign</a> + +.. |makepkg#signature_checking| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/Makepkg#Signature_checking">makepkg#signature_checking</a> + +.. |pkgbuild#using_vcs_sources| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/PKGBUILD.5.en#USING_VCS_SOURCES">PKGBUILD#USING_VCS_SOURCES</a> + +.. |chain of trust| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Chain_of_trust">chain of trust</a> + +.. |supply chain attack| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Supply_chain_attack">supply chain attack</a> + +.. |arch linux rfc| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/rfcs">Arch Linux RFC</a> + +.. |reproducible builds| raw:: html + + <a target="blank" href="https://reproducible-builds.org/">Reproducible Builds</a> + +.. |rebuilderd| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/rebuilderd.1">rebuilderd</a> + +.. |makerepropkg| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/makerepropkg.1">makerepropkg</a> + +.. |diffoscope| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/diffoscope.1">diffoscope</a> + +.. |repro| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/repro.8">repro</a> + +.. |buildinfo| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/core/pacman/BUILDINFO.5.en">.BUILDINFO</a> + +.. |pkgbuild#pkgbase| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#pkgbase">PKGBUILD#pkgbase</a> + +.. |libalpm_databases| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/libalpm_databases.3">libalpm_databases</a> + +.. |repo-add| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/repo-add.8">repo-add</a> + +.. |repo-remove| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/repo-add.8">repo-remove</a> + +.. |ssh| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/ssh.1">ssh</a> + +.. |web of trust| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Web_of_trust">web of trust</a> + +.. |main keys| raw:: html + + <a target="blank" href="https://archlinux.org/master-keys/">main keys</a> + +.. |pacman.conf#package_and_database_signature_checking| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/pacman.conf.5#PACKAGE_AND_DATABASE_SIGNATURE_CHECKING">pacman.conf#PACKAGE_AND_DATABASE_SIGNATURE_CHECKING</a> + +.. |pacman-key| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/pacman-key.8">pacman-key</a> + +.. |archlinux-keyring| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/archlinux-keyring/">archlinux-keyring</a> + +.. |sequoia| raw:: html + + <a target="blank" href="https://sequoia-pgp.org/">sequoia</a> + +.. |archlinux-keyring package| raw:: html + + <a target="blank" href="https://archlinux.org/packages/core/any/archlinux-keyring/">archlinux-keyring package</a> + +.. |Levente Polyak| raw:: html + + <a target="blank" href="https://leventepolyak.net/">Levente Polyak</a> + +.. |dynamic linking| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Dynamic_linker">dynamic linking</a> + +.. |soname| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Soname">soname</a> + +.. |application binary interface| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Application_binary_interface">application binary interface</a> + +.. |find-libprovides| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/find-libprovides.1">find-libprovides</a> + +.. |find-libdeps| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/find-libdeps.1">find-libdeps</a> + +.. |gdb| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/gdb.1">gdb</a> + +.. |debug packages and debuginfod| raw:: html + + <a target="blank" href="https://archlinux.org/news/debug-packages-and-debuginfod/">debug packages and debuginfod</a> + +.. |pkgbuild#install| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/PKGBUILD#install">PKGBUILD#install</a> + +.. |user identifier| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/User_identifier">user identifier</a> + +.. |group identifier| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Group_identifier">group identifier</a> + +.. |UID/ GID database| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/DeveloperWiki:UID_/_GID_Database">UID/ GID database</a> + +.. |systemd| raw:: html + + <a target="blank" href="https://systemd.io/">systemd</a> + +.. |systemd-sysusers| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/systemd-sysusers.8">systemd-sysusers</a> + +.. |extended file attributes| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Extended_file_attributes#Linux">extended file attributes</a> + +.. |setuid| raw:: html + + <a target="blank" href="https://en.wikipedia.org/wiki/Setuid">setuid</a> + +.. |chown| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/chown.1">chown</a> + +.. |systemd-tmpfiles| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/systemd-tmpfiles.8">systemd-tmpfiles</a> + +.. |php| raw:: html + + <a target="blank" href="https://www.php.net/">PHP</a> + +.. |python| raw:: html + + <a target="blank" href="https://www.python.org/">Python</a> + +.. |ruby| raw:: html + + <a target="blank" href="https://www.ruby-lang.org/en/">Ruby</a> + +.. |c| raw:: html + + <a target="blank" href="https://www.iso.org/standard/74528.html">C</a> + +.. |c++| raw:: html + + <a target="blank" href="https://isocpp.org/">C++</a> + +.. |rust| raw:: html + + <a target="blank" href="https://www.rust-lang.org/">Rust</a> + +.. |#archlinux| raw:: html + + <a target="blank" href="ircs://irc.libera.chat/archlinux">#archlinux</a> + +.. |#archlinux-aur| raw:: html + + <a target="blank" href="ircs://irc.libera.chat/archlinux-aur">#archlinux-aur</a> + +.. |libera.chat| raw:: html + + <a target="blank" href="https://libera.chat/">libera.chat</a> + +.. |list of my packages| raw:: html + + <a target="blank" href="https://archlinux.org/packages/?sort=&q=&maintainer=dvzrv&flagged=">list of my packages</a> + +.. |unofficial user repositories| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/Unofficial_user_repositories">unofficial user repositories</a> + +.. |[realtime]| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/Unofficial_user_repositories#realtime">[realtime]</a> + +.. |[pro-audio-legacy]| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/Unofficial_user_repositories#pro-audio-legacy">[pro-audio-legacy]</a> + +.. |getting involved| raw:: html + + <a target="blank" href="https://wiki.archlinux.org/title/Getting_involved">getting involved</a> + |