diff options
| -rw-r--r-- | posts/2022/managing-binary-package-repositories.rst | 193 |
1 files changed, 193 insertions, 0 deletions
diff --git a/posts/2022/managing-binary-package-repositories.rst b/posts/2022/managing-binary-package-repositories.rst new file mode 100644 index 0000000..89f203f --- /dev/null +++ b/posts/2022/managing-binary-package-repositories.rst @@ -0,0 +1,193 @@ +.. title: Managing binary package repositories +.. slug: managing-binary-package-repositories +.. date: 2022-07-02 13:19:59 UTC+02:00 +.. tags: arch linux, packaging, repod, dbscripts +.. category: archlinux +.. link: +.. description: +.. type: text + +In `Packaging for Arch Linux +<https://sleepmap.de/2022/packaging-for-arch-linux>`_ I described the ins and +outs of binary repository management and some of the issues that come with the +tooling currently used by |arch linux|. + +In this article I will highlight the work on new tooling and its features. + +Since my last write-up on this topic, the project formerly known as +``arch-repo-management`` has been renamed to ``repod`` (as in *repo-d*) and has +just seen its first minor release. 🎉 + +You can find its documentation at https://repod.archlinux.page. + +.. TEASER_END + +.. note:: + + The https://archlinux.page domain has recently been acquired by Arch Linux to + serve as the common platform for documentation of projects driven by the + distribution. + +Please note, that |repod| 0.1.0 is still alpha grade software and **should +not** be used to actually manage binary package repositories at this point in +time! + +However, it is already possible to do a few things with the software and if you +are able to test it or are interested in helping develop it, that is very much +welcomed! + +On Arch Linux you can install it using |pacman|: + +.. code:: sh + + pacman -S repod + +On other distributions (even on macOS!) you may install it using |pip| (|repod| +is |available on pypi|) until your respective package management system makes +the software available to you on a system level: + +.. code:: sh + + pip install --user repod + +Working on repod +---------------- + +Since 2021 I have been on and off working on what is now |repod|. The project +is written in typed |python| and is extensively tested using |pytest|. + +Work first began after |arch conf| 2019, at which a working group had looked +into improving the workflows currently employed by the distribution and pushing +for tooling that would allow moving away from an |svn| monorepo based approach +to a deconstructed |git| setup. + +A proof of concept (PoC) to mimic the behavior of |dbscripts| had been created, +but after 2019 this work laid dormant. + +Over 2021 I have spent time to transform parts of the PoC into a Python project +following best practices for development (e.g. type hints following |pep0484|, +100% test coverage, data validation using |pydantic| models), exposing first +features in scripts. + +In 2022 more work has been done to extend validation and transform the project +into a package based setup for easier handling and extension in the future. + +Concepts of repod +----------------- + +Contrary to |dbscripts|, |repod| follows a paradigm in which it is largely +decoupled from the |source repository| of the binary packages it maintains and +aims at becoming a self-contained service. + +Package files and their signatures are consumed, relevant metadata is extracted +and transferred to a |management repository|, which is where the state of +each |binary repository| is kept. + +Available packages in a given |binary repository| are exposed to |pacman| via +|sync database| files. The |management repository| contents can be +(transparently and reproducibly) transformed into |sync database| files and +vice versa. + +The above functionality is exposed on the command line via |repod-file|. + +Upcoming work +------------- + +As mentioned above, |repod| is not yet stable and still misses quite a few +features. +The following topics (and more) will be worked on in the next milestones (not +necessarily in this order): + +* file handling (moving package files from staging areas to actual repositories) +* signature validation (PGP signature validation of package files) +* handling of debug packages +* consolidate schema of |management repository| +* improve logging throughout the project +* git backend to transparently expose changes to the |management repository| +* caching for |management repository| state (e.g. to allow fast searches) +* API to interact with a |repod| instance over the wire in an authenticated fashion +* client-side tooling to interact with |repod|'s API + +There is still a lot of work to be done, so if you have a background in +|Python| development and are interested in working on a project that is very +close to the distribution and will likely improve the workflow of many people +while making binary repository management more transparent to the user, have a +look at the project's |contribution guidelines| and do not hesitate to reach +out! + +.. |arch linux| raw:: html + + <a target="blank" href="https://archlinux.org">Arch Linux</a> + +.. |repod| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/repod">repod</a> + +.. |pacman| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/pacman.8">pacman</a> + +.. |pip| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/pip.1">pip</a> + +.. |available on pypi| raw:: html + + <a target="blank" href="https://pypi.org/project/repod/">available on pypi</a> + +.. |python| raw:: html + + <a target="blank" href="https://www.python.org/">Python</a> + +.. |pytest| raw:: html + + <a target="blank" href="https://docs.pytest.org/en/latest/">pytest</a> + +.. |arch conf| raw:: html + + <a target="blank" href="https://conf.archlinux.org/">Arch Conf</a> + +.. |svn| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/svn.1">svn</a> + +.. |git| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/git.1">git</a> + +.. |dbscripts| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/dbscripts">dbscripts</a> + +.. |pep0484| raw:: html + + <a target="blank" href="https://peps.python.org/pep-0484/">PEP 0484</a> + +.. |pydantic| raw:: html + + <a target="blank" href="https://pydantic-docs.helpmanual.io/">pydantic</a> + +.. |source repository| raw:: html + + <a target="blank" href="https://repod.archlinux.page/repositories/source_repository.html">source repository</a> + +.. |management repository| raw:: html + + <a target="blank" href="https://repod.archlinux.page/repositories/management_repository.html">management repository</a> + +.. |binary repository| raw:: html + + <a target="blank" href="https://repod.archlinux.page/repositories/binary_repository.html">binary repository</a> + +.. |sync database| raw:: html + + <a target="blank" href="https://repod.archlinux.page/repositories/sync_database.html">sync database</a> + +.. |repod-file| raw:: html + + <a target="blank" href="https://man.archlinux.org/man/repod-file.1">repod-file</a> + +.. |contribution guidelines| raw:: html + + <a target="blank" href="https://gitlab.archlinux.org/archlinux/repod/-/blob/79a0fa9aa4a4d9def6dad24805f9bbff0388f734/CONTRIBUTING.md">contribution guidelines</a> + |