From 700e7da54bc57133d5283e5ed0941dade61f5b20 Mon Sep 17 00:00:00 2001 From: David Runge Date: Sat, 2 Jul 2022 13:24:29 +0200 Subject: Add post about managing binary package repositories posts/2022/managing-binary-package-repositories.rst: Add post about managing binary package repositories which concerns itself with the first minor version release of repod. --- .../2022/managing-binary-package-repositories.rst | 193 +++++++++++++++++++++ 1 file changed, 193 insertions(+) create mode 100644 posts/2022/managing-binary-package-repositories.rst diff --git a/posts/2022/managing-binary-package-repositories.rst b/posts/2022/managing-binary-package-repositories.rst new file mode 100644 index 0000000..89f203f --- /dev/null +++ b/posts/2022/managing-binary-package-repositories.rst @@ -0,0 +1,193 @@ +.. title: Managing binary package repositories +.. slug: managing-binary-package-repositories +.. date: 2022-07-02 13:19:59 UTC+02:00 +.. tags: arch linux, packaging, repod, dbscripts +.. category: archlinux +.. link: +.. description: +.. type: text + +In `Packaging for Arch Linux +`_ I described the ins and +outs of binary repository management and some of the issues that come with the +tooling currently used by |arch linux|. + +In this article I will highlight the work on new tooling and its features. + +Since my last write-up on this topic, the project formerly known as +``arch-repo-management`` has been renamed to ``repod`` (as in *repo-d*) and has +just seen its first minor release. 🎉 + +You can find its documentation at https://repod.archlinux.page. + +.. TEASER_END + +.. note:: + + The https://archlinux.page domain has recently been acquired by Arch Linux to + serve as the common platform for documentation of projects driven by the + distribution. + +Please note, that |repod| 0.1.0 is still alpha grade software and **should +not** be used to actually manage binary package repositories at this point in +time! + +However, it is already possible to do a few things with the software and if you +are able to test it or are interested in helping develop it, that is very much +welcomed! + +On Arch Linux you can install it using |pacman|: + +.. code:: sh + + pacman -S repod + +On other distributions (even on macOS!) you may install it using |pip| (|repod| +is |available on pypi|) until your respective package management system makes +the software available to you on a system level: + +.. code:: sh + + pip install --user repod + +Working on repod +---------------- + +Since 2021 I have been on and off working on what is now |repod|. The project +is written in typed |python| and is extensively tested using |pytest|. + +Work first began after |arch conf| 2019, at which a working group had looked +into improving the workflows currently employed by the distribution and pushing +for tooling that would allow moving away from an |svn| monorepo based approach +to a deconstructed |git| setup. + +A proof of concept (PoC) to mimic the behavior of |dbscripts| had been created, +but after 2019 this work laid dormant. + +Over 2021 I have spent time to transform parts of the PoC into a Python project +following best practices for development (e.g. type hints following |pep0484|, +100% test coverage, data validation using |pydantic| models), exposing first +features in scripts. + +In 2022 more work has been done to extend validation and transform the project +into a package based setup for easier handling and extension in the future. + +Concepts of repod +----------------- + +Contrary to |dbscripts|, |repod| follows a paradigm in which it is largely +decoupled from the |source repository| of the binary packages it maintains and +aims at becoming a self-contained service. + +Package files and their signatures are consumed, relevant metadata is extracted +and transferred to a |management repository|, which is where the state of +each |binary repository| is kept. + +Available packages in a given |binary repository| are exposed to |pacman| via +|sync database| files. The |management repository| contents can be +(transparently and reproducibly) transformed into |sync database| files and +vice versa. + +The above functionality is exposed on the command line via |repod-file|. + +Upcoming work +------------- + +As mentioned above, |repod| is not yet stable and still misses quite a few +features. +The following topics (and more) will be worked on in the next milestones (not +necessarily in this order): + +* file handling (moving package files from staging areas to actual repositories) +* signature validation (PGP signature validation of package files) +* handling of debug packages +* consolidate schema of |management repository| +* improve logging throughout the project +* git backend to transparently expose changes to the |management repository| +* caching for |management repository| state (e.g. to allow fast searches) +* API to interact with a |repod| instance over the wire in an authenticated fashion +* client-side tooling to interact with |repod|'s API + +There is still a lot of work to be done, so if you have a background in +|Python| development and are interested in working on a project that is very +close to the distribution and will likely improve the workflow of many people +while making binary repository management more transparent to the user, have a +look at the project's |contribution guidelines| and do not hesitate to reach +out! + +.. |arch linux| raw:: html + + Arch Linux + +.. |repod| raw:: html + + repod + +.. |pacman| raw:: html + + pacman + +.. |pip| raw:: html + + pip + +.. |available on pypi| raw:: html + + available on pypi + +.. |python| raw:: html + + Python + +.. |pytest| raw:: html + + pytest + +.. |arch conf| raw:: html + + Arch Conf + +.. |svn| raw:: html + + svn + +.. |git| raw:: html + + git + +.. |dbscripts| raw:: html + + dbscripts + +.. |pep0484| raw:: html + + PEP 0484 + +.. |pydantic| raw:: html + + pydantic + +.. |source repository| raw:: html + + source repository + +.. |management repository| raw:: html + + management repository + +.. |binary repository| raw:: html + + binary repository + +.. |sync database| raw:: html + + sync database + +.. |repod-file| raw:: html + + repod-file + +.. |contribution guidelines| raw:: html + + contribution guidelines + -- cgit v1.2.3-54-g00ecf