path: root/posts/2023/operating-system-bias-in-next-generation-internet-and-nlnet.md

<!--
.. title: Operating System Bias in Next Generation Internet and NLnet
.. slug: operating-system-bias-in-next-generation-internet-and-nlnet
.. date: 2023-11-16 13:00:00 UTC+01:00
.. tags: alpm, arch linux, european commission, funding, guix, next generation internet, ngi, nix, nixos, nixos foundation, nlnet, package manager, software development
.. category: archlinux
.. link: 
.. description: 
.. type: text
-->

In [Grants for Operating Systems] I discussed my journey through the grant application writing business since beginning of last year.
To keep things light and somewhat focused, I left out a topic, that I would like to write about in more detail in the following sections.

It's about selection bias in grants provided by Next Generation Internet ([NGI]), that can be applied for directly or through [NLnet].

<!-- TEASER_END -->

---
📝 Before going into further detail, I would like to point out several things:

* I believe funding programs such as [NGI] and [NLnet] are a very important pillar of a free and decentralized computing world
* I *do not* wish any harm upon any of the involved organizations or any of their employees
* This article is here to document my personal experiences and findings based on publicly available data, in the hopes of addressing what appears to be a selection bias
* Although none of the [Arch Linux] related funding requests discussed in my previous article have been approved, I have been involved with Operating System agnostic projects, funded through [NLnet], but applied for by other people, in the past
---

## Round two

With my latest [NLnet] application for funding work on [ALPM] projects, I managed to reach round two.

To give a bit of preliminary background on what those projects are about (a more detailed article on them will follow):

During the work on [repod] I noticed, that many of the metadata files used in [Arch Linux]'s packaging do not have a specification and no common parsers and validators. This required writing a lot of additional code for the consumption and validation of those files and I noticed that several efforts across other languages and projects existed.
I realized, that it would be best to provide central specifications, parsers and writers, which could be used all across the stack. As such, the [ALPM] projects are a very [Arch Linux] packaging specific attempt at improving the tooling of this community-driven distribution.

Given the above, it felt rather strange to read loaded questions in my set of round two questions, that implied users seeking more guarantees should just move to distribution agnostic functional [package manager]s, such as [Nix] or [Guix], as those could be combined with [Arch Linux] without effort.

I attempted to reply by outlining why funding for the packaging subsystem of community-driven distributions is important and that "just replace everything with Nix" is not the answer for [Arch Linux] and its users.

  > Using a package management system other than pacman on Arch Linux is not supported and is in fact likely to break the system.
  > Users that wish to use nix or guix can use dedicated distributions such as NixOS.
  >
  > Arch Linux and its maintainers have spent decades building up expertise in integrating their custom package management system with various init systems, dedicated tooling and languages.
  > Arch Linux follows central principles (https://wiki.archlinux.org/title/Arch_Linux#Principles), that encourage a simple, rolling release, “follow upstream” approach. As such it differs from other distributions.
  >
  > Arch Linux’s thousands of users are familiar with its package management system. A radical change, such as replacing the package management subsystem is nothing that can be done on a whim or is even realistic given the steep learning curve of learning one.
  >
  > Working on the Arch Linux Package Management framework ensures the diversity in a small set of original, community driven Linux distributions (such as Debian, Arch Linux, NixOS), for which public funding is essential.

Other questions revolved around topics such as comparison with other standardization and file format efforts, allotted time for the work (which was deemed too high), future changes to file formats and generalizing parsers using structured format.

As mentioned in my previous article, my application was rejected after another six weeks after my reply. As the rejection message was generic, I do not know whether any of the answers (or all of them) were dissatisfactory.

## Investigating Bias

The [package manager] related questions in my second round review struck me as very odd and led me to do some deeper investigation into the [NLnet] and [NGI] funding setup and to ask a few people, that received OS-agnostic funding about their experiences.

People's experience with the review process seems to be largely identical to mine. However, in at least one occasion an applicant got an actual specific (non-generic) reason for their rejected [NLnet] application towards an [NGI Zero] grant (which was surprising even to them).
In at least one other case, a person was asked to make their application run on [NixOS] after receiving their final payment.

I started to look further into the [NixOS] story in this context and discovered several connections between [NLnet] and [NixOS Foundation].

For the unaware: [Nix] is the system [package manager] used on [NixOS] and [NixOS Foundation] has the mission to *"[..] support the Nix ecosystem's infrastructure, and projects implementing the purely functional deployment model."* [^1]
I use [Nix] and [NixOS] interchangeably throughout the following subsections, as the [package manager] is tightly coupled with the Operating System.

Through membership in [NGI Zero], [NixOS Foundation] is part of [NGI Zero Core], [NGI Zero Entrust], [NGI Zero PET] and [NGI Zero Review] (see [background info on NGI Zero Core], [background info on NGI Zero Entrust], [background info on NGI Zero PET] and [background info on NGI Zero Review], respectively).

In an attempt at giving an overview of [Nix] and [Guix] vs. other Operating System and system [package manager] projects funded by [NLnet], I went through several program pages to collect affiliated projects. This proved to be not so easy, as the website follows varying style approaches and does not offer a search functionality, which hinders filtering by keyword. So please take the following numbers with a grain of salt!

I manually searched through overview pages linked to in the following sections, using `nix` and `guix` as search term to correlate projects, that stand in some relation to [NixOS Foundation]/ [NixOS] or [Guix].
I did the same using the `OS`, `operating system` and `package` search terms to sieve through projects, that have some relation to other specific general-purpose Operating Systems and do not provide generic features (e.g. VPN or firewall stack on Linux, etc.), or concern themselves with other system [package manager]s.
The percentages for occurrences are rounded up to the 2nd position after decimal point.

Do note, that the [NLnet] projects largely appear to be not tied to specific Operating Systems!

### NGI Assure

There are 145 [ongoing NGI Assure projects].

Eight [Nix] related (5.52%):

* [https://nlnet.nl/project/Dream2nix](https://nlnet.nl/project/Dream2nix)
* [https://nlnet.nl/project/Dream2nix-Python](https://nlnet.nl/project/Dream2nix-Python)
* [https://nlnet.nl/project/Nix-TypeInference](https://nlnet.nl/project/NixOS-Services)
* [https://nlnet.nl/project/NixOS-Clevis](https://nlnet.nl/project/NixOS-Services)
* [https://nlnet.nl/project/NixOS-Services](https://nlnet.nl/project/NixOS-Services)
* [https://nlnet.nl/project/NixOS-UEFI](https://nlnet.nl/project/NixOS-UEFI)
* [https://nlnet.nl/project/Tvix](https://nlnet.nl/project/Tvix)
* [https://nlnet.nl/project/p4-nix](https://nlnet.nl/project/p4-nix)

Seven [Guix] related (4.83%):

* [https://nlnet.nl/project/GNUMes-ARM_RISC-V](https://nlnet.nl/project/GNUMes-ARM_RISC-V)
* [https://nlnet.nl/project/GNUMes-RISCV](https://nlnet.nl/project/GNUMes-RISCV)
* [https://nlnet.nl/project/GNUMes-RISCV-bootstrap](https://nlnet.nl/project/GNUMes-RISCV-bootstrap)
* [https://nlnet.nl/project/GNUMesTower](https://nlnet.nl/project/GNUMesTower)
* [https://nlnet.nl/project/Gash](https://nlnet.nl/project/Gash)
* [https://nlnet.nl/project/Guix-P2P](https://nlnet.nl/project/Guix-P2P)
* [https://nlnet.nl/project/Guix-Riscv64](https://nlnet.nl/project/Guix-Riscv64)

Three other OSes, mostly special purpose or mobile (2.07%):

* [https://nlnet.nl/project/MaemoLeste-Telepathy](https://nlnet.nl/project/MirageVPN)
* [https://nlnet.nl/project/MirageVPN](https://nlnet.nl/project/MirageVPN)
* [https://nlnet.nl/project/OpenCryptoLinux](https://nlnet.nl/project/OpenCryptoLinux)

I was not able to find any other system [package manager] related projects.

### NGI Zero Core

There are 21 [ongoing NGI Zero Core projects].

I was not able to find any [Nix] related projects.

One [Guix] related (4.76%):

* [https://nlnet.nl/project/GuixDaemon-Guile](https://nlnet.nl/project/GuixDaemon-Guile)

Four other Operating System related projects, mostly special purpose or mobile (19.05%):

* [https://nlnet.nl/project/MobileSettings](https://nlnet.nl/project/MobileSettings)
* [https://nlnet.nl/project/SlintAndroid](https://nlnet.nl/project/SlintAndroid)
* [https://nlnet.nl/project/WPE-Android](https://nlnet.nl/project/WPE-Android)
* [https://nlnet.nl/project/pmOS-23-24](https://nlnet.nl/project/pmOS-23-24)

I was not able to find any other system [package manager] related projects.

### NGI Zero Entrust

There are 150 [ongoing NGI Zero Entrust projects].

Six [Nix] related (4%) projects:

* [https://nlnet.nl/project/CloudHostingServicePortability](https://nlnet.nl/project/CloudHostingServicePortability)
* [https://nlnet.nl/project/Genealogos](https://nlnet.nl/project/Genealogos)
* [https://nlnet.nl/project/GorgonCI](https://nlnet.nl/project/GorgonCI)
* [https://nlnet.nl/project/Liminix](https://nlnet.nl/project/Liminix)
* [https://nlnet.nl/project/NixDebugAdaptor](https://nlnet.nl/project/NixDebugAdaptor)
* [https://nlnet.nl/project/SelfPrivacy](https://nlnet.nl/project/SelfPrivacy)

I was not able to find any [Guix] related projects.

Six other Operating System related projects, mostly special purpose or Android (4%):

* [https://nlnet.nl/project/Irdest-OpenWRT-BLE](https://nlnet.nl/project/Irdest-OpenWRT-BLE)
* [https://nlnet.nl/project/Makatea](https://nlnet.nl/project/Makatea)
* [https://nlnet.nl/project/Replicant-Pinephone](https://nlnet.nl/project/Replicant-Pinephone)
* [https://nlnet.nl/project/SeedVault-Integrity](https://nlnet.nl/project/SeedVault-Integrity)
* [https://nlnet.nl/project/Spectrum-Applications](https://nlnet.nl/project/Spectrum-Applications)
* [https://nlnet.nl/project/Trenchboot-AEM](https://nlnet.nl/project/Trenchboot-AEM)

I was not able to find any other system [package manager] related projects.

### NGI Zero PET

There are 144 [ongoing NGI Zero PET projects].

Three [Nix] related (2.1%):

* [https://nlnet.nl/project/Robotnix](https://nlnet.nl/project/Robotnix)
* [https://nlnet.nl/project/Spectrum](https://nlnet.nl/project/Spectrum)
* [https://nlnet.nl/project/mobile-nixos](https://nlnet.nl/project/mobile-nixos)

Three [Guix] related (2.1%):

* [https://nlnet.nl/project/Cuirass](https://nlnet.nl/project/Cuirass)
* [https://nlnet.nl/project/GNUMes](https://nlnet.nl/project/GNUMes)
* [https://nlnet.nl/project/GNUMes-fullsource](https://nlnet.nl/project/GNUMes-fullsource)

Eight other Operating System related projects, mostly special purpose or Android (5.56%):

* [https://nlnet.nl/project/Seedvault](https://nlnet.nl/project/Seedvault)
* [https://nlnet.nl/project/WireGuardonWindows](https://nlnet.nl/project/WireGuardonWindows)
* [https://nlnet.nl/project/postmarketOS](https://nlnet.nl/project/postmarketOS)
* [https://nlnet.nl/project/seL4-64bitVMM](https://nlnet.nl/project/seL4-64bitVMM)
* [https://nlnet.nl/project/BetrustedOS](https://nlnet.nl/project/BetrustedOS)
* [https://nlnet.nl/project/Dataspaces/](https://nlnet.nl/project/Dataspaces/)
* [https://nlnet.nl/project/Replicant-graphics](https://nlnet.nl/project/Replicant-graphics)
* [https://nlnet.nl/project/ReplicantUpdate](https://nlnet.nl/project/ReplicantUpdate)

I was not able to find any other system [package manager] related projects.

### Internet Hardening Fund

There are 24 [projects of the Internet Hardening Fund].

One [Nix] related (4.17%):

* [https://nlnet.nl/project/webservicesecurity](https://nlnet.nl/project/webservicesecurity)

I was neither able to find any [Guix] related projects, nor any related to other Operating Systems or other system [package manager]s.

### User-Operated Internet Fund

There are eleven [projects of the User-Operated Internet Fund].

One is an Operating System specific project (9.10%).

* [https://nlnet.nl/project/Armbian/](https://nlnet.nl/project/Armbian/)

I was neither able to find any [Nix] or [Guix] related projects, nor any related to other system [package manager]s.

### NGI Zero Review

There are no publicly associated projects with the [NGI Zero Review] program, but the program itself promotes the use of [Nix] for *"[b]est practices on packaging and reproducible builds"* [^2] (see this [pull request towards the NixOS homepage to replace this problematic terminology]) for projects, that are mentored by it. It is unclear whether [NixOS Foundation] is compensated for this mentoring role.

### Further data on NixOS Foundation and NGI Zero

According to a [Summer of Nix 2022 interview], *"the [European Commission] through [DG CNECT] has partnerships with [NLnet] and the [NixOS Foundation]"*, funding several projects.
Furthermore, the [European Commission] appears to be facilitating and encouraging the use of [NixOS] internally, trying to replace other operating systems. The platform [code.europa.eu] is mentioned as a place for development of software and services related to European Union institutions.

When looking at the [NixOS Foundation's Financial Summary] for 2022, it shows an influx of 140.000€ of *"[f]unds from NLnet Foundation for the specific programs (i.e. Summer of Nix)"*. These are potentially for some of the above mentioned projects in the various [NLnet] related programs, for which [NixOS Foundation] may be handling payments to individuals (although [NLnet] grants are usually given to individuals). However, from reading the statement alone, it is unclear whether this is tied to individual grants, compensation for work on [NGI Zero Review], or even other things.

The above data points at a direct or at least indirect monetary conflict of interest for [NixOS Foundation] in the context of [NGI Zero], which in my opinion ultimately serves as a bias for any decision making process done in the context of that coalition.
My believe is, that the decision making process is therefore intrinsically skewed, because [NGI Zero] appears to be set up to promote one specific Operating System ([NixOS]) and [package manager] ([Nix]).
Looking at the numbers, also [NGI Assure] appears to be affected by this.

## Conclusion

Considering the previous sections provides a rather depressing outlook for non-[NixOS] general-purpose Linux distributions, as well as for non-[Nix]/[Guix] package management systems, when it comes to funding opportunities through [NLnet] or [NGI].

Of the 495 [NLnet] projects I sieved through (mostly superficially), 18 (3.64%) appear to be [Nix] related, eleven (2.22%) [Guix] related, while 22 (4.44%) account for the *entire rest* of Operating System specific projects. I was not able to find any other system [package manager] related project.

While I am convinced, that people related to [NixOS] write great applications, for me it is hard to believe that there are so few (good) applications by developers working on other packaging ecosystems, that not a single one was ever able to receive funding.
Relatedly, I neither believe, that it is right to ask those developers to replace their ecosystem with [Nix], nor be judged on the base of working on something that is not [Nix].
As such I believe that the above list of projects does not provide a balanced funding reality.

Therefore I would like to extend my points on funding organizations in my previous article:

* Technological bias in funding organizations claiming to *"[..] reflect the openness, diversity and the inclusion that are at the core of European values"* [^3] should be circumvented, or otherwise clearly stated.

Analogous, I would like to extend the advice for people trying to get their work funded by:

* If you are working on an Operating System related topic and you are not working on [NixOS], consider whether your time will be well spent on applying for [NGI Zero] related funds or probably even [NLnet] programs in general, as - given the publicly available data - there appears to be a bias on the [European Commission] level at play, that will very likely lead to your project not being selected or it getting very hard to be selected.
* If you are working on a system [package manager] other than [Nix] or [Guix], there is currently no data supporting the assumption, that this work would be funded when applying with [NLnet].

As I am not sure how well received this article will be with some of the organizations, I have neither mentioned people, that have helped review or write my applications, nor those, that I have asked about their experiences.

While I hope, that there will be no backlash towards people I interacted with, I realize, that this article may make some people uncomfortable and even undermine my chances of ever getting funding in the future. Either way, I believe it was the right thing to do and I arrive at the following conclusions and questions for myself:

* Given the above data points, why does [NGI] (and [NLnet] by extension) not more clearly state, that they are (seemingly) not interested in funding work on other [package manager] ecosystems? While I do not know how many others have applied for similar projects to mine, nor do I claim to speak for this unknown number of people, I would have loved to not waste my time on an application, that seems to have little chance of ever being accepted.
* The reasoning behind enforcing one specific [package manager] and Operating System in [NGI] is intransparent to outsiders. It is unclear to me when this decision was made, and under what circumstances the [European Commission] decided on it. The previously mentioned [Summer of Nix 2022 interview] seems to indicate, that the [European Commission] wants to switch to [NixOS] for its own services. This begs the question: Why do they not just contract with one of the [consulting companies available for professional support]? Biased funding on the other hand will have a huge impact on the ecosystem at large (and in my opinion not for the better).
* There is *a lot of work* to be done in all community-driven Linux distributions and this work has merit. Focusing only on one distribution will achieve two things: Destroying diversity and invalidating the work thousands of people have been doing in their free time for decades (often even trying to make a living by providing services around those Operating Systems).

There are many open questions and my hopes are, that the [European Commission], [NGI] and [NLnet] reevaluate their focus on seemingly funding only one packaging ecosystem.
I would be happy to receive feedback from people related to other Linux distributions, that interacted with [NGI] and [NLnet], as well as from officials involved with the decision making process in the [European Commission], [NGI] and [NLnet] and will update the post accordingly.
Writing this, I am still hopeful, that my post can be a first step towards improving the current situation and that funds directed at critical infrastructure projects will be distributed more evenly amongst widely used Operating System projects (big and small, well marketed and quiet).

[^1]: [https://nixos.org/community/index.html](https://nixos.org/community/index.html)
[^2]: [https://nlnet.nl/NGI0/review](https://nlnet.nl/NGI0/review)
[^3]: [https://digital-strategy.ec.europa.eu/en/library/internet-humans-how-we-would-internet-future-be](https://digital-strategy.ec.europa.eu/en/library/internet-humans-how-we-would-internet-future-be)

[Grants for Operating Systems]: https://sleepmap.de/2023/grants-for-operating-systems/
[NLnet]: https://nlnet.nl/
[ALPM]: https://gitlab.archlinux.org/archlinux/alpm
[repod]: https://gitlab.archlinux.org/archlinux/repod
[Arch Linux]: https://archlinux.org
[NGI]: https://www.ngi.eu/
[Guix]: https://guix.gnu.org/
[package manager]: https://en.wikipedia.org/wiki/Package_manager
[European Commission]: https://en.wikipedia.org/wiki/European_Commission
[NixOS]: https://nixos.org/
[Nix]: https://nixos.org/
[NixOS Foundation's Financial Summary]: https://discourse.nixos.org/t/nixos-foundations-financial-summary-a-transparent-look-into-2022/28107#sources-of-incoming-funds-2
[NGI]: https://www.ngi.eu
[NGI Assure]: https://www.ngi.eu/ngi-projects/ngi-assure
[User-Operated Internet Fund]: https://nlnet.nl/useroperated/index.html
[ALPM]: https://gitlab.archlinux.org/archlinux/alpm
[NGI Zero]: https://www.ngi.eu/ngi-projects/ngi-zero/
[NLnet's current projects]: https://nlnet.nl/project/current.html
[postmarketOS]: https://postmarketos.org/
[QubesOS]: https://www.qubes-os.org/
[NixOS Foundation]: https://nixos.org/nixos/foundation.html
[NGI Zero Core]: https://nlnet.nl/core
[NGI Zero Entrust]: https://nlnet.nl/entrust
[NGI Zero PET]: https://nlnet.nl/PET
[NGI Zero Review]: https://nlnet.nl/NGI0/review
[ongoing NGI Assure projects]: https://nlnet.nl/thema/NGIAssure.html
[ongoing NGI Zero Core projects]: https://nlnet.nl/thema/NGIZeroCore.html
[ongoing NGI Zero Entrust projects]: https://nlnet.nl/thema/NGI0Entrust.html
[ongoing NGI Zero PET projects]: https://nlnet.nl/thema/NGIZeroPET.html
[projects of the Internet Hardening Fund]: https://nlnet.nl/internethardening/
[projects of the User-Operated Internet Fund]: https://nlnet.nl/thema/User-operatedInternetFund.html
[background info on NGI Zero Core]: https://nlnet.nl/core/background/index.html
[background info on NGI Zero Entrust]: https://nlnet.nl/entrust/background/index.html
[background info on NGI Zero PET]: https://nlnet.nl/PET/background/index.html
[background info on NGI Zero Review]: https://nlnet.nl/NGI0/review/background/index.html
[pull request towards the NixOS homepage to replace this problematic terminology]: https://github.com/NixOS/nixos-homepage/pull/1077
[Summer of Nix 2022 interview]:https://www.youtube.com/watch?v=I7wdcJ3YhoU&t=310s
[DG CNECT]: https://knowledge4policy.ec.europa.eu/organisation/dg-cnect-dg-communications-networks-content-technology_en
[code.europa.eu]: https://code.europa.eu
[consulting companies available for professional support]: https://nixos.org/community/commercial-support