the phoronix comment section speculation on whether alpine itself will adopt systemd is... frankly hilarious

so, let's talk about it, as one of the people driving the work.

a disclaimer, however: this is just my personal view, so take that accordingly: i have not been a member of the TSC in several years, although i am considering running for both TSC and Council positions. my motives for doing so, however, are entirely unrelated to whether Alpine ships systemd packaging or not.

first: if you are using Alpine in OCI container environments, this work is unlikely to impact you at all. if anything, it will enable smaller container images in some cases.

Alpine on OCI runtimes is not a full distribution, but instead a base image + enough to access the packages distribution.

note I say "packages distribution" here. this is important.

over the past few years, Alpine is growing at a fast rate. we now have downstream distributions which ship their own policies and default package preferences. postmarketOS is an example of a downstream distribution, but there are smaller projects also popping up.

in other words, Alpine has grown into different things for different people:

- the original project: an operating system focused on minimalism
- a toolkit for building custom distributions (like postmarketOS, for example)
- a packaging community (and distribution of binary packages), this is what Docker-type (e.g. OCI) containers primarily interact with when they talk about using Alpine

Alpine users consume one or more of these work streams, but they are distinct.

understanding this, what does this actually mean for systemd in Alpine?

well... yes, it is coming. sort of.

*but*, importantly, it is coming as part of a drive to make the package collection init-system agnostic.

the work to track that is here: https://gitlab.alpinelinux.org/alpine/aports/-/issues/15725

importantly, what is NOT coming? the Alpine Linux operating system itself using systemd. this is not in the cards, has never been in the cards, and most likely will never be in the cards.

this is because OpenRC is a key part of the Alpine OS identity. OpenRC is part of what makes Alpine feel like Alpine.

@ariadne tbh is there any phoronix comment section that isn't chock full of absurd takes and hilarity

@ariadne phoronix comment section is always bad. The recent discussion around gcc adding algol68, GCC diagnostic improvements and even the glibc fma improvements just show how it is bad/hilarious the comment section has become.

the vision here is that:

- people who want an Alpine-like OS with OpenRC use Alpine
- people who want an Alpine-like OS with another init manager use a derivative (in the case of systemd, that will most likely be postmarketOS)
- people who do not need an init system at all (e.g. Docker containers) simply don't use any init system

"but what about libsystemd?"

what about it? the desired end state policy-wise is that we are not linking to any sort of "kitchen sink" library in Alpine packages. this obviously includes libsystemd.

so there will be work to do to allow packages to integrate with systemd, if it is in use, but not integrate with systemd if systemd is not in use.

yes, this work will be hard. but in Alpine we are not afraid of hard work.

the reality is that some Alpine users want to use Alpine-like operating systems with some components swapped out for ones that fit their needs better.

that means Alpine with dinit, or Alpine with s6-rc, and yes, Alpine with systemd.

at the meta-distribution level, the goal is to make all of them work well, so users can make the right decision for their needs.

"but what about *-systemd packages?"

these are systemd unit files. postmarketOS configurations using systemd use them, obviously, but the intention is also to allow them to be used as a source of truth for generating other service definitions in the future.

yes, making that happen will require some work too. but again, in Alpine we are not afraid of hard work.

@ariadne I don't understand. This is exactly what libsystemd is. Integrate it system is used, not integrate if not used.

@waldi we want it to be a *runtime* decision, not a *buildtime* decision

@ariadne 🍿

@ariadne I have said it before and will say it again, and am prepared to argue it in front of any committee with extensive technical study work to back it up: using systemd unit files as a source of truth for generating other service definitions is a very bad idea.

Because the way systemd sees services is entirely holistic, with systemd-specific properties, and that view often maps badly with other approaches. Using unit files as source of truth would skew the service organization in a way that would make it difficult for other service managers to be maintained, and that is the beginning of the end. It is a slippery slope you do not want to tread on if you want to maintain service manager diversity.

If anything, a common declarative independent syntax for service definition should be used, and service files for every other service manager, including systemd if people wish, should be generated from it. I have volunteered to be part of that effort.

but even if you do not want to use systemd itself, there are systemd components that Alpine uses today which are either forked versions or directly used components.

elogind, eudev, etc.

these forked versions lag behind their systemd counterpart. this does not benefit Alpine, as we have to duplicate efforts to keep these components working.

we would like to use the original components instead, even in settings where the init system is not systemd.

@ska sure, i agree with you. but moving from unit files to something even more generic is a smaller leap for us to make.

@ariadne Yes, this is libsystemd. It does the runtime decision.

@waldi yes, what i am telling you is that we *do not* want libsystemd installed in non-systemd settings

@waldi for example: 99% of the time, libsystemd consumers just want sd_notify(). we rather those consumers link against a library which only provides sd_notify().

@waldi remember the xz-utils backdoor? alpine was 100% unaffected, because we will not tolerate linking against a huge library just to get stuff like sd_notify().

@ariadne No. You know how these things work. A temporary state lingers while devs are supposedly working on the final thing; people extend the temporary state because things have to work in the meantime; ultimately too many things depend on it, and the temporary state becomes permanent.

Start with the correct design from the get go. You have already put in some thought into this project; so have @navi and I, and we could get something working rather quickly without the unit file step.

Find someone with extensive knowledge of systemd unit files, who knows what kind of parameters are needed, and who is willing to write the systemd backend; include them into the group; and we can get it done the right way.

(Edit: typo)

@ska the goal is to allow *upstreams* to provide information to us about how they would like their services managed in the distribution. right now these upstreams do this in the form of systemd unit files. i am a pragmatist.

@ariadne Then you are in practice already accepting systemd unit files as a source of truth, and we have already lost. You can now stop pretending Alpine-the-distro is not going to switch to systemd, because ultimately that is what will happen.

@ska deriving service manager data from upstream-provided data does not mean “use systemd as service manager”, even if that data happens to be in the format used by a competing service manager.

a more generic format is lovely, but pointless if it doesn’t get used.

@ariadne Not used? I thought the point was to automatically generate service definitions from it, for all the backends we support.

Creating that format, and publishing it along with the conversion tools, could encourage adoption. Especially if it's easier to write a service file in that format and run the systemd backend than to write the systemd unit file directly.

If Alpine is not afraid of hard work, then I sure as heck hope it's not afraid of medium work that would serve its stated goals better.

@ariadne The alternative, and I know it is not pleasant, requires manpower. It requires someone to go over the service files upstreams are providing you and convert them into the agnostic language. Most of the time this will be easy. Sometimes this will not.

I volunteer to do part of this work, because I have experience in the matter. I think it is necessary if we want to avoid putting a finger in the systemd gears.

@ska @ariadne the way i understood it is package maintainers would use the unit files to map expectations of the upstream project into another init system, without automatic translation, correct me if im wrong though

@SRAZKVT @ska i would say it depends on the service. in many cases, automatic conversion is probably fine. ultimately it would come down to the packager to decide what to do regardless, however.

@SRAZKVT @ariadne That would be a lot less bad, but only an experienced human could do the conversions properly.

Honestly, I think we should stop normalizing the fact that upstreams provide policy. The fuck? Policy is downstream's job, that's what they're here for. (And systemd is to blame for that normalization.)

Distros should start taking back control of how they run their services; if upstream want to provide unit files, sure, why not, but they should be used as hints, as inspiration, not as a source of truth.

@ska @SRAZKVT i think alpine has no qualms about overriding upstreams if we want to do something differently…

@ariadne that cesspool never ceases to amuse me

@wouter @waldi by creating replacement libraries for libsystemd users that are appropriately scoped

@ariadne @waldi that + 100 other stuff + systemd are actually the ones driving the ecosystem to dlopen()

@ska @SRAZKVT @ariadne using downstream services is crazy a lot of work and makes no sense. you have no idea how much stuff changes that you need to adapt, there is so much redundant work between distros with minor differences, but these differences are noticed by users and confuses them. Or like Alpine sometimes does already, just copy Gentoo services, but then Gentoo will be the next upstream for services. Shipping them upstream is clearly the only way forward.

@ska @SRAZKVT @ariadne and if you want to do that the current type is systemd. Changing that requires a lot of work, not in Alpine, but in the whole ecosystem.

@ariadne @waldi It really feels like a mistake by the systemd folks that sd_notify was ever bundled into libsystemd.so, and not provided as a small static library instead.

@fossdd @SRAZKVT @ariadne crazy? that's how it was done for more than 15 years with every distro under the sun. What changed is that systemd came along and made things complex and more work for distros, and distros caved and chose to accept service files from upstream.

If anything, what is crazy is putting the burden of writing the service files on upstream. As you say, there are still differences between distros, and that means that if I want to publish a package, as upstream, I need to write service files for all the distros I want my software in? that's bonkers. Why do upstreams provide policy as systemd files? Because systemd is ubiquitous so providing one unit file will work for all the distros that have adopted systemd, and that's the only sane alternative if upstreams want to get anything done.

At the risk of sounding conspiratorial, it very much looks like it was all done on purpose, and it is infuriating that distros are falling for it.

Yes, packaging services is a lot of work, and that's what distros are for, that is their goddamn job, always has been, and at the same time it is what gives them some power in the ecosystem, and some value: having unified policies that work for all their services. Seeing them all relinquishing their power, giving it all to systemd, is incredibly sad.

@ska @fossdd @ariadne even among distros using systemd, there could be conflicts for example in executable names, if used in a script they would still need to maintain changes to the services (though those are much easier to handle than rewriting the entire service)

@ska @fossdd @ariadne to me it would be much better if upstream provided docs about the environment in which the program is expecting to run, annd maybe a unit file as example, but it shouldn't be much more than that imo

@SRAZKVT @fossdd @ariadne Yes, a hundred times yes. That's what I try to do with documentation when I write something that needs a service file, e.g. tipidee. Give information on what the service needs to run, how to provide it with the correct environment; and let users, or in this case the distro maintainers, come up with the details of how to implement that, because they are the ones who know how their system works.

@fogti @ariadne The systemd maintainers also consider this not longer a good idea. Even a static function is enough.

@ariadne @ska

One relatively complex part of systemd unit files seems to be the sandboxing and resource limits. As far as I am aware there are other (way more universal) ways to deal with that which would then alsow work when no service manager is involved. was it apparmor?

Then there is logging which needs developing a policy to unify this for alpine and then likely patching many service files. Ideally something which mainly does forwarding to (at users request) files or syslog. Services which require config file changes for this could be a big headack.

Readyness ether needs conditional arguments/env or additional compat binaries.

Services with multiple/dynamic instances or similarly complex constructs seem to me to require service manager specific logic with no good way to generalize this.

Configuration of services can hopefully be a service manager specific effort. Should work as long as we do not ship anything which wants to do scripted service reconfiguring.

The rest is likely so simple that maintaing a 3 line spec is probably not a noteworthy maintainance effort.

@sertonix @ariadne s6 and OpenRC have support for resource limits and sandboxing, logging policy (s6 even provides a default logging architecture that every service can override if it wishes), and readiness. This is entirely manageable.

It's an 80/20 situation. 80% of services can be trivially converted to/from unit files; it's the remainder that requires more dedicated effort and more expertise.

@ska @fossdd @SRAZKVT @ariadne What changed is that systemd came along and made the advanced sandboxing, security, and administrative features of the Linux kernel actually usable by everyday software authors and sysadmins.

Now upstreams can provide a service file that works for 99% of cases. Distro maintainers can still patch it if they need to add or drop a line of two. Users can override behaviour with drop in config. And nobody has to rewrite broken init shell scripts.

@ariadne if it's not insane, that's a major step up!

@ska @SRAZKVT @ariadne I have two counterarguments to this:

1. Upstream knows about things like what seccomp filters their service can tolerate and what resources it needs to function properly.
2. Per-distro, per-package work is N*M, where N is the number of distros and M is the number of packages. As both N and M get larger, this is quadratic. Upstream effort is only proportional to the number of packages.

@smlx @fossdd @SRAZKVT @ariadne yeah I'm sure it's exactly what happened, without any more complexity or nuance

@ska @fossdd @SRAZKVT @ariadne In my experience, systemd has a few advantages:

1. It has vastly better interactive debugging capabilities. systemctl status and journalctl -u are incredibly useful. All of these commands work even when a service is being started or stopped.
2. It provides support for an enormous number of Linux sandboxing features, rather than requiring run scripts to reimplement them.
3. It uses cgroups to not only limit resources, but also to keep track of processes. Even if a process double-forks, systemd will not lose track of what service it belongs to. If a service is stopped with the default KillMode, it means that no processes in that service are running.
4. It provides a per-user service manager, which some applications integrate with directly.

That’s not to say that systemd is perfect. It isn’t. That said, in my experience it does the job, and does it well.

@alwayscurious @SRAZKVT @ariadne

1. What resources it needs: yes, that's why it's good to provide an example service file. It is interesting that you mention seccomp filters because these are precisely one of the things that have to be tailored by distros. A distro using musl cannot use the same seccomp filters as a distro using glibc.

2. No, upstream effort is proportional to the number of distros. An upstream that takes upon itself to provide service files and only does it for one generic distro is not doing the job properly, and only creates more work for a lot of downstreams which will have to do the whole work again anyway.

@ska @SRAZKVT @ariadne

1. systemd breaks system calls into groups, abstracting over individual system calls.
2. Which cross-distro differences are you thinking of?

@alwayscurious @SRAZKVT @ariadne

1. That's irrelevant. A function call such as readdir() (random example, I don't know if it's the case) can be implemented wildly differently by different libcs, and a seccomp filter that is good for glibc will not necessarily be good for musl, and vice-versa. It is basically impossible for an application to provide a seccomp filter that works in all cases; that's the very reason why OpenBSD's acme-client stopped supporting Linux. See https://github.com/kristapsdz/acme-client-portable/blob/master/Linux-seccomp.md
2. Well, the most obvious difference is at the root of this thread: if I provide a systemd unit file, this will cause work for distros that use OpenRC and not systemd. There are other differences, but this one is glaring.

@alwayscurious @ska @fossdd @ariadne nobody said it doesn't do some things well. if that were the case, distros wouldn't have gone to it, but it definitely has flaws in its design and how it's implemented

what is said is services overly relying on a single service runner for deployment just isn't viable due to how much of the logic would then be specific to it

@ariadne

importantly, what is NOT coming? the Alpine Linux operating system itself using systemd. this is not in the cards, has never been in the cards, and most likely will never be in the cards.

Do you mean that systemd won’t be packaged in aports (even though there are already other non-OpenRC init systems packaged), or just that Alpine won’t default to using systemd?

@noisytoot the thread covers it. yes, systemd will be in aports. yes, there will be a derivative distribution that uses systemd (postmarketOS today, but probably others later). no we will not start linking everything against libsystemd — if people want packages to integrate better with systemd they will have to find a different solution than that.

@ska @SRAZKVT @ariadne There are only certain system calls that even make sense to use. For instance, readdir() is not going to open a socket in any reasonable implementation. It’s going to be a thin wrapper around malloc and one of the getdents*. syscalls.

Also, if one fails unexpected system calls with ENOSYS, it might well trigger fallback paths meant for older kernels.

For getaddrinfo(), I’d assume that it can allocate AF_INET, AF_INET6, and AF_UNIX sockets and use any of the synchronous filesystem syscalls.

@alwayscurious @ska @SRAZKVT the problem isn’t that, the problem is that musl tends to use newer syscalls as soon as they are available and the seccomp filters have to be fixed to include the new versions. we have to fix lots of seccomp filters in alpine.

@ariadne @ska @SRAZKVT Does musl have fallback paths for old kernels or does it check the kernel version?

@alwayscurious @ska @SRAZKVT yes, it does. but docker for example kills instead of returns ENOSYS.

@ariadne @ska @SRAZKVT I consider that a docker bug.

@alwayscurious @ska @SRAZKVT i agree, but we have to exist in the world we exist in