I did manage to capture the moment it hung in strace, once (FD 62 is /proc/kmsg):

ewfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0444, st_size=3664, ...}, 0) = 0                                                                                                                  [0/1258]
write(65, "2025-08-30 02:16:58 localhost sh"..., 84) = 84
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0444, st_size=3664, ...}, 0) = 0
write(66, "2025-08-30 02:16:58 localhost sh"..., 84) = 84
read(6, "Service sshd-15 has been disable"..., 1024) = 91
epoll_wait(12, [], 8, 0)                = 0
rt_sigprocmask(SIG_BLOCK, NULL, [HUP INT TERM CHLD], 8) = 0
pselect6(64, [3 63], [], [], {tv_sec=0, tv_nsec=0}, NULL) = 0 (Timeout)
rt_sigprocmask(SIG_BLOCK, NULL, [HUP INT TERM CHLD], 8) = 0
pselect6(63, [3 62], [], [], {tv_sec=0, tv_nsec=0}, NULL) = 0 (Timeout)
epoll_ctl(12, EPOLL_CTL_MOD, 63, {events=EPOLLIN|EPOLLRDHUP|EPOLLONESHOT, data=0x3f}) = 0
epoll_ctl(12, EPOLL_CTL_MOD, 62, {events=EPOLLIN|EPOLLRDHUP|EPOLLONESHOT, data=0x3e}) = -1 ENOENT (No such file or directory)
epoll_ctl(12, EPOLL_CTL_ADD, 62, {events=EPOLLIN|EPOLLRDHUP|EPOLLONESHOT, data=0x3e}) = 0
getpid()                                = 1
write(64, "shepherd[1]: Service sshd-15 has"..., 48) = 48
getpid()                                = 1
write(64, "shepherd[1]: Transient service s"..., 69) = 69
epoll_wait(12, [{events=EPOLLIN, data=0x3f}], 8, 0) = 1
read(6, 0x7efc2b719020, 1024)           = -1 EAGAIN (Resource temporarily unavailable)
epoll_ctl(12, EPOLL_CTL_MOD, 6, {events=EPOLLIN|EPOLLRDHUP|EPOLLONESHOT, data=0x6}) = 0
epoll_ctl(12, EPOLL_CTL_DEL, 62, 0x7ffca20392c4) = 0
epoll_wait(12, [], 8, 0)                = 0
recvfrom(63, "shepherd[1]: Service sshd-15 has"..., 1024, 0, 0x7ffca2039250, [112 => 0]) = 48
rt_sigprocmask(SIG_BLOCK, NULL, [HUP INT TERM CHLD], 8) = 0
pselect6(63, [3 62], [], [], {tv_sec=0, tv_nsec=0}, NULL) = 0 (Timeout)
rt_sigprocmask(SIG_BLOCK, NULL, [HUP INT TERM CHLD], 8) = 0
pselect6(64, [3 63], [], [], {tv_sec=0, tv_nsec=0}, NULL) = 1 (in [63], left {tv_sec=0, tv_nsec=0})
recvfrom(63, "shepherd[1]: Transient service s"..., 1024, 0, 0x7ffca2039250, [112 => 0]) = 69
epoll_wait(12, [], 8, 0)                = 0
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0444, st_size=3664, ...}, 0) = 0
write(65, "2025-08-30 02:16:58 localhost sh"..., 78) = 78
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0444, st_size=3664, ...}, 0) = 0
write(66, "2025-08-30 02:16:58 localhost sh"..., 78) = 78
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0444, st_size=3664, ...}, 0) = 0
write(65, "2025-08-30 02:16:58 localhost sh"..., 99) = 99
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0444, st_size=3664, ...}, 0) = 0
write(66, "2025-08-30 02:16:58 localhost sh"..., 99) = 99
epoll_wait(12, [], 8, 0)                = 0
rt_sigprocmask(SIG_BLOCK, NULL, [HUP INT TERM CHLD], 8) = 0
pselect6(64, [3 63], [], [], {tv_sec=0, tv_nsec=0}, NULL) = 0 (Timeout)
rt_sigprocmask(SIG_BLOCK, NULL, [HUP INT TERM CHLD], 8) = 0
pselect6(63, [3 62], [], [], {tv_sec=0, tv_nsec=0}, NULL) = 0 (Timeout)
epoll_ctl(12, EPOLL_CTL_MOD, 63, {events=EPOLLIN|EPOLLRDHUP|EPOLLONESHOT, data=0x3f}) = 0
epoll_ctl(12, EPOLL_CTL_MOD, 62, {events=EPOLLIN|EPOLLRDHUP|EPOLLONESHOT, data=0x3e}) = -1 ENOENT (No such file or directory)
epoll_ctl(12, EPOLL_CTL_ADD, 62, {events=EPOLLIN|EPOLLRDHUP|EPOLLONESHOT, data=0x3e}) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [HUP INT TERM CHLD], 8) = 0
epoll_wait(12, [{events=EPOLLIN, data=0x1b}], 8, 1199999) = 1

My old server (coconut) died earlier today (I think the PSU failed) but I still haven’t moved anything off of it, so I moved the drive to another desktop (a librebooted OptiPlex 7020) to keep it running on new hardware so I can keep procrastinating moving everything to hazelnut (the new server, which currently only runs Akkoma).

Apart from network interface names being different (enp4s0 became enp0s25), this should have worked perfectly, but it didn’t…

Shortly after booting, shepherd (the init system Guix uses) started using 100% CPU and I couldn’t log in on a tty, SSH in, or use sudo (it just hung after the password prompt forever). This was very weird, because exactly the same software had worked fine on the previous hardware with no such problems for over 5 years. I tried booting an older generation and it seemed to work. I then updated the system (I couldn’t stay on the older generation because it was vulnerable to several kernel LPEs) and rebooted. After rebooting to the new system, the issue happened again. This happened several times, including on the older version that I initially thought was unaffected.

I have no idea what could be causing this. It seems like it must be hardware-related since it never occurred on the old hardware, but what hardware issue could cause such as specific issue? I thought it might be filesystem corruption, but btrfs scrub reports no errors.

It’s also not consistently reproducible. Usually it happened within the first few minutes, but now it has an uptime of over an hour and still hasn’t happened again.