Raspberry Pi dhcpcd upgrade breaks network on Raspbian bullseye

Part 1: The upgrade of the month

My Raspberry Pi is running on Raspbian 11 (code name bullseye) since the end of August.

On September 27, I performed my system upgrade routine using a simple

# apt update
# apt upgrade

The apt history looks like

Start-Date: 2021-09-27  18:15:04
Commandline: apt upgrade
Requested-By: username (42)
Install: [...]
Upgrade: [...], dhcpcd5:armhf (1:8.1.2-1+rpt1, 1:8.1.2-1+rpt2), [...], raspberrypi-kernel:armhf (...), [...]
End-Date: 2021-09-27  18:20:53

so the dhcpcd5 package (the DHCP client package) was being upgraded.

Nothing to worry about, so the upgrade was performed 🧘🏻.

Everything went fine and no error was reported 👍.

My Raspberry Pi kernel had been upgraded, and no errors was reported, so why should I worry ? So I hit the reboot command and waited for my Raspberry Pi to bring back online.

I waited… and waited… and waited again… ⏳ … and it never came back.


That moment I realized my Raspberry Pi was hosted remotely That moment I realized my Raspberry Pi was hosted remotely

Part 2: Is this me or network is unreachable ?

After a few days of outage, I finally plugged an HDMI cable and a keyboard to the unreachable Raspberry Pi.

The error was easy to spot:

systemd[297]: dhcpcd.service: Failed to locate executable /usr/lib/dhcpcd5/dhcpcd: No such file or directory
systemd[297]: dhcpcd.service: Failed at step EXEC spawning /usr/lib/dhcpcd5/dhcpcd: No such file or directory
systemd[1]: dhcpcd.service: Control process exited, code=exited, status=203/EXEC
systemd[1]: dhcpcd.service: Failed with result 'exit-code'.
systemd[1]: Failed to start DHCP Client Daemon.
systemd[1]: dhcpcd.service: Scheduled restart job, restart counter is at 5.
systemd[1]: Stopped DHCP Client Daemon.
systemd[1]: dhcpcd.service: Start request repeated too quickly.
systemd[1]: dhcpcd.service: Failed with result 'exit-code'.
systemd[1]: Failed to start DHCP Client Daemon.

Okay, so my Raspberry Pi box was unable to obtain an IP adress because its DHCP client was failing to start. That’s a new one.


So let’s investigate this dhcpcd service:

root@raspberrypi:~# systemctl cat dhcpcd
# /lib/systemd/system/dhcpcd.service
[Unit]
Description=DHCP Client Daemon
Wants=network.target
Before=network-online.target
Documentation=man:dhcpcd(8)

[Service]
Type=forking
ExecStart=/usr/sbin/dhcpcd -b
Restart=always

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/dhcpcd.service.d/wait.conf
[Service]
ExecStart=
ExecStart=/usr/lib/dhcpcd5/dhcpcd -q -w

Okay, so it seems that the /etc/systemd/system/dhcpcd.service.d/wait.conf systemd override file points to a bad dhcpcd executable /usr/lib/dhcpcd5/dhcpcd when the real executable is:

root@raspberrypi:~# type dhcpcd
dhcpcd is /usr/sbin/dhcpcd

Part 3: This Jane, is the Internet !

I’ve specified the good dhcpcd executable in the /etc/systemd/system/dhcpcd.service.d/wait.conf file:

--- /etc/systemd/system/dhcpcd.service.d/wait.conf.orig 2021-10-03 18:43:27.958753332 +0100
+++ /etc/systemd/system/dhcpcd.service.d/wait.conf      2021-10-03 16:02:52.096425389 +0100
@@ -1,3 +1,3 @@
 [Service]
 ExecStart=
-ExecStart=/usr/lib/dhcpcd5/dhcpcd -q -w
+ExecStart=/usr/sbin/dhcpcd -q -w

and then restarted the service using

# systemctl daemon-reload
# systemctl restart dhcpcd

and voila, my Raspberry Pi box has obtained an IP adress and able to talk to the world again !

External resources