Proxmox network interface instability

Proxmox network interface instability

Almost a year ago, I did setup Proxmox on an old gaming machine with 8 CPU and 16 Gb of ram.

I wish I knew earlier about Proxmox as is such a great tool that anyone doing a bit of self-hosting should consider using.

I am not going to explain what Proxmox is, or how to install.

There are many good videos on youtube that cover that in details.

Instead, what I want to dig into, is the Proxmox network interface, that did cause me some trouble.

Everything was working great, until one morning, I woke up and saw that my area was under a power outage, the laptop was off, since the battery life span is very short.

Power came back, rebooted the laptop, moved onto the desktop station, opened the browser just to get greeted by an error.

My DNS server wasn't resolving any address, that meant something went wrong.

I host my own DNS server on Proxmox, so if that is not working I can't surf the internet.

Opening Proxmox on the browser interface wasn't working.

The ping command wasn't reaching the laptop IP address.

The network was fundamentally broken.

After few minutes spent thinking, I opened the lid of my laptop and logged in.

First thing, I checked the status of the virtual machines

root@home:~# qm list
      VMID NAME                 STATUS     MEM(MB)    BOOTDISK(GB) PID       
       100 pi-hole-recursive-dns running    4096              32.00 2179      
       101 Ubuntu               running    4096              40.00 2369      
       102 kasm                 running    4096              65.00 2838      
       103 Azuracast-ubuntu-server running    4096             100.00 5242      
       104 ubuntu-server-searx  stopped    4096              32.00 0      

To my surprise the VM's were up and running but I couldn't load the Proxmox web interface on my Desktop.

Since I wasn't able to ping my laptop, I guessed that maybe just rebooting my router would fix it, but it didn't.

Now, I knew there was some issue with the Proxmox machine connecting to the router.

The router seemed to be working fine as I could ping other devices in the house but not the laptop so I started searching until I found some people saying that the problem could be etc/network/interfaces file in the Proxmox machine.

I was desperate 😁 , which led me to post my interfaces file on Reddit.

root@home:~# cat /etc/network/interfaces
#etc/network/interfaces

auto lo
iface lo inet loopback

iface enp4s0 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.0.11/24
    gateway 192.168.0.1
    bridge-ports enp4s0
    bridge-stp off
    bridge-fd 0

iface wlp3s0 inet manual

A fellow redditor, pointed out that for some reason I was missing a line compared to his interface file.

auto enp4s0

So, I opened up /etc/network/interfaces with nano and edited the file

root@home:~#sudo nano /etc/network/interfaces
#etc/network/interfaces

auto lo
iface lo inet loopback

auto enp4s0

iface enp4s0 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.0.11/24
    gateway 192.168.0.1
    bridge-ports enp4s0
    bridge-stp off
    bridge-fd 0

iface wlp3s0 inet manual

Rebooted the laptop and then everything went back working.

As of today, I still have no idea why that line fixed my problem.

I should probably study a bit more networking. 😁

Of course my problems didn't end there.

Few months later, my area suffered another power outage, that put me back in the same situation as few months prior, the connection between my laptop and the router was gone once again.

This time, I was confident enough that I could fix the issue the same way I did above even though that wasn't really the case. 😥

Didn't know what to do.

Went back to my old friend "stackoverflow" to find an answer but had no luck.

Luckyly Proxmox has a forum, where I opened this thread:

Losing network connection after each Proxmox reboot

Unfortunately, before anyone answered I had already found an fix.

This is what my /etc/network/interfaces looked like currently

root@home:~# cat /etc/network/interfaces
#etc/network/interfaces

auto lo
iface lo inet loopback

auto enp4s0

iface enp4s0 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.0.11/24
    gateway 192.168.0.1
    bridge-ports enp4s0
    bridge-stp off
    bridge-fd 0

iface wlp3s0 inet manual

The /etc/network/interfaces is identical to the one from few months back.

Kept searching and it turns out that there is a command to check if there is an error in /etc/network/interfaces so I gave it a go

ifup vmbr0
error: vmbr0: bridge port enp4s0 does not exist

I finally had the culprit.

The enp4s0 witch stand for ethernet network peripheral # serial # didn't exist.

systemd.net-naming-scheme — Network device naming schemes

So I thought of listing all the Ethernet network interface

root@home:~# ls /sys/class/net
bonding_masters  fwbr101i0  fwbr105i0  fwln102i0  fwpr100p0  fwpr103p0  tap100i0  tap103i0
enp3s0           fwbr102i0  fwln100i0  fwln103i0  fwpr101p0  fwpr105p0  tap101i0  veth105i0
fwbr100i0        fwbr103i0  fwln101i0  fwln105i0  fwpr102p0  lo         tap102i0  vmbr0

or

root@home:~# hwinfo --short --network
network interface:                                              
  fwln102i0            Ethernet network interface
  fwbr101i0            Ethernet network interface
  fwln105i0            Ethernet network interface
  fwpr101p0            Ethernet network interface
  tap100i0             Ethernet network interface
  fwln101i0            Ethernet network interface
  tap103i0             Ethernet network interface
  veth105i0            Ethernet network interface
  fwbr100i0            Ethernet network interface
  enp3s0               Ethernet network interface
  fwpr100p0            Ethernet network interface
  fwbr103i0            Ethernet network interface
  fwpr103p0            Ethernet network interface
  lo                   Loopback network interface
  fwln100i0            Ethernet network interface
  tap102i0             Ethernet network interface
  fwln103i0            Ethernet network interface
  vmbr0                Ethernet network interface
  fwbr102i0            Ethernet network interface
  fwpr102p0            Ethernet network interface
  fwbr105i0            Ethernet network interface
  tap101i0             Ethernet network interface
  fwpr105p0            Ethernet network interface

As you can see there is no enp4s0 network interface but enp3s0, so all I had to do was changing the /etc/network/interfaces to point to enp3s0

root@home:~#sudo nano /etc/network/interfaces
#etc/network/interfaces

auto lo
iface lo inet loopback

auto enp3s0

iface enp3s0 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.0.11/24
    gateway 192.168.0.1
    bridge-ports enp3s0
    bridge-stp off
    bridge-fd 0

iface wlp3s0 inet manual

Rebooted the laptop and solved the problem!

As it was pointed out by a Proxmox staff member "Stoiko Ivanov", here is the root of the issue:

This happens every now and then and is dependent on the BIOS/Motherboard and its pci-device enumeration.

(it's a result of systemd's predictable network interface names) - see e.g.:
https://forum.proxmox.com/threads/latest-systemd-udev-breaks-network-device-naming.53238
and
https://forum.proxmox.com/threads/webui-unreachable-after-adding-nvme-drive.85500

Put shortly - usually after you've fixed it it should stay stable (until the next BIOS update or major kernel update)

Show Comments