OPNsense as VM in Xen: Network interfaces down after Debian dom0 updates

[astronaut]

Hi,

it seems that I have run into the same issue as https://forum.opnsense.org/index.php?topic=28174.msg136837#msg136837. I have OPNsense running in the latest version (22.1.8_1) as a domU in Xen 4.14.5-pre. Debian Bullseye is running as the dom0. I updated the debian dom0 recently, and today, after restarting the dom0 for the first time since updating it, OPNsense network interfaces are down.

During OPNsense boot, for each virtual network interface, there appears the message "reconfiguring interface due to feature change". After that, the interfaces are down. I can ping my own IP, but no other IPs, error is "interface is down". "ip a" in dom0 shows that the virtual interfaces in dom0 are down. The corresponding network bridge seems to be up. There are no obvious errors in dom0 logs. Other domU networks (Debian, OpenWRT) are working normally. For cross checking reasons, I tried attaching only one virtual interface to OPNsense instead of my original three, but the result was the same.

Exemplary dmesg output in OPNsense:

Code: [Select]

xn0: 2 link states coalesced
xn0: link state changed to UP
lo0: link state changed to UP
xn0: performing interface reset due to feature change
xn0: backend features: feature-sg feature-gso-tcp4
xn0: performing interface reset due to feature change

ifconfig in OPNsens shows for the virtual network interface:

Code: [Select]

xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        ether xx:xx:xx:xx:xx:xx
        inet xxx.xxx.xxx.x netmask 0xffffff00 broadcast xxx.xxx.xxx.xxx
        inet6 xxxx:xxxx:xxxx:xxxx::xxxx prefixlen 64
        inet6 fe80::xxxx:xxxx:xxxx:xxxx%xn0 prefixlen 64 scopeid 0x5
        media: Ethernet manual
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

ip a in dom0 shows for the virtual interface:

Code: [Select]

64: vif12.0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master xenbr3 state DOWN group default qlen 32
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fcff:ffff:feff:ffff/64 scope link
       valid_lft forever preferred_lft forever

Besides the "reconfiguring interface due to feature change" message, I have noticed that for each created xen vif, there is one vif.emu visible in the interfaces list on the dom0 for a short time. The .emu interface seems to be connected to PV drivers, but I don't know enough about this topic to be sure if this is related. Also, I don't know if this behavior is completely normal or not.

There are some seemingly related reports on issues with FreeBSD and Xen, but nothing that helped me pinpoint this specific issue. Does anybody have hints on what good next steps for finding the cause for this error could be? I am not very familiar with FreeBSD and xen networking scripts, so I appreciate any tips or links that help me to dig deeper.

[defaultuserfoo]

Why don't you use KVM/QEMU instead of Xen and Fedora instead of Debian?  OPNsense does work with that if you can figure out passing through the network cards.

[astronaut]

Hi defaultuserfoo,

I appreciate your hint. I might consider KVM and another distribution if there is no other solution. At the moment, I would prefer to find the cause of this issue and get my system up again.

[astronaut]

Hi everybody,

after a lot of fiddling I found out that downgrading Xen packages to 4.14.3+32-g9de3671772-1~deb11u1 and using an older kernel 5.10.0-10-amd64 makes the network usable again. Kernel 5.10.0-14-amd64 did not work. I suspect that the kernel alone is the culprit, but I did not check that thoroughly, might as well be a combination of kernel and xen packages.

BTW, the boot message "reconfiguring interface due to feature change" is still appearing.

Of course, it is still not an ideal situation. I would like to help finding out what the reason is and welcome any directions i might look into...

[astronaut]

One more piece of information: There is a bug report for FreeBSD virtual networks in Xen VM: https://lore.kernel.org/all/CAKhsbWbCa49F3y0xwJ6FpewShNnO5iDngTvDmupdY+E_qj3jGw@mail.gmail.com/T/#m9b4ca5fb26b8940c4f2bb76f6aef42fe68fc385f which sounds related.

In this thread, there is one comment that says it wouldn't matter which kernel and xen version is used, as the error lies in FreeBSD netfront, which contradicts my experience with downgrading xen and kernel and get a running system again. However, one xen developer reported recent changes in the xen netback driver.

Is there a way to find out if the fix has been integrated in the FreeBSD base of OPNsense? If yes, we probably have some other cause...

[tokade]

Hi all,

I can confirm the same behavior for ubuntu 20.04 with xen 4.11.

After upgrading ubuntu server 20.04 from kernel 5.4.0-113 to kernel 5.4.0-117 the opnsense domU (OPNsense 22.1.8_1-amd64) is unusable.

EDIT: Today kernel 5.4.0-120 was available, but same result The bridges used for the guests are all up, but the vifs for the opnsense domU (FreeBSD 13.0-STABLE) are all down, so no network connection to opnsense available.

Kind regards
Torsten

[astronaut]

There is FreeBSD code related to the above mentioned mailing list exchange which needs review: https://reviews.freebsd.org/D33876

[tokade]

Updated my ubuntu to kernel 5.4.0-121 and opnsense to 22.1.9. Neither helped to bring opnsense into a state with interfaces working. Bridges on dom0 still down (no-carrier) while opnsense shows interfaces up.

comibination ubuntu kernel 5.4.0-113 and opnsense 22.1.9 working fine.

[amw]

Debian is only working with kernel 5.10.0-10-amd64. It may work with OPNsense 22.7 (Freebsd 13.1).

Is there someone who can test this?

[spi39492]

Have the same issue.

Tried opnsense 22.1.7, 22.1.8 and 22.1.9, neither one is working with dom0 kernel 5.10.0-15-amd64 or even 4.19.0-20-amd64. Moved back to dom0 kernel 4.19.0-18-amd64.

[astronaut]

IIUC, there have been security related changes in Xen netback driver, which is part of the dom0 kernel. The changes address potential harmful behavior of the netfront driver, which is part of the HVM domU. Previously tolerated behavior now, with new kernels, triggers a stop of the interface.

The only solution (other than downgrading the kernel, which of course should only be a temporary "fix" at most) is to change the behavior of the FreeBSD netfront driver. I am by no means a developer, therefore I am not sure if (a) the fix described in https://reviews.freebsd.org/D33876 does address the issue described in this thread and (b) if it has already been pulled and is e.g. part of FreeBSD 13.1. The fix status "needs review" sounds rather like it, well, needs review before being pulled into FreeBSD.

Hence my two questions to the community:

• Can anybody confirm or reject my hypothesis? How can I find out if the fix has been pulled?
• And, if it turns out my interpretation of this situation is correct, how can the review process be sped up?

Edit: FreeBSD 13.1 changelog does not mention any changes regarding Xen network drivers.

Regarding the kernel 4.19.0-20-amd64 I have no idea why that didn't work. Perhaps the netback changes had already been backported?

[astronaut]

I just had a more thorough look into the FreeBSD review system, and I can confirm my initial assumption that https://reviews.freebsd.org/D33876 is still open. This leads to my second question from above: What can I do to speed up the review process?

[amw]

anyone tried 22.7?

[tokade]

Not yet, but hope it will help with our problem.
Still waiting till some bugs are fixed.

[magenbrot]

anyone tried 22.7?

I just tried 22.7 booting from the DVD iso -> not working, same problem.