OPNsense Forum
Archive => 22.1 Legacy Series => Topic started by: astronaut on June 11, 2022, 10:11:00 pm
-
Hi,
it seems that I have run into the same issue as https://forum.opnsense.org/index.php?topic=28174.msg136837#msg136837 (https://forum.opnsense.org/index.php?topic=28174.msg136837#msg136837). I have OPNsense running in the latest version (22.1.8_1) as a domU in Xen 4.14.5-pre. Debian Bullseye is running as the dom0. I updated the debian dom0 recently, and today, after restarting the dom0 for the first time since updating it, OPNsense network interfaces are down.
During OPNsense boot, for each virtual network interface, there appears the message "reconfiguring interface due to feature change". After that, the interfaces are down. I can ping my own IP, but no other IPs, error is "interface is down". "ip a" in dom0 shows that the virtual interfaces in dom0 are down. The corresponding network bridge seems to be up. There are no obvious errors in dom0 logs. Other domU networks (Debian, OpenWRT) are working normally. For cross checking reasons, I tried attaching only one virtual interface to OPNsense instead of my original three, but the result was the same.
Exemplary dmesg output in OPNsense:
xn0: 2 link states coalesced
xn0: link state changed to UP
lo0: link state changed to UP
xn0: performing interface reset due to feature change
xn0: backend features: feature-sg feature-gso-tcp4
xn0: performing interface reset due to feature change
ifconfig in OPNsens shows for the virtual network interface:
xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN
ether xx:xx:xx:xx:xx:xx
inet xxx.xxx.xxx.x netmask 0xffffff00 broadcast xxx.xxx.xxx.xxx
inet6 xxxx:xxxx:xxxx:xxxx::xxxx prefixlen 64
inet6 fe80::xxxx:xxxx:xxxx:xxxx%xn0 prefixlen 64 scopeid 0x5
media: Ethernet manual
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ip a in dom0 shows for the virtual interface:
64: vif12.0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master xenbr3 state DOWN group default qlen 32
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
inet6 fe80::fcff:ffff:feff:ffff/64 scope link
valid_lft forever preferred_lft forever
Besides the "reconfiguring interface due to feature change" message, I have noticed that for each created xen vif, there is one vif.emu visible in the interfaces list on the dom0 for a short time. The .emu interface seems to be connected to PV drivers, but I don't know enough about this topic to be sure if this is related. Also, I don't know if this behavior is completely normal or not.
There are some seemingly related reports on issues with FreeBSD and Xen, but nothing that helped me pinpoint this specific issue. Does anybody have hints on what good next steps for finding the cause for this error could be? I am not very familiar with FreeBSD and xen networking scripts, so I appreciate any tips or links that help me to dig deeper.
-
Why don't you use KVM/QEMU instead of Xen and Fedora instead of Debian? OPNsense does work with that if you can figure out passing through the network cards.
-
Hi defaultuserfoo,
I appreciate your hint. I might consider KVM and another distribution if there is no other solution. At the moment, I would prefer to find the cause of this issue and get my system up again.
-
Hi everybody,
after a lot of fiddling I found out that downgrading Xen packages to 4.14.3+32-g9de3671772-1~deb11u1 and using an older kernel 5.10.0-10-amd64 makes the network usable again. Kernel 5.10.0-14-amd64 did not work. I suspect that the kernel alone is the culprit, but I did not check that thoroughly, might as well be a combination of kernel and xen packages.
BTW, the boot message "reconfiguring interface due to feature change" is still appearing.
Of course, it is still not an ideal situation. I would like to help finding out what the reason is and welcome any directions i might look into...
-
One more piece of information: There is a bug report for FreeBSD virtual networks in Xen VM: https://lore.kernel.org/all/CAKhsbWbCa49F3y0xwJ6FpewShNnO5iDngTvDmupdY+E_qj3jGw@mail.gmail.com/T/#m9b4ca5fb26b8940c4f2bb76f6aef42fe68fc385f (https://lore.kernel.org/all/CAKhsbWbCa49F3y0xwJ6FpewShNnO5iDngTvDmupdY+E_qj3jGw@mail.gmail.com/T/#m9b4ca5fb26b8940c4f2bb76f6aef42fe68fc385f) which sounds related.
In this thread, there is one comment that says it wouldn't matter which kernel and xen version is used, as the error lies in FreeBSD netfront, which contradicts my experience with downgrading xen and kernel and get a running system again. However, one xen developer reported recent changes in the xen netback driver.
Is there a way to find out if the fix has been integrated in the FreeBSD base of OPNsense? If yes, we probably have some other cause...
-
Hi all,
I can confirm the same behavior for ubuntu 20.04 with xen 4.11.
After upgrading ubuntu server 20.04 from kernel 5.4.0-113 to kernel 5.4.0-117 the opnsense domU (OPNsense 22.1.8_1-amd64) is unusable.
EDIT: Today kernel 5.4.0-120 was available, but same result The bridges used for the guests are all up, but the vifs for the opnsense domU (FreeBSD 13.0-STABLE) are all down, so no network connection to opnsense available.
Kind regards
Torsten
-
There is FreeBSD code related to the above mentioned mailing list exchange which needs review: https://reviews.freebsd.org/D33876 (https://reviews.freebsd.org/D33876)
-
Updated my ubuntu to kernel 5.4.0-121 and opnsense to 22.1.9. Neither helped to bring opnsense into a state with interfaces working. Bridges on dom0 still down (no-carrier) while opnsense shows interfaces up.
comibination ubuntu kernel 5.4.0-113 and opnsense 22.1.9 working fine.
-
Debian is only working with kernel 5.10.0-10-amd64. It may work with OPNsense 22.7 (Freebsd 13.1).
Is there someone who can test this?
-
Have the same issue.
Tried opnsense 22.1.7, 22.1.8 and 22.1.9, neither one is working with dom0 kernel 5.10.0-15-amd64 or even 4.19.0-20-amd64. Moved back to dom0 kernel 4.19.0-18-amd64.
-
IIUC, there have been security related changes in Xen netback driver, which is part of the dom0 kernel. The changes address potential harmful behavior of the netfront driver, which is part of the HVM domU. Previously tolerated behavior now, with new kernels, triggers a stop of the interface.
The only solution (other than downgrading the kernel, which of course should only be a temporary "fix" at most) is to change the behavior of the FreeBSD netfront driver. I am by no means a developer, therefore I am not sure if (a) the fix described in https://reviews.freebsd.org/D33876 (https://reviews.freebsd.org/D33876) does address the issue described in this thread and (b) if it has already been pulled and is e.g. part of FreeBSD 13.1. The fix status "needs review" sounds rather like it, well, needs review before being pulled into FreeBSD.
Hence my two questions to the community:
- Can anybody confirm or reject my hypothesis? How can I find out if the fix has been pulled?
- And, if it turns out my interpretation of this situation is correct, how can the review process be sped up?
Edit: FreeBSD 13.1 changelog does not mention any changes regarding Xen network drivers.
Regarding the kernel 4.19.0-20-amd64 I have no idea why that didn't work. Perhaps the netback changes had already been backported?
-
I just had a more thorough look into the FreeBSD review system, and I can confirm my initial assumption that https://reviews.freebsd.org/D33876 (https://reviews.freebsd.org/D33876) is still open. This leads to my second question from above: What can I do to speed up the review process?
-
anyone tried 22.7?
-
Not yet, but hope it will help with our problem.
Still waiting till some bugs are fixed.
-
anyone tried 22.7?
I just tried 22.7 booting from the DVD iso -> not working, same problem. :(
-
With the review status of the patch https://reviews.freebsd.org/D33876 (https://reviews.freebsd.org/D33876) and no changelog entry for FreeBSD, I am not surprised that 22.7 has the same issue.
In the meantime, I started working on building OPNsense with the patch included in order to find out if that solves our issue. However, with not much experience in building tools and even less experience in FreeBSD, I did not really get far. (I now have a running FreeBSD installation with XTerm, though. Yay. :)) My first succesful build still had the network issue, but I am not entirely sure if I did everything right. Was somebody else more successful than me?
-
Hi astronaut,
any news on this topic on your side with debian? I still can't boot my ubuntu 20.04 server with the newest kernel otherwise opnsense has no connectivity.
Regards Torsten
-
Hi Torsten,
No news from my side. I didn't get around to do more tests with compiling a kernel with the integrated patch yet.
My first successful OPNsense build _should_ have the patch included, but it didn't solve the networking issue. Because I don't have much experience in configuring kernels and bulding, even less so in FreeBSD, I am not sure if I made a mistake while compiling OPNsense, if there is some setting I need to toggle for using the new xenback driver, or if the patch doesn't solve this issue after all. I can of course share the newly created .iso file for testing if anybody is interested.
My next step woudl be to reduce complexity and test if a vanilla FreeBSD DomU has the same or a very similar networking issue. Then I would compile a patched kernel for vanilla FreeBSD to find out if the patch solves the networking issue there. If it does, then I would continue towards finding out why the patch doesn't seem to work in OPNsense. But my time is limited...
Any suggestions are welcome. :-)
-
Good news! I just tested my OPNsense build with patched kernel again on my test DomU, and now the network issue is gone! I can ping other networks, Dom0 bridge and DomU interfaces are up. So it seems to be confirmed that the patch solves the network issue, at least in my setup (Debian 11.5 with kernel 5.10.0-19-amd64 as Dom0, OPNsense 22.7 as DomU).
I don't know if my first test was not done properly or if something else (Dom0 kernel?) changed in the meantime...? Anyway, I will do more tests in the next couple of days on my actual firewall DomU (when my familiy is sleeping :-) and report what I learn from these tests. I can also make available my patched 13.1 kernel, if somebody is interested in testing it. Please let me know.
-
Hi Astronaut,
sounds good so far. I updated my opnsense to 22.7.6 but same when booting with the newest ubuntu kernel - no network.
I would be intrested in your kernel and test it under Ubuntu.
Kind regards
Torsten
-
More good news: I installed the newly compiled kernel today on my "productive" home firewall, and things turned out well. Dom0 is running Debian 11.5, kernel version is 5.10.0-19-amd64, so already a couple of updates after the last functional version 5.10.0-10-amd64. Network in OPNsense 22.7 DomU is up. No changes were required besides installing the patched kernel on a normal 22.7 system. I didn't do any thorough tests, though, so be cautious, there might still be hidden traps.
Here are the instructions on how to compile OPNsense: https://github.com/opnsense/tools
Here is the link to the kernel patch: https://reviews.freebsd.org/D33876 (https://reviews.freebsd.org/D33876)
Note: I enabled options XENHVM
device xenpci in the kernel config, but that's probably not a must.
Here is the instruction on how to install the kernel file:https://docs.opnsense.org/development/how-tos/kernel_debugging.html (https://docs.opnsense.org/development/how-tos/kernel_debugging.html)
If you want to try my 22.7-kernel, send me a PM. Be aware, I am far from knowing precisely what I did. :-)
Of course, it would be nice if the patch could be pulled into an official FreeBSD version and hence OPNsense soon... Anybody out there who could support the review process?
-
Short update: The FreeBSD patch https://reviews.freebsd.org/D33876 (https://reviews.freebsd.org/D33876) was commited today.
-
I can also try and test it if you want.
-
Since 22.7.7 (at least that is the version where I noticed the change), this issue seems to have disappeared. I have the standard Debian Bullseye kernel 5.10.0-19-amd64 installed on Dom0, and OPNsense is running normally, all interfaces are up. No special kernel is needed anymore. Fingers crossed. :-)
-
Since 22.7.7 (at least that is the version where I noticed the change), this issue seems to have disappeared. I have the standard Debian Bullseye kernel 5.10.0-19-amd64 installed on Dom0, and OPNsense is running normally, all interfaces are up. No special kernel is needed anymore. Fingers crossed. :-)
I can't confirm that at least for Debian kernel linux-image-4.19.0-20-amd64 although I am on current opnsense 22.7. The last working kernel version for me is linux-image-4.19.0-18-amd64.
-
I can't confirm that at least for Debian kernel linux-image-4.19.0-20-amd64 although I am on current opnsense 22.7. The last working kernel version for me is linux-image-4.19.0-18-amd64.
I'm now at kernel 5.10.0-20-amd64 and OPNsense 22.7.11, and everything is working flawlessly. You state that you are on OPNsense 22.7. Perhaps upgrading OPNsense helps? As I've written, for me, it only started working again with later versions of OPNsense (e. g. 22.7.7).
If that doesn't help, perhaps trying a newer kernel (e. g. 5.x) is also worth a try?
-
I'm now at kernel 5.10.0-20-amd64 and OPNsense 22.7.11, and everything is working flawlessly. You state that you are on OPNsense 22.7. Perhaps upgrading OPNsense helps? As I've written, for me, it only started working again with later versions of OPNsense (e. g. 22.7.7).
If that doesn't help, perhaps trying a newer kernel (e. g. 5.x) is also worth a try?
I'm also on latest OPNsense 22.7.11-amd64. The next downtime I can try a more recent Debian kernel version.
Have you experienced a change between 4.x and 5.x Debian kernel versions?
Beside updating Debian kernel and opnsense - have you changed any other settings?
Since I virtualize pfsense or opnsense I need to switch off tx checksumming on every opnsense interface on the host opnsense is running, so for instance if opnsense has several virtual interfaces every time I start opnsense I need to run ethtool -K ${int} tx off on the host for every single opnsense interface. With this issue discussed here there is no difference if I disable tx checksumming or not, but that's one of the changes I need to keep in mind.
Update:
I updated my virtualization host to kernel version 5.10.0-20-amd64 as well and can confirm that there are no more issues regarding "reconfiguring interface due to feature change".