Hi,
I upgraded to 23.1.r2, everything seemed to go fine as usual. After reboot, no errors, but I noticed in the console menu where the interfaces with their IPs are listed that the LAN interface had no IP assigned.
I reloaded the services from the menu, and the ip was assigned. Everything good. So I rebooted to check if it was just an incident, but the same behaviour: no IP assigned to LAN on boot. :(
The LAN interface is of type LAGG with static ip configuration. Working since 2y without an issue.
Reloading services works everytime, but obviously it's not an ideal scenario in case I'm not home and the fw reboots (power loss, etc.).
Any hint of what to look for to further debug the issue?
Thanks for any help.
Are you saying a static IP is missing after boot? That's highly unlikely, but perhaps the best step is to look for ifconfig errors first:
# opnsense-log | grep ifconfig
Cheers,
Franco
Hi Franco,
I've tested 3 reboots, what happens is that as soon as it reboots, services are coming up, interfaces go UP, I can actually ping 10.1.10.1, but after 3-4sec. while other services are coming up, the IP is not pingable anymore, the reboot completes, the menu appears and none of the interfaces have the IP, two of these interfaces have a static IP and it shows empty. A reload of all the services fixes everything.
Here's the log you requested:
<171>1 2023-01-23T04:22:41+01:00 OPNsense.axel.dom php-cgi 18472 - [meta sequenceId="169"] /interfaces_lagg_edit.php: The command `/sbin/ifconfig 'igb0' mtu '9000'' failed to execute
<171>1 2023-01-23T04:22:41+01:00 OPNsense.axel.dom php-cgi 18472 - [meta sequenceId="170"] /interfaces_lagg_edit.php: The command `/sbin/ifconfig 'igb1' mtu '9000'' failed to execute
Perhaps you should share your full setup. It's probably hard to reproduce otherwise.
Cheers,
Franco
What do you need exactly? Is there a command to share the complete config? Never needed to ask for support before, OPNsense always worked great. :)
Perhaps for now you merely could post the member configuration of the LAGG in question, the "ifconfig lagg0" output and the IP configuration in the interface settings (that you would expect on it).
At least the ifconfig will show us if it's there. If it is, it might be a firewall rule or otherwise...
Cheers,
Franco
Thanks for the help Franco. I didn't touch any rule before upgrading from 22.7.11. And if it's a rule, it's strange just reloading services fixes the ip binding.
Here's the info requested:
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: LAN (lan)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
ether 02:d4:32:81:89:00
inet 10.1.10.1 netmask 0xfffffe00 broadcast 10.1.11.255
laggproto lacp lagghash l2,l3,l4
laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
(https://i.imgur.com/o4Hqwky.png)
So this looks as expected actually:
inet 10.1.10.1 netmask 0xfffffe00 broadcast 10.1.11.255
(perhaps you posted this in the working state, not sure)
What service(s) are you restarting (or which button or command do you run) and then which part of the system do you access? Is this about the GUI in particular?
Cheers,
Franco
Yes, I'm giving you the config in the working state, from remote. Once I get home, I can reproduce the issue rebooting, and access shell via local console.
In order to fix the issue, after reboot, in the boot menu there is a "Reload all services" option on the right column. Once I run that, the binding is ok.
It's all a bit strange. Reload all services sort of acts like a dry reboot. It does the same things as the boot would minus a few bits.
Cheers,
Franco
Even stranger: during reboot, for some seconds the IP is bound, I can ping OPNsense from the LAN, then after a couple of seconds, while completing reboot, the IP goes away.
It's like one of the scripts unbinds it in some way. I don't know if it's something related to the LAGG, but I don't think so, because it happens also to the ONT_LAN interface, that is a simple interface.
Unfortunately from the console I don't know exactly what to do to give you the most information possible. Because in the non-working state OPNsense is isolated so I can't send anything, I wil do some screenshots with my phone. :)
What commands should I run to debug the non-working state?
Just came home, did a quick test (my family gets angry when OPNsense is down..:):
In non-working state, I ran the opnsense-log command and also ifconfig, here's the output:
OPNsense-log (from timestamps they seem old entries, not related to this reboot):
<171>1 2023-01-23T04:22:41+01:00 OPNsense.axel.dom php-cgi 18472 - [meta sequenceId="169"] /interfaces_lagg_edit.php: The command `/sbin/ifconfig 'igb0' mtu '9000'' failed to execute
<171>1 2023-01-23T04:22:41+01:00 OPNsense.axel.dom php-cgi 18472 - [meta sequenceId="170"] /interfaces_lagg_edit.php: The command `/sbin/ifconfig 'igb1' mtu '9000'' failed to execute
LOG ifconfig:
igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
ether 02:d4:32:81:89:00
hwaddr 40:62:31:0c:0e:e2
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
igb1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
ether 02:d4:32:81:89:00
hwaddr 40:62:31:0c:0e:e3
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
igb2: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
ether 40:62:31:0c:0e:e4
media: Ethernet autoselect
status: no carrier
nd6 options=9<PERFORMNUD,IFDISABLED>
igb3: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
ether 40:62:31:0c:0e:e5
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
igb4: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
ether 40:62:31:0c:0e:e5
hwaddr 40:62:31:0c:0e:e6
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
igb5: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
ether 40:62:31:0c:0e:e5
hwaddr 40:62:31:0c:0e:e7
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
enc0: flags=41<UP,RUNNING> metric 0 mtu 1536
groups: enc
nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33160
groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
syncpeer: 0.0.0.0 maxupd: 128 defer: off
syncok: 1
groups: pfsync
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: LAN (lan)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
ether 02:d4:32:81:89:00
laggproto lacp lagghash l2,l3,l4
laggport: igb0 flags=0<>
laggport: igb1 flags=0<>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: ONT_LAN (opt2)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
ether 40:62:31:0c:0e:e5
laggproto lacp lagghash l2,l3,l4
laggport: igb3 flags=0<>
laggport: igb4 flags=0<>
laggport: igb5 flags=0<>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
vlan01: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: ONT_WAN (opt3)
options=4000000<NOMAP>
ether 40:62:31:0c:0e:e5
groups: vlan
vlan: 835 vlanproto: 802.1q vlanpcp: 7 parent interface: lagg1
media: Ethernet autoselect
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
pppoe0: flags=8890<POINTOPOINT,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN_FTTH (wan)
groups: WAN_GROUP
nd6 options=9<PERFORMNUD,IFDISABLED>
This is the end of the reboot, you can see no IPs are bound:
(https://i.imgur.com/HRzs28C.jpg)
Then I choose option 11 (Reload all services) and after it ends, everything's ok, the IPs are bound:
(https://i.imgur.com/ielC8AZ.jpg)
It seems like on reboot, some services don't complete and when triggered manually they do...but I'm just speculating...
Let me know if you need other info, and thanks again.
Funky... does running the following script break it again and require a reload?
# /usr/local/etc/rc.syshook.d/start/90-sysctl
If yes, what sysctl values do you set? The same would potentially happen from the tunables GUI.
Cheers,
Franco
You're the man Franco! :)
I quickly checked Tunables, and I had 3-4 entries that I used when trying to solve the PPPoE performance issues. Now I deleted all custom entries, rebooted, and everything's good again.
Questions:
- why this problem only with v23? With v22 I had no issues, even with those tunables.
- is there a way to reset the tunables table to default with a command? I'm not sure everything's at default now.
Thanks again...now I'll make a bectl snapshot, just in case.
The new kernel patching in 23.1 may react differently to the tunables being used. It's half way to FreeBSD 13.2 now (not released yet) vs. the more officially aligned FreeBSD 13.1 reference used in 22.7.
However, tunables (sysctls) were not being reinvoked after boot sequence completes in 22.7. That's where the educated guess came from actually.
If you can reproduce the issue on 22.7.11 with your previous tunables using the following...
# /usr/local/sbin/pluginctl -s sysctl restart
It's the tunables itself that cause this (might be good to know which ones cause this) and if not reproducible it's likely 23.1 kernel patching. Yet even in this case it's caused by the "bad" tunables and it would be interesting to narrow this down.
Cheers,
Franco
PS: There is a trash can icon at the right top table corner on the System: Settings: Tunables page. It will reset your tunables to the factory defaults.
Quote from: franco on January 23, 2023, 11:09:45 PM
However, tunables (sysctls) were not being reinvoked after boot sequence completes in 22.7. That's where the educated guess came from actually.
That explains why, it makes sense now.
Quote
If you can reproduce the issue on 22.7.11 with your previous tunables using the following...
# /usr/local/sbin/pluginctl -s sysctl restart
I should reinstall 22.7.11 and restore the old config...is it important for you to know? In that case I will allocate some spare time to do it (during the night when family sleeps...).
Quote
It's the tunables itself that cause this (might be good to know which ones cause this) and if not reproducible it's likely 23.1 kernel patching. Yet even in this case it's caused by the "bad" tunables and it would be interesting to narrow this down.
I didn't save them...I deleted all custom ones...they came cherry-picking from this guide: https://calomel.org/freebsd_network_tuning.html
I have old backups of the config in git, can those be used to recover the tunables?
Hmm, either way the issue will resurface but at least we will know how to handle it. There isn't too much we can do with 23.1 being out on Thursday and the scope seems limited to me.
Thanks for your help!
Cheers,
Franco
So you want me to do it or not? :)
Is there any way to extract the tunables from a backup of the configuration?
Sure, the <sysctl/> section in the config.xml contains the relevant tunables.
If you can find the time to narrow it down that would be helpful indeed.
Cheers,
Franco
Is there a way to decrypt the file to analyze it offline?
In order to identify the tunable, I would have to delete 1 at a time and reboot, correct?
I'd extract the current one in plain text from your system and pull an old one that very likely contains it from the backup and decrypt (import) it on the OPNsense to download.
You can then delete all sections except <sysctl/> and do a diff between them. That should narrow it down quite a bit already.
Cheers,
Franco
I forgot I had both google drive backups (encrypted) and GIT backups configured. So I can diff on GH.
I'll let you know the list of tunables...
Quote from: franco on January 24, 2023, 01:28:25 PM
You can then delete all sections except <sysctl/> and do a diff between them. That should narrow it down quite a bit already.
I downloaded the two copies from GH and diffed manually, in attach the diff file.
I'll also post the contents here, if preferred:
76,80d75
< <tunable>net.inet.tcp.syncookies</tunable>
< <value>0</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
141,145d135
< <tunable>net.inet.tcp.tso</tunable>
< <value>0</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
176,191d165
< <descr>Enable/disable sending of ICMP redirects in response to IP packets for which a better,
< and for the sender directly reachable, route and next hop is known.
< </descr>
< <tunable>net.inet.ip.redirect</tunable>
< <value>0</value>
< </item>
< <item>
< <descr>
< Redirect attacks are the purposeful mass-issuing of ICMP type 5 packets. In a normal network, redirects
< to the end stations should not be required. This option enables the NIC to drop all inbound ICMP redirect
< packets without returning a response.
< </descr>
< <tunable>net.inet.icmp.drop_redirect</tunable>
< <value>1</value>
< </item>
< <item>
202,317d175
< <tunable>net.inet.rss.enabled</tunable>
< <value>0</value>
< <descr>https://forum.opnsense.org/index.php?topic=24409.0</descr>
< </item>
< <item>
< <tunable>net.isr.bindthreads</tunable>
< <value>1</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>net.isr.maxthreads</tunable>
< <value>-1</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>machdep.hyperthreading_allowed</tunable>
< <value>0</value>
< <descr>Disable HyperThreading:
< https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>hw.em.rx_process_limit</tunable>
< <value>-1</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>kern.random.harvest.mask</tunable>
< <value>65887</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>net.inet.tcp.soreceive_stream</tunable>
< <value>1</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>net.pf.source_nodes_hashsize</tunable>
< <value>1048576</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>net.inet.ip.maxfragpackets</tunable>
< <value>0</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>net.inet.ip.maxfragsperpacket</tunable>
< <value>0</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>dev.igb.0.fc</tunable>
< <value>0</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>dev.igb.1.fc</tunable>
< <value>0</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>dev.igb.2.fc</tunable>
< <value>0</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>dev.igb.3.fc</tunable>
< <value>0</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>dev.igb.4.fc</tunable>
< <value>0</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>dev.igb.5.fc</tunable>
< <value>0</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>net.inet6.ip6.auto_linklocal</tunable>
< <value>0</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>dev.igb.0.eee_control</tunable>
< <value>0</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>dev.igb.1.eee_control</tunable>
< <value>0</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>dev.igb.2.eee_control</tunable>
< <value>0</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>dev.igb.3.eee_control</tunable>
< <value>0</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>dev.igb.4.eee_control</tunable>
< <value>0</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
< <tunable>dev.igb.5.eee_control</tunable>
< <value>0</value>
< <descr>https://calomel.org/freebsd_network_tuning.html</descr>
< </item>
< <item>
322,331d179
< </item>
< <item>
< <tunable>net.isr.numthreads</tunable>
< <value>-1</value>
< <descr>https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203856#c11</descr>
< </item>
< <item>
< <tunable>net.inet.rss.bits</tunable>
< <value>1</value>
< <descr>https://forum.opnsense.org/index.php?topic=24409.0</descr>
The ones that stick out are dev.igb.X.eee_control and dev.igb.X.fc -- are you using the standard driver or did you install a different version from Intel itself (as e.g. available in the ports tree)?
The drivers are not really modified between 22.7 and 23.1 so it might have been in there for a while.
Cheers,
Franco
No I never changed from the default driver. I remember having some weird problems and then I found that guide and applied some settings. But basically it's leftovers from 2y ago. I forgot to remove them, but didn't give me problems until 23.x.
Hope it's useful in case someone else has strange issues with 23.x.
Thanks a lot as always for your kind help, it's one of the reasons I chose OPNsense vs pfSense. :)
I have an apu2d4.
Upgraded from 22.7.11 to 23.1 with no apparent errors.
However, after upgrade, networks (LAN/WAN) didn't work. No response.
Working on the console, I finally tried to manually stop and start the network interfaces.
"ifconfig igb0 down" then "ifconfig igb0 up". (same for igb1) Both interfaces started working.
After a reboot, I still needed to stop/start the interfaces to get working.
Previously, I did everything I could think of to optimize network performance including setting dev.igb.X.eee_control=0 in Tunables.
Using the recommendation earlier in this discussion, I deleted dev.igb.X.eee_control from Tunables.
Now, after reboot, network interfaces are starting and working normally. I'm keeping dev.igb.X.fc=0
Just telling my experience to possibly help others.
Thanks, that's helpful! We will try to see if we can reproduce this in the lab and see why this happens.
To reiterate: dev.igb.X.eee_control=0 bad :)
Cheers,
Franco
can confirm - removing dev.igb.X.eee_control solved the problem.
Quote from: franco on January 27, 2023, 09:12:23 AM
Thanks, that's helpful! We will try to see if we can reproduce this in the lab and see why this happens.
Franco, is there a way to quickly reset tunables table to default?
Is
dev.igb.X.fc=0 recommended for Intel NICs?
From my experience I've never had to deal with disabling flow control for speed...
> Franco, is there a way to quickly reset tunables table to default?
Quote from: franco on January 23, 2023, 11:12:50 PM
PS: There is a trash can icon at the right top table corner on the System: Settings: Tunables page. It will reset your tunables to the factory defaults.
Cheers,
Franco
Sorry, I totally missed your post and the trash can. :)
Thanks a lot.
I was configuring my network optimization based on information from https://calomel.org/freebsd_network_tuning.html
I disabled flow-control (dev.igb.0.fc=0 # (default 3)) as described in this section of sysctl configuration:
# Intel i350-T2 igb(4): flow control manages the rate of data transmission
# between two nodes preventing a fast sender from overwhelming a slow receiver.
# Ethernet "PAUSE" frames will pause transmission of all traffic types on a
# physical link, not just the individual flow causing the problem. By disabling
# physical link flow control the link instead relies on native TCP or QUIC UDP
# internal congestion control which is peer based on IP address and more fair
# to each flow. The options are: (0=No Flow Control) (1=Receive Pause)
# (2=Transmit Pause) (3=Full Flow Control, Default). A value of zero(0)
# disables ethernet flow control on the Intel igb(4) interface.
# http://virtualthreads.blogspot.com/2006/02/beware-ethernet-flow-control.html
#
dev.igb.0.fc=0 # (default 3)
Admittedly, on my lightly-loaded home network, it probably makes no difference whether flow control is on or off.
I'm just trying to get best performance out of my little apu2d4 firewall.
Good day, maybe this can help someone in the future
I can also confirm that removing the tunable dev.igb.X.eee_control solves the issues I had with the interfaces.
I use a HP T620 plus with an Intel i340-t4 card running OPNsense bare metal.
Here an overview of what happened:
- After applying the update and bringing the system to v23.1, the interfaces (igb0-igb3) went up for 2 minutes.
- The PPPoE connection on igb0 dropped after two minutes, giving the error: ppp >> [opt2_link0] LCP: no reply to 1 echo request(s).
- Logically, the virtual IPTV WAN on igb1 went down as well.
- The LAN interfaces igb2 and igb3 are bonded in a LAGG, which also went down after 2 minutes.
- The internal Realtek NIC was not affected, which makes sense.
I somehow managed to get a public IP address by restoring the PPPoE connection, with the tunable still in place.
But ipv6 and the IPTV connection didn't want to connect regardless.
By removing the tunable, all is well now.
Thanks for the heads-up.
P.S. making ZFS snapshots before an update saved my day. Rollback went really smooth and was able to be back online within 10 minutes.