Upgraded to 23.1.r2: no LAN ip after reboot

[franco]

PS: There is a trash can icon at the right top table corner on the System: Settings: Tunables page. It will reset your tunables to the factory defaults.

[alexdelprete]

However, tunables (sysctls) were not being reinvoked after boot sequence completes in 22.7. That's where the educated guess came from actually.

That explains why, it makes sense now.

If you can reproduce the issue on 22.7.11 with your previous tunables using the following...
# /usr/local/sbin/pluginctl -s sysctl restart

I should reinstall 22.7.11 and restore the old config...is it important for you to know? In that case I will allocate some spare time to do it (during the night when family sleeps...).

It's the tunables itself that cause this (might be good to know which ones cause this) and if not reproducible it's likely 23.1 kernel patching. Yet even in this case it's caused by the "bad" tunables and it would be interesting to narrow this down.

I didn't save them...I deleted all custom ones...they came cherry-picking from this guide: https://calomel.org/freebsd_network_tuning.html

I have old backups of the config in git, can those be used to recover the tunables?

[franco]

Hmm, either way the issue will resurface but at least we will know how to handle it. There isn't too much we can do with 23.1 being out on Thursday and the scope seems limited to me.

Thanks for your help!


Cheers,
Franco

[alexdelprete]

So you want me to do it or not? :)

Is there any way to extract the tunables from a backup of the configuration?

[franco]

Sure, the <sysctl/> section in the config.xml contains the relevant tunables.

If you can find the time to narrow it down that would be helpful indeed.


Cheers,
Franco

[alexdelprete]

Is there a way to decrypt the file to analyze it offline?

In order to identify the tunable, I would have to delete 1 at a time and reboot, correct?

[franco]

I'd extract the current one in plain text from your system and pull an old one that very likely contains it from the backup and decrypt (import) it on the OPNsense to download.

You can then delete all sections except <sysctl/> and do a diff between them. That should narrow it down quite a bit already.


Cheers,
Franco

[alexdelprete]

I forgot I had both google drive backups (encrypted) and GIT backups configured. So I can diff on GH.

I'll let you know the list of tunables...

[alexdelprete]

You can then delete all sections except <sysctl/> and do a diff between them. That should narrow it down quite a bit already.

I downloaded the two copies from GH and diffed manually, in attach the diff file.

I'll also post the contents here, if preferred:

Code Select Expand


76,80d75
<       <tunable>net.inet.tcp.syncookies</tunable>
<       <value>0</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
141,145d135
<       <tunable>net.inet.tcp.tso</tunable>
<       <value>0</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
176,191d165
<       <descr>Enable/disable sending of ICMP redirects in response to IP packets for which a better,
<         and for the sender directly reachable, route and next hop is known.
<       </descr>
<       <tunable>net.inet.ip.redirect</tunable>
<       <value>0</value>
<     </item>
<     <item>
<       <descr>
<         Redirect attacks are the purposeful mass-issuing of ICMP type 5 packets. In a normal network, redirects
<         to the end stations should not be required. This option enables the NIC to drop all inbound ICMP redirect
<         packets without returning a response.
<       </descr>
<       <tunable>net.inet.icmp.drop_redirect</tunable>
<       <value>1</value>
<     </item>
<     <item>
202,317d175
<       <tunable>net.inet.rss.enabled</tunable>
<       <value>0</value>
<       <descr>https://forum.opnsense.org/index.php?topic=24409.0</descr>
<     </item>
<     <item>
<       <tunable>net.isr.bindthreads</tunable>
<       <value>1</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>net.isr.maxthreads</tunable>
<       <value>-1</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>machdep.hyperthreading_allowed</tunable>
<       <value>0</value>
<       <descr>Disable HyperThreading:&#xD;
< https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>hw.em.rx_process_limit</tunable>
<       <value>-1</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>kern.random.harvest.mask</tunable>
<       <value>65887</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>net.inet.tcp.soreceive_stream</tunable>
<       <value>1</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>net.pf.source_nodes_hashsize</tunable>
<       <value>1048576</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>net.inet.ip.maxfragpackets</tunable>
<       <value>0</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>net.inet.ip.maxfragsperpacket</tunable>
<       <value>0</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>dev.igb.0.fc</tunable>
<       <value>0</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>dev.igb.1.fc</tunable>
<       <value>0</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>dev.igb.2.fc</tunable>
<       <value>0</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>dev.igb.3.fc</tunable>
<       <value>0</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>dev.igb.4.fc</tunable>
<       <value>0</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>dev.igb.5.fc</tunable>
<       <value>0</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>net.inet6.ip6.auto_linklocal</tunable>
<       <value>0</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>dev.igb.0.eee_control</tunable>
<       <value>0</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>dev.igb.1.eee_control</tunable>
<       <value>0</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>dev.igb.2.eee_control</tunable>
<       <value>0</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>dev.igb.3.eee_control</tunable>
<       <value>0</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>dev.igb.4.eee_control</tunable>
<       <value>0</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
<       <tunable>dev.igb.5.eee_control</tunable>
<       <value>0</value>
<       <descr>https://calomel.org/freebsd_network_tuning.html</descr>
<     </item>
<     <item>
322,331d179
<     </item>
<     <item>
<       <tunable>net.isr.numthreads</tunable>
<       <value>-1</value>
<       <descr>https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203856#c11</descr>
<     </item>
<     <item>
<       <tunable>net.inet.rss.bits</tunable>
<       <value>1</value>
<       <descr>https://forum.opnsense.org/index.php?topic=24409.0</descr>


[franco]

The ones that stick out are dev.igb.X.eee_control and dev.igb.X.fc -- are you using the standard driver or did you install a different version from Intel itself (as e.g. available in the ports tree)?

The drivers are not really modified between 22.7 and 23.1 so it might have been in there for a while.


Cheers,
Franco

[alexdelprete]

No I never changed from the default driver. I remember having some weird problems and then I found that guide and applied some settings. But basically it's leftovers from 2y ago. I forgot to remove them, but didn't give me problems until 23.x.

Hope it's useful in case someone else has strange issues with 23.x.

Thanks a lot as always for your kind help, it's one of the reasons I chose OPNsense vs pfSense. :)

[Gary7]

I have an apu2d4.
Upgraded from 22.7.11 to 23.1 with no apparent errors.
However, after upgrade, networks (LAN/WAN) didn't work. No response.
Working on the console, I finally tried to manually stop and start the network interfaces.
"ifconfig igb0 down" then "ifconfig igb0 up".  (same for igb1)  Both interfaces started working.
After a reboot, I still needed to stop/start the interfaces to get working.

Previously, I did everything I could think of to optimize network performance including setting dev.igb.X.eee_control=0 in Tunables.
Using the recommendation earlier in this discussion, I deleted dev.igb.X.eee_control from Tunables.
Now, after reboot, network interfaces are starting and working normally.  I'm keeping dev.igb.X.fc=0

Just telling my experience to possibly help others.

[franco]

Thanks, that's helpful! We will try to see if we can reproduce this in the lab and see why this happens.

To reiterate: dev.igb.X.eee_control=0 bad :)


Cheers,
Franco

[ryp43]

can confirm - removing dev.igb.X.eee_control solved the problem.

[alexdelprete]

Thanks, that's helpful! We will try to see if we can reproduce this in the lab and see why this happens.

Franco, is there a way to quickly reset tunables table to default?

Is

Code Select Expand

dev.igb.X.fc=0 recommended for Intel NICs?