Topics

Over the last couple of months my OPNsense VM is just dying or possibly crashing inside my Proxmox environment and I have to recycle the VM and we loose our internet when the VM hangs/dying/crashes happens.   

This has happened a number of times over the last couple of months just spontaneously and I've tried to patch to the latest patches for OPNsense when this has happened but still crashing sporadically and most recently today.

Any idea what is going and which logs in FreeBSD would help narrow down what is causing this?

It was working for the last 1-1.5 years without issues and suddenly OPNsense is just crashing

I'm currently in the process of tweaking my internet connection to use Baby Jumbo Frames (RFC4638) to set the MTU to 1508 bytes on the physical/virtual ethernet connections that are the underlying "hardware" for a PPPoE interface (MTU set to 1500 byes so full ethernet frame can be passed without fragmenting).

However, I'm noticing that my VLAN interface to the ISP  (my ISP requires that the PPPoE be on a tagged VLAN) has a MTU setting of 1508 in the OPNsense GUI but at the OS level in FreeBSD (Hardened BSD currently but potatoes / tomatoes) the MTU is showing up as 1500.

I'm wondering if this should be considered a bug?    It seems that my pings with MTU 1472 still get through to sites like 1.1.1.1 / 9.9.9.9  (google is doing something funky at 8.8.8.8 ) but it seems appropriate that I may have to use the syshook for "startup" as as a WORKAROUND to force the ifconfig vtnet2_vlan35 to MTU 1508

https://docs.opnsense.org/development/backend/autorun.html
https://forum.opnsense.org/index.php?topic=16159.0

_______________________________________________
I'm virtualizing the OPNsense environment in QEMU/KVM via Proxmox.

Proxmox

Code Select Expand


root@proxmox:~# ifconfig enp1s0f0 | grep mtu
enp1s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1508
root@proxmox:~# ifconfig vmbr2 | grep mtu
vmbr2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1508
root@proxmox:~#


OPNsense

Code Select Expand


root@OPNsense:~ # ifconfig vtnet2 | grep mtu
vtnet2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1508
root@OPNsense:~ # ifconfig vtnet2_vlan35 | grep mtu
vtnet2_vlan35: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500    <------ 1508?
root@OPNsense:~ # ifconfig pppoe0 | grep mtu
pppoe0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1500
root@OPNsense:~ #
root@OPNsense:~ # ping -D -s 1472 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 1472 data bytes
1480 bytes from 1.1.1.1: icmp_seq=0 ttl=56 time=15.369 ms
1480 bytes from 1.1.1.1: icmp_seq=1 ttl=56 time=15.325 ms
1480 bytes from 1.1.1.1: icmp_seq=2 ttl=56 time=15.203 ms
1480 bytes from 1.1.1.1: icmp_seq=3 ttl=56 time=15.326 ms
1480 bytes from 1.1.1.1: icmp_seq=4 ttl=56 time=15.681 ms
1480 bytes from 1.1.1.1: icmp_seq=5 ttl=56 time=15.238 ms
^C
--- 1.1.1.1 ping statistics ---
6 packets transmitted, 6 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 15.203/15.357/15.681/0.156 ms
root@OPNsense:~ #

root@OPNsense:~ # ping -D -s 1472 9.9.9.9
PING 9.9.9.9 (9.9.9.9): 1472 data bytes
1480 bytes from 9.9.9.9: icmp_seq=0 ttl=53 time=47.753 ms
1480 bytes from 9.9.9.9: icmp_seq=1 ttl=53 time=47.782 ms
1480 bytes from 9.9.9.9: icmp_seq=2 ttl=53 time=47.756 ms
1480 bytes from 9.9.9.9: icmp_seq=3 ttl=53 time=47.717 ms
1480 bytes from 9.9.9.9: icmp_seq=4 ttl=53 time=47.807 ms
^C
--- 9.9.9.9 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 47.717/47.763/47.807/0.030 ms
root@OPNsense:~ #

root@OPNsense:~ # ping -D -s 1472 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 1472 data bytes
76 bytes from 8.8.8.8: icmp_seq=0 ttl=116 time=1.770 ms
wrong total length 96 instead of 1500
76 bytes from 8.8.8.8: icmp_seq=1 ttl=116 time=1.914 ms
wrong total length 96 instead of 1500
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.770/1.842/1.914/0.072 ms
root@OPNsense:~ #






Also the question came up in another forum while troubleshooting this if setting the MRU/MTU to 1508 is needed if the max-payload is set to 1500 for PPPoE interface but it seems I was getting issues when MRU wasn't being set or perhaps I was being thrown off by the google's 8.8.8.8 which is my go to for testing stuff when I was initially trying to set this up. 

I'm wondering if this max-payload is being set in the PPPoE interface MTU setting where it calculates it from 1508 -> 1500 or 1500 -> 1492.   Looks like the MRU setting is on the vtnet2_vlan35 interface which PPPoE uses in the advanced settings.

Code Select Expand


set pppoe max-payload 1500 will override mru and mtu
see last part of 5.7 http://mpd.sourceforge.net/doc5/mpd.html


My mpd_want.conf that pppd is using looks like this currently.

Code Select Expand


startup:
  # configure the console
  set console close
  # configure the web server
  set web close

default:
pppoeclient:
  create bundle static wan
  set bundle enable ipv6cp
  set iface name pppoe0
  set iface route default
  set iface disable on-demand
  set iface idle 0
  set iface enable tcpmssfix
  set iface up-script /usr/local/opnsense/scripts/interfaces/ppp-linkup.sh
  set iface down-script /usr/local/opnsense/scripts/interfaces/ppp-linkdown.sh
  set ipcp ranges 0.0.0.0/0 0.0.0.0/0
  create link static wan_link0 pppoe
  set link action bundle wan
  set link disable multilink
  set link keep-alive 10 60
  set link max-redial 0
  set link disable chap pap
  set link accept chap pap eap
  set link disable incoming
  set pppoe max-payload 1500      <------ is this being set by MTU 1508->1500 on pppoe interface page?
  set link mru 1508                      <------ MRU is being set to 1508 on Link Parameters ( vtnet2_vlan35 ) page
  set auth authname "xxxxxxxx"
  set auth password xxxxxxxx
  set pppoe service ""
  set pppoe iface vtnet2_vlan35
open




Many thanks in advance

I emailed the project directly but they said they don't have a lot of resource to look at this but perhaps I can help contribute this change if we can vet it for the OPNsense project.....


This is what I wrote to the project@opensense.org email
_____________________
With OPNsense moving to FreeBSD 13.1 in the new year can you look at enabling TCP BBR congestion control kernel module in the OPNsense project so that people can rate limit their connections using BBR?

Apparently this option in Linux has had great results in people being able to get their full Gigabit internet upload and download speeds because the network interface floods the ONT interface otherwise and Linux users have been using BBR to allow the full data rates.

I use OPNsense having previously used pfSense on FreeBSD 11.x base and that project hasn't kept up with the latest FreeBSD changes.  I don't want to switch to a linux firewall project because they don't have an easy to use interface as seen in pfSense and OPNsense.

I don't think it is anything more than turning some default options and make options in the kernel config to enable the ability to use it as it still requires a sysctl option to turn on     

Just like to avoid having to redo this work if I get it to work every time there is a new release of OPNsense

Perhaps if I get it working after the 22.1 release is out I can provide the steps to OPNsense project for inclusion if it is a simple step and the OPNsense GUI can be enhanced later to allow end user selection of the congestion control algorithm including FQ codel stuff

Can you provide your kernel config file for the 22.1 FreeBSD target?

This is the page referencing TCP BBR for FreeBSD FYI.    I understand it is just these added to kernel configuration file at build but let me test and get back to project

makeoptions WITH_EXTRA_TCP_STACKS=1
options TCPHPTS

https://lists.freebsd.org/pipermail/freebsd-current/2020-April/075930.html

If you can point me to the appropriate forum/relevant section of forum I can also report back my results via the forum and link back to your project when it is proven


As far as real kernel modules I know someone in another forum that developed some changes to the bxe driver to allow syncing at 2.5Gbps but not sure if he submitted upstream to FreeBSD and Linux community (bnx2x) but that's relatively easy to build at different kernel releases.   I can approach FreeBSD project about incorporating those changes