Messages

Thanks for you effort. I had already considered doing some performance tests with pfsense as well.

I am having the same performance issues. And we are not alone.

In the past I did compare the routing performance of OPNsense and Ubuntu (https://forum.opnsense.org/index.php?topic=18754.msg91547#msg91547). Pretty clear result: OPNsense crap, Ubuntu full wirespeed.

Surprisingly, a rollback to OPNsense 20.1 gives me full wirespeed.

Something seems to be broken in OPNsense.

[Direction]

https://www.mayrhofer.eu.org/post/firewall-throughput-opnsense-openwrt/

When IPsec is active - even if the relevant traffic is not part of the IPsec policy - throughput is decreased by nearly 1/3. This seems like a real performance issue / bug in the FreeBSD/HardenedBSD kernel. I will need to try with VTI based IPsec routing to see if the in-kernel policy matching is a problem.

Well spotted! Exactly the same negative observation here on my end with IPsec policy based VPN. 

Here is a first estimate of how IPsec affects my routing speed in the LAN:

Direction	IPsec enabled	IPsec disabled
Server -> OPnsense -> Client	48.1 MB/s	74.2 MB/s
Server <- OPnsense <- Client	49.9 MB/s	61.1 MB/s

Overall, the routing speed remains very disappointing. Especially considering I had full routing performance up until OPNsense 20.1.

During my testing, I noticed that OPNsense doesn't seem to be utilizing all NIC queues. Two out of four NIC queues process almost no traffic and are bored.

Code Select Expand

dev.ix.2.queue3.rx_packets: 2959840
dev.ix.2.queue2.rx_packets: 2158082
dev.ix.2.queue1.rx_packets: 9861
dev.ix.2.queue0.rx_packets: 4387

dev.ix.2.queue3.tx_packets: 2967255
dev.ix.2.queue2.tx_packets: 2160888
dev.ix.2.queue1.tx_packets: 15955
dev.ix.2.queue0.tx_packets: 8725


Any take on this?

I digged a bit further.

States on the PPPoE interface won't be deleted by interface_bring_down() function for two reasons:

• PPPoE interface will be deleted when killing mpd5. Hence, if (does_interface_exist($realif)) in interface_bring_down() function will always return false and $pfctlflush array will remain empty.
  
  
• Even if the PPPoE interface still would be alive, state killing would not succeed as states by default are floating (i. e. not bound to interfaces). Hence, state killing command mwexecf('/sbin/pfctl -i %s -Fs', $dev); in interface_bring_down() function won't delete any states at all.
  
  That no states are bound to the PPPoE interface can be easily verified with the following command:
  
  pfctl -i pppoe0 -s states

Houston, we have a problem, do we?

What I don't understand is why one would like to kill mpd5 process in this situation.

Thanks. I'll have a look.

In Analogie zu meinem Schaubild müsstest Du bei Dir das blaue LAN-Kabel vom Glasfasermodem mit dem WAN-Port der OPNsense verbinden.

In der OPNsense musst Du dann die WAN-Verbindung einrichten und die Zugangsdaten, welche Du von von der WOBCOM erhalten hast, eintragen. Spezifische Einstellungen/Parameter musst Du ggf. bei der WOBCOM erfragen. Hier kocht jeder Anbieter sein eigenes Süppchen.

During my testing I've experienced one unsightly issue. States won't be killed when the files with the cached IP addresses are deleted from /var/db. Now one might wonder if that can happen. Unfortunately, the answer is yes. The files will be deleted once the pppoe interface will be removed (maybe due to pulling out the WAN network cable or when clicking pppoe disconnect in the GUI).

For that reason I would like to suggest, that the cache files should remain untouched when the pppoe interface will be removed.

Anyone having a clue which script or code snippet deletes the IP cache files when the (pppoe) interface is removed?

[Testing:]

Testing: ('$' means execute the following command on the command line)

• [...]

Result:
The DNS query to 8.8.8.8 times out, because it hits the outdated NAT state table entry and the request will be translated to the wrong (outdated) source IPv4 address. These packets will be dropped by the ISP. As long as the client fires DNS requests to 8.8.8.8 the existing NAT state table entry will NEVER expire, because the timer resets to its starting value on every DNS query.

After manually deleting the outdated NAT state table entry ($ pfctl -k 0.0.0.0/0 -k 8.8.8.8 ), DNS queries to 8.8.8.8 will be successfully answered :-)

For this reason, deleting the outdated NAT state table entries is essential after the WAN IP has changed.

Thanks for the example, which illustrates the problem well.

I agree that it basically affects all TCP / UDP communication.

IMHO, the state table should always be cleaned up for invalid entries when changing the IP address. OPNsense actually does it quite well. For 100% perfection, however, any entries that are referenced should really be removed from the state table.

I therefore suggest that we add the following line to the rc.newwanip script, as already mentioned:

Code Select Expand

mwexec ('/ sbin / pfctl -k 0.0.0.0/0 -k'. $cacheip);

With this additional line we really don't break anything.

Kurze Frage dazu. Wo genau schließt ihr euer Opnsense an? Wie funktioniert es dann mit Festnetztelefon?
Ich habe auch ein Glasfaseranschluss nur nicht von Telekom sondern regionaler Anbieter. Habe im Keller zwei Geräte von dem Anbieter. Von dem ersten mit LWL zu dem zweiten, von dem zweiten cat.7 in die Wohnung an die FritzBox ins LAN1 Buchse, nicht ins WAN. Danach gehe ich auf die Sense ins WAN.
Wie geht man da vor? Anbieter anrufen und fragen ob das geht?

Das scheint mir eine sehr komische Konstruktion zu sein. Aber ohne genaue Gerätebezeichnung können wir eh nur raten.

Mein Setup (Telekom) sieht wie folgt aus:

Code Select Expand


                                                                                                                       
        +--------------+            +--------------+            +--------------+                                       
        |              |        WAN |              | LAGG       |              |                                       
        |  Glasfaser-  +------------+  OPNsense    +------------+  VLAN-       |                                       
        |  modem       |        ix0 |              | ix1        |  Switch      |                                       
        |              |            |              | ix2        |              |                                       
        +--------------+            +--------------+            +-+---+---+----+                                       
                                                                  |   |   |                                             
                                                   -              |   |   |                                             
                                                                  |   |   +--------------+                             
                                                                  |   |   |              |                             
                                                                  |   |   | Fritz!Box    |                             
                                                                  |   |   | (VLAN 100)   |                             
                                                                  |   |   |              |                             
                                                                  |   |   +--------------+                             
                                                                  |   |                                                 
                                                                  |   +--------------+                                 
                                                                  |   |              |                                 
                                                                  |   | WiFI-AP      |                                 
                                                                  |   | (VLAN 200)   |                                 
                                                                  |   |              |                                 
                                                                  |   +--------------+                                 
                                                                  |                                                     
                                                                  +--------------+                                     
                                                                  |              |                                     
                                                                  | ...          |                                     
                                                                  | (VLAN ...)   |                                     
                                                                  |              |                                     
                                                                  +--------------+                                     


Der Verbindungsaufbau erfolgt bei der Telekom via PPPoE. Wichtig ist die Übermittlung der VLAN ID 7 am WAN-Port.

Für Einstellungen Du für Deinen Anbieter vornehmen musst, kann Dir nur Dein Anbieter sagen.

Ich habe meinen Netzwerkgeräten und es sind wirklich zahlreiche allen eine feste IP in der Fritte zugewiesen wie kann ich diese beibehalten oder muss ich dann über OPNsense neue IP-Adressen vergeben?

Persönlich würde ich den DHCP-Server auf der OPNsense laufen lassen. D. h. Du müsstest die IP-Adressen dann neu einrichten.

Sorry for pulling this old thread out again.

Historically, I had enabled the setting "Reset all states when a dynamic IP address changes" to avoid any stale states which would lead to problems with my VoIP setup.

I completely missed out that I don't need this setting any longer as since OPNsense 21.1 WAN IP address changes are detected by rc.newwanip script and states of outdated WAN IPs will be removed from the state stable.

The beauty with the code changes in rc.newwanip script is that LAN connections now keep alive on WAN IP address changes while state entries originating from the old WAN IP will still be killed.

However, I am asking myself if the respective code snippet in rc.newwanip should be amended by mwexec('/sbin/pfctl -k 0.0.0.0/0 -k ' . $cacheip); to ensure that also all state entries destinating at the old WAN IP will be killed. The code snippet would then look like as follows:

Code Select Expand


if (is_ipaddr($cacheip) && $ip != $cacheip && !isset($config['system']['ip_change_kill_states'])) {
        log_error("IP address change detected, killing states of old ip $cacheip");
        mwexec('/sbin/pfctl -k ' . $cacheip);
        mwexec('/sbin/pfctl -k 0.0.0.0/0 -k ' . $cacheip);
}


During my testing I've experienced one unsightly issue. States won't be killed when the files with the cached IP addresses are deleted from /var/db. Now one might wonder if that can happen. Unfortunately, the answer is yes. The files will be deleted once the pppoe interface will be removed (maybe due to pulling out the WAN network cable or when clicking pppoe disconnect in the GUI).

For that reason I would like to suggest, that the cache files should remain untouched when the pppoe interface will be removed.

I've tested the patch. So far the patch works and unbound does not hang on cache-load command any longer.
However, I'm not sure if we get any kind of side effects with this patch.

Interestingly, during my testing I figured out that I don't need the setting "Reset all states when a dynamic IP address changes" any longer. Historically, I had enabled this option to avoid any stale states which would lead to problems with my VoIP setup. I completely missed out that since OPNsense 21.1 WAN IP address changes are detected by rc.newwanip script and that states of the outdated IP will be removed from the state stable.

Regarding state killing I would like to add some more findings and suggestions in this thread https://forum.opnsense.org/index.php?topic=8766.0.

Ich möchte gerne OPNsense mit einem Mini-Forum PC nutzen und meine vorhanden FRITZbox 7590 nur noch als Modem verwenden

Wieso die Fritte als Modem verwenden? Du hast doch ein Glasfasermodem von der Telekom.

Ich hab's bei mir wie folgt gelöst:

         Glasfasermodem <-> OPNsense <-> Switch

Meine alte FritzBox hängt am VLAN-Switch für Telefonie.
WLAN übernimmt bei mir ein separater Access-Point, der ebenfalls am Switch hängt. Die Trennung der WLANs (Privat, Gäste, IoT, ...) erfolgt per VLAN.

Thanks, schnipp!

I'll take a look at your Github tickets.

[Restart of OPNsense with WAN being connected:]

On Github somova has suspected that the problem could be related to the state reset for dial-up connections (which kills all network connections incl. loopback connections).

Because of this, I ran a few tests.

• Restart of OPNsense with WAN being connected:
  unbound hangs on cache-load command
  
• Restart of OPNsense without WAN being connected:
  unbound properly starts up
  
• Reconnecting WAN:
  state reset kicks in, unbound restarts and hangs on cache-load command

Now disabling "Dynamic state reset" under Firewall -> Settings -> Advanced...

• Restart of OPNsense with WAN being connected:
  unbound properly starts up

So, it could well be that somova is right in his assumption.

Hey,

this issue with Unbound still persists in OPNsense 21.7.3_3.

Unound runs briefly and then stops/crashes after a few secs.

As somova pointed out on Github it looks like the problem is that the cache-load command of unbound hangs.

Any solution?