Messages

With pf and ipfw it's quite easy to write a single line rule that lets you define the allowed ICMP and ICMP6 types, but with OpnSense you have to create an individual rule for each type. Can this be refactored to allow selecting multiple types just like you can select multiple interfaces?

�

Code Select Expand


feld@gw:~ $ netstat -I ax0
Name    Mtu Network        Address                             Ipkts                 Ierrs                 Idrop       Opkts                 Oerrs                  Coll
ax0    1500 <Link#5>       f4:90:ea:00:62:2d                  382472  18446744073709551610                     0  1703087022                     0                     0
ax0       - fe80::%ax0/64  fe80::f690:eaff:fe00:622d%ax0           0                     -                     -           0                     -                     -



this is impossible :)

Please expand the configuration options for os-frr or allow us to edit the config file as there is a ton of missing functionality. I really want to use peer groups so I can connect peers/neighbors and accept announcements from any address in a given subnet which is a fairly normal deployment option but this is not possible due to the severely limited configuration.

Sorry about the late reply, but this problem has been resolved for a while. I'm not sure which Opnsense update fixed it, but it did.

I was experiencing this with both wireguard-kmod and wireguard-go

there may be some VLAN 0 voodoo going on?

I have confirmed there is vlan0, if anyone is curious

Code Select Expand


13:41:09.525982 a0:f3:e4:63:0b:7b > f4:90:ea:00:62:2e, ethertype 802.1Q (0x8100), length 70: vlan 0, p 0, ethertype IPv4, 78.192.134.61.13103 > 75.13.68.65.13000: Flags [R.], seq 1, ack 59, win 509, options [nop,nop,TS val 3436954555 ecr 1177160148], length 0
13:41:09.553622 a0:f3:e4:63:0b:7b > f4:90:ea:00:62:2e, ethertype 802.1Q (0x8100), length 60: vlan 0, p 0, ethertype IPv4, 94.102.61.38.50380 > 75.13.68.70.5004: Flags [S], seq 794064754, win 65535, length 0
13:41:09.605744 a0:f3:e4:63:0b:7b > f4:90:ea:00:62:2e, ethertype 802.1Q (0x8100), length 180: vlan 0, p 0, ethertype IPv4, 205.251.197.161.53 > 172.13.126.189.32944: 2930 NXDomain*-$ 0/1/1 (134)


To limit the exposure of the previous patches perhaps this can help? https://github.com/opnsense/src/commit/fb81510bd

To install...

# opnsense-update -zkr 23.1.8_1

This kernel is working. The interface is not flapping now.

The SFP+ module is SFP-10G-T from FS.com that is Juniper coded

https://www.fs.com/products/73107.html?attribute=27&id=425

No netmap, no VLANs on that interface. That's my A&TT upstream interface and I know I had to turn off hardware vlan filtering because I'm bypassing their CPE hardware and there may be some VLAN 0 voodoo going on? I may have to capture some packets and see what they look like.

I will get you the exact model of the SFP+ module tomorrow

The changelog says:

Code Select Expand


src: axgbe: fix link issues for gigabit external SFP PHYs and 100/1000 fiber modules
src: axgbe: apply RRC to miibus attached PHYs and add support for variable bitrate 25G SFP+ DACs
src: axgbe: properly release resource in error case


My dmesg with this kernel shows one of my links constantly flapping:

Code Select Expand


ax1: Link is DOWN
ax1: link state changed to DOWN
ax1: Link is UP - 10Gbps/Full - flow control off
ax1: link state changed to UP
ax1: Link is DOWN
ax1: link state changed to DOWN
ax1: Link is UP - 10Gbps/Full - flow control off
ax1: link state changed to UP
ax1: Link is DOWN
ax1: link state changed to DOWN
ax1: Link is UP - 10Gbps/Full - flow control off
ax1: link state changed to UP
ax1: Link is DOWN
ax1: link state changed to DOWN
ax1: Link is UP - 10Gbps/Full - flow control off
ax1: link state changed to UP
ax1: Link is DOWN
ax1: link state changed to DOWN
ax1: Link is UP - 10Gbps/Full - flow control off
ax1: link state changed to UP
ax1: Link is DOWN
ax1: link state changed to DOWN
ax1: Link is UP - 10Gbps/Full - flow control off
ax1: link state changed to UP
ax1: Link is DOWN
ax1: link state changed to DOWN
ax1: Link is UP - 10Gbps/Full - flow control off
ax1: link state changed to UP
ax1: Link is DOWN
ax1: link state changed to DOWN
ax1: Link is UP - 10Gbps/Full - flow control off
ax1: link state changed to UP
ax1: Link is DOWN
ax1: link state changed to DOWN


Booting into the previous kernel fixes it.

My hardware is the official DEC840

Dropping the negative cache TTL as low as possible is really useful.

e.g., I try to resolve a host that's been offline for a while, then boot up the machine. It gets an IP from DHCP, sets its hostname, and now Unbound should know about it. But it doesn't return any results because there's a negative cache entry still there.

test net.isr.dispatch=deferred vs net.isr.dispatch=direct

[net.isr.dispatch=direct]

Majority of the issue was net.isr.dispatch=direct which should be net.isr.dispatch=deferred so multiple CPU cores are used. I can hit ~7gbit on an iperf to the firewall and I've been able to get my full 2gbit through it.

I don't know why this isn't the default value in Opnsense. I understand why it's not in FreeBSD, but a networking appliance should be tuned out of the box for maximum networking performance. Hope to see this and more auto-tuning improvements in the future.

I also would have expected Opnsense to automatically recognize this hardware and apply specific tuning for it. It is one of their flagship products after all.

The inability to get a full 10gbit iperf to the firewall when the DEC840 spec sheet specifically states "14.4Gbps firewall throughput" and "Firewall Port to Port Throughput: 9Gbps" makes me wonder if the Opnsense team has ever actually hit those numbers with this hardware or if they're just advertising theoretical max?

OpnSense DEC840 which is supposed to be able to handle passing ~15gbit of traffic

Speedtest from the firewall:

Code Select Expand


# speedtest --server-id=47746

   Speedtest by Ookla

      Server: AT&T - Miami, FL (id: 47746)
         ISP: AT&T Internet
Idle Latency:     3.53 ms   (jitter: 0.50ms, low: 3.06ms, high: 4.12ms)
    Download:  2327.36 Mbps (data used: 2.6 GB)
                  5.18 ms   (jitter: 1.65ms, low: 2.79ms, high: 26.40ms)
      Upload:   378.54 Mbps (data used: 685.6 MB)
                  3.01 ms   (jitter: 1.79ms, low: 2.03ms, high: 55.43ms)
Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/bbd0ee99-ad99-4e32-b3c9-ad05daf8bd84


Speedtest through the firewall (notice slow upload)

Code Select Expand


# speedtest --server-id=47746

   Speedtest by Ookla

      Server: AT&T - Miami, FL (id: 47746)
         ISP: AT&T Internet
Idle Latency:     4.17 ms   (jitter: 0.94ms, low: 3.06ms, high: 6.49ms)
    Download:  2295.81 Mbps (data used: 1.5 GB)
                  5.08 ms   (jitter: 2.15ms, low: 2.79ms, high: 53.90ms)
      Upload:   329.78 Mbps (data used: 362.9 MB)
                  4.05 ms   (jitter: 1.37ms, low: 3.12ms, high: 16.97ms)
Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/2f29bb86-def6-4379-ad30-7292ad3e1926


iperf3 from the same machine *to* the Opnsense firewall, normal and reverse

Code Select Expand


root@dev:/ # iperf3 -c gw
Connecting to host gw, port 5201
[  5] local 10.27.3.230 port 31205 connected to 10.27.3.254 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   272 MBytes  2.28 Gbits/sec  413    472 KBytes
[  5]   1.00-2.00   sec   287 MBytes  2.41 Gbits/sec    2    614 KBytes
[  5]   2.00-3.00   sec   255 MBytes  2.14 Gbits/sec   61    593 KBytes
[  5]   3.00-4.00   sec   280 MBytes  2.35 Gbits/sec   23   17.0 KBytes
[  5]   4.00-5.00   sec   261 MBytes  2.19 Gbits/sec   82    257 KBytes
[  5]   5.00-6.00   sec   257 MBytes  2.15 Gbits/sec   14    133 KBytes
[  5]   6.00-7.00   sec   254 MBytes  2.13 Gbits/sec   20    737 KBytes
[  5]   7.00-8.00   sec   260 MBytes  2.18 Gbits/sec   70    512 KBytes
[  5]   8.00-9.00   sec   268 MBytes  2.25 Gbits/sec  140    737 KBytes
[  5]   9.00-10.00  sec   266 MBytes  2.23 Gbits/sec  116    714 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.60 GBytes  2.23 Gbits/sec  941             sender
[  5]   0.00-10.00  sec  2.60 GBytes  2.23 Gbits/sec                  receiver

iperf Done.
root@dev:/ # iperf3 -R -c gw
Connecting to host gw, port 5201
Reverse mode, remote host gw is sending
[  5] local 10.27.3.230 port 12997 connected to 10.27.3.254 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   254 MBytes  2.13 Gbits/sec
[  5]   1.00-2.02   sec   262 MBytes  2.16 Gbits/sec
[  5]   2.02-3.00   sec   257 MBytes  2.19 Gbits/sec
[  5]   3.00-4.00   sec   250 MBytes  2.10 Gbits/sec
[  5]   4.00-5.00   sec   234 MBytes  1.97 Gbits/sec
[  5]   5.00-6.00   sec   244 MBytes  2.05 Gbits/sec
[  5]   6.00-7.00   sec   251 MBytes  2.11 Gbits/sec
[  5]   7.00-8.00   sec   229 MBytes  1.92 Gbits/sec
[  5]   8.00-9.00   sec   248 MBytes  2.08 Gbits/sec
[  5]   9.00-10.00  sec   238 MBytes  1.99 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.41 GBytes  2.07 Gbits/sec   14             sender
[  5]   0.00-10.00  sec  2.41 GBytes  2.07 Gbits/sec                  receiver

iperf Done.


I actually expect more than this. With a loopback to my own server through my switch I can do 9gbit with a single stream. If I do multiple streams to the Opnsense firewall I can hit 4.2gbit max


So where is this mysterious bottleneck coming from? I did have the ipsec.ko loaded from an old setup, but I had no policies. Module completely gone. No amount of tuning or interface settings changes seems to matter.

How do I get this thing to actually push line rate? I've even swapped from 10gbase-t to fiber in case it was something odd with the media, but same results.

edit: I setup another test scenario where I do a speed test over wifi from my laptop to my server using Librespeed and when I hit it directly through my AP on the same switch connected to the server I can do 300/300, but when I force my traffic to go through the firewall (same segment, same VLAN) the download speed (my server's upload) can't break 100

There is something very peculiar going on

> But, the Swap is always around 59%, so several GB worth of swapping at all times.

This only means this data is being stored in swap because it's not being actively used. This does not mean there's a performance issue. You should only worry if you see a lot of page in / page out activity. That would indicate that the memory in swap is being used during normal operations because everything required for the current process is not able to fit into memory.

Keep in mind that FreeBSD's swap behavior is different than Linux.

Let your system use 100% of the swap if it wants to. This is just the kernel optimizing memory for other applications that are making large allocations. Only worry when you see excessive paging.

What I meant by multiple threads is the '-P' option on iperf (if that is what you use) which you absolutely need to measure speeds above gigabit. A single TCP connection will not be sufficient to test at those speeds.

I just did an iperf3 without -P and hit 9gbit, so I don't think that's true.