Messages

Maybe try this:
Go to Services -> UnboundDNS -> Advanced
Then in "Rebind protection networks" remove 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16
And then hit Apply button

Sorry it wont work, I missed that it's about Registering static Mappings

I've made some changes:

On the WAN interface:

• Assigned static IPv6 addresses:
  
  Code Select Expand
  2001:db8:b000::11/48 on wan-lab1
2001:db8:b000::12/48 on wan-lab2

• Configured the IPv6 gateway:
  
  Code Select Expand
  2001:db8:b000::1

• Configured a VIP (WAN Interface CARP):
  
  Code Select Expand
  2001:db8:b000::10/48


On the LAN interface:

• I assigned static IPv6 addresses:
  
  Code Select Expand
  2001:db8:b000:300::1/56 on lan-lab1
2001:db8:b000:300::2/56 on lan-lab2


In ISC DHCPv6 LAN Interface, I have:

• Enable:
  
  Code Select Expand
  True
• Subnet:
  
  Code Select Expand
  2001:db8:b000:300::
• Subnet mask:
  
  Code Select Expand
  /56
• Available range:
  
  Code Select Expand
  2001:db8:b000:300:: - 2001:db8:b000:3ff:ffff:ffff:ffff:ffff
• Range:
  From:
  
  Code Select Expand
  2001:db8:b000:300::3 To:
  
  Code Select Expand
  2001:db8:b000:3ff:ffff:ffff:ffff:ffff

In Router Advertisements for Lan:

• Router Advertisements:
  
  Code Select Expand
  Assisted (tried Unmanaged/Managed)
• Router Priority:
  
  Code Select Expand
  High
• Source Address:
  
  Code Select Expand
  Automatic
• Advertise Default Gateway:
  
  Code Select Expand
  True
• DNS options
  
  Code Select Expand
  Use the DNS configuration of the DHCPv6 server: True

IPv6 "world" directly from lab1/lab2 works fine (I can ping google ipv6 dns).

Now my Lan client host gets IPv6 from Lab1 DHCPv6

First question why it is /128??

Code Select Expand

noc@noc-NUC8i3BEK:~$ ip a s
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether xx:xx:xx:xx:7c:f3 brd ff:ff:ff:ff:ff:ff
    altname enp0s31f6
    inet 10.255.5.30/24 brd 10.255.5.255 scope global dynamic noprefixroute eno1
       valid_lft 200sec preferred_lft 200sec
    inet6 2001:db8:b000:3d4:d156:9f78:d2a8:51bb/128 scope global dynamic noprefixroute
       valid_lft 198sec preferred_lft 85sec
    inet6 fe80::92b2:3746:d197:5546/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
noc@noc-NUC8i3BEK:~$

IPv6 routes on client host:

Code Select Expand

noc@noc-NUC8i3BEK:~$ ip -6 ro
2001:db8:b000:3d4:d156:9f78:d2a8:51bb dev eno1 proto kernel metric 100 pref medium
2001:db8:b000:300::/56 dev eno1 proto ra metric 100 pref medium
fe80::/64 dev eno1 proto kernel metric 1024 pref medium
default via fe80::3eec:efff:fedd:11b4 dev eno1 proto ra metric 20100 pref medium
noc@noc-NUC8i3BEK:~$

When NAT is enabled (Source: LAN_net, Dest: !LAN_Net, NAT Address: 2001:db8:b000::10):

Code Select Expand

noc@noc-NUC8i3BEK:~$ ping 2001:db8:b000::1
PING 2001:db8:b000::1 (2001:db8:b000::1) 56 data bytes
64 bytes from 2001:db8:b000::1: icmp_seq=1 ttl=63 time=0.393 ms
^C
--- 2001:db8:b000::1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.393/0.393/0.393/0.000 ms

noc@noc-NUC8i3BEK:~$ ping 2001:4860:4860::8888
PING 2001:4860:4860::8888 (2001:4860:4860::8888) 56 data bytes
64 bytes from 2001:4860:4860::8888: icmp_seq=1 ttl=116 time=1.39 ms
64 bytes from 2001:4860:4860::8888: icmp_seq=2 ttl=116 time=1.43 ms
^C
--- 2001:4860:4860::8888 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 1.391/1.409/1.427/0.018 ms
noc@noc-NUC8i3BEK:~$

But when NAT rule is disabled:
I can ping lab1-lan ipv6 address:

Code Select Expand

noc@noc-NUC8i3BEK:~$ ping6 2001:db8:b000:300::1
PING 2001:db8:b000:300::1 (2001:db8:b000:300::1) 56 data bytes
64 bytes from 2001:db8:b000:300::1: icmp_seq=1 ttl=64 time=0.240 ms
64 bytes from 2001:db8:b000:300::1: icmp_seq=2 ttl=64 time=0.384 ms
^C
--- 2001:db8:b000:300::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1035ms
rtt min/avg/max/mdev = 0.240/0.312/0.384/0.072 ms

I can ping lab1-wan ipv6 address:

Code Select Expand

noc@noc-NUC8i3BEK:~$ ping6 2001:db8:b000::11
PING 2001:db8:b000::11 (2001:db8:b000::11) 56 data bytes
64 bytes from 2001:db8:b000::11: icmp_seq=1 ttl=64 time=0.314 ms
64 bytes from 2001:db8:b000::11: icmp_seq=2 ttl=64 time=0.385 ms
^C
--- 2001:db8:b000:300::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1034ms
rtt min/avg/max/mdev = 0.314/0.349/0.385/0.035 ms

When I try to ping Lab1 IPv6 WANs GW, I can see that packet is going out from Lab1 and is reaching that GW

Code Select Expand

noc@noc-NUC8i3BEK:~$ ping6 2001:db8:b000::1
PING 2001:db8:b000::1 (2001:db8:b000::1) 56 data bytes
^C
--- 2001:db8:b000::1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2037ms


Code Select Expand

root@lab1:~ # tcpdump -ni lagg0_vlan52 host 2001:db8:b000::1 and not host 2001:db8:b000::11
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on lagg0_vlan52, link-type EN10MB (Ethernet), snapshot length 262144 bytes

12:16:35.053213 IP6 2001:db8:b000:3d4:d156:9f78:d2a8:51bb > 2001:db8:b000::1: ICMP6, echo request, id 5298, seq 1, length 64
12:16:36.066610 IP6 2001:db8:b000:3d4:d156:9f78:d2a8:51bb > 2001:db8:b000::1: ICMP6, echo request, id 5298, seq 2, length 64
12:16:37.090509 IP6 2001:db8:b000:3d4:d156:9f78:d2a8:51bb > 2001:db8:b000::1: ICMP6, echo request, id 5298, seq 3, length 64

tcpdump from that GW:

Code Select Expand

f1b-core01#tcpdump interface vlan 52 verbose filter host 2001:db8:b000::1 and not host 2001:db8:b000::11
tcpdump: listening on vlan52, link-type EN10MB (Ethernet), capture size 262144 bytes

14:54:27.170987 xx:xx:xx:xx:3b:66 > xx:xx:xx:xx:51:bb, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::21c:73ff:fe88:3b66 > ff02::1:ffa8:51bb: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:b000:3d4:d156:9f78:d2a8:51bb
  source link-address option (1), length 8 (1): xx:xx:xx:xx:3b:66
    0x0000:  001c 7388 3b66
14:54:28.183078 xx:xx:xx:xx:3b:66 > xx:xx:xx:xx:51:bb, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::21c:73ff:fe88:3b66 > ff02::1:ffa8:51bb: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:b000:3d4:d156:9f78:d2a8:51bb
  source link-address option (1), length 8 (1): xx:xx:xx:xx:3b:66
    0x0000:  001c 7388 3b66


GW (2001:db8:b000::1) has route to this host:

Code Select Expand

f1b-core01#show ipv6 route 2001:db8:b000:3d4:d156:9f78:d2a8:51bb

VRF: default
Routing entry for 2001:db8:b000:3d4:d156:9f78:d2a8:51bb
Codes: C - connected, S - static, K - kernel, O3 - OSPFv3, B - BGP, R - RIP, A B - BGP Aggregate, I L1 - IS-IS level 1, I L2 - IS-IS level 2, DH - DHCP, NG - Nexthop Group Static Route, M - Martian, DP - Dynamic Policy Route, L - VRF Leaked

 C        2001:db8:b000::/48 [0/1]
           via Vlan52, directly connected

That GW sends "who has":
tcpdump from that GW:

Code Select Expand

f1b-core01#tcpdump interface vlan 52 verbose filter host 2001:db8:b000::1 and not host 2001:db8:b000::11
tcpdump: listening on vlan52, link-type EN10MB (Ethernet), capture size 262144 bytes

14:54:27.170987 xx:xx:xx:xx:3b:66 > xx:xx:xx:xx:51:bb, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::21c:73ff:fe88:3b66 > ff02::1:ffa8:51bb: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:b000:3d4:d156:9f78:d2a8:51bb
  source link-address option (1), length 8 (1): xx:xx:xx:xx:3b:66
    0x0000:  001c 7388 3b66
14:54:28.183078 xx:xx:xx:xx:3b:66 > xx:xx:xx:xx:51:bb, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::21c:73ff:fe88:3b66 > ff02::1:ffa8:51bb: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:b000:3d4:d156:9f78:d2a8:51bb
  source link-address option (1), length 8 (1): xx:xx:xx:xx:3b:66
    0x0000:  001c 7388 3b66


I can see it on lab1-wan interface:

Code Select Expand

root@lab1:~ # tcpdump -ni lagg0_vlan52 host 2001:db8:b000::1
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on lagg0_vlan52, link-type EN10MB (Ethernet), snapshot length 262144 bytes

14:22:32.860398 IP6 2001:db8:b000::1 > ff02::1:ffa8:51bb: ICMP6, neighbor solicitation, who has 2001:db8:b000:3d4:d156:9f78:d2a8:51bb, length 32
14:22:33.890708 IP6 2001:db8:b000::1 > ff02::1:ffa8:51bb: ICMP6, neighbor solicitation, who has 2001:db8:b000:3d4:d156:9f78:d2a8:51bb, length 32
14:22:34.914744 IP6 2001:db8:b000::1 > ff02::1:ffa8:51bb: ICMP6, neighbor solicitation, who has 2001:db8:b000:3d4:d156:9f78:d2a8:51bb, length 32

And nothing else happens...

Do you have any idea what might be wrong?
How can I ensure that client computers in the LAN network receive a /64 instead of a /128?
Why doesn't lab1 (OPNsense) respond to "neighbor solicitation," even though it clearly knows this host is in its LAN network?

Any information, suggestions, or feedback is welcome—even if it's not entirely accurate, it might still help or point me in the right direction to solve the problem.

Regards
Borys

One of my OPNsense VM on OpenStack was freezing as well - kernel panic. Check the logs to see if it's not happening to you.
Paste output from

Code Select Expand

uname -a. Reinstalling the kernel and syslog-ng helped me.

Hi,

I have my own IPv6 prefix from IANA, let's say 2001:db8::/32.

I've delegated the prefix 2001:db8:b000::/40 for my lab.

So far, through trial and error, I've managed to create a working IPv6 "like IPv4" network configuration, but I'm aware that's not the way it should be done.

Here's what I did to achieve a theoretically working IPv6 configuration:
2 machines lab1 and lab2 (HA).

On the WAN interface:

• Assigned static IPv6 addresses:
  
  Code Select Expand
  2001:db8:b000::11/64 on wan-lab1
2001:db8:b000::12/64 on wan-lab2

• Configured the IPv6 gateway:
  
  Code Select Expand
  2001:db8:b000::1

• Configured a VIP (WAN Interface CARP):
  
  Code Select Expand
  2001:db8:b000::10/64


On the LAN interface:

• I assigned static IPv6 addresses:
  
  Code Select Expand
  2001:db8:b003::2/64 on lan-lab1
2001:db8:b003::3/64 on lan-lab2

• Configured a VIP (LAN Interface CARP):
  
  Code Select Expand
  2001:db8:b003::1/64


In ISC DHCPv6 LAN Interface, I have:

• Enable:
  Code Select Expand
  True
• Subnet:
  Code Select Expand
  2001:db8:b003::
• Subnet mask:
  Code Select Expand
  /64
• Available range:
  Code Select Expand
  2001:db8:b003:: - 2001:db8:b003:0:ffff:ffff:ffff:ffff
• Range:
  From:
  Code Select Expand
  2001:db8:b003:: To:
  Code Select Expand
  2001:db8:b003:0:ffff:ffff:ffff:ffff

In Router Advertisements for Lan:

• Router Advertisements:
  Code Select Expand
  Unmanaged
• Router Priority:
  Code Select Expand
  High
• Source Address:
  Code Select Expand
  Automatic
• Advertise Default Gateway:
  Code Select Expand
  True

Firewall NAT Outbound Rule:

Code Select Expand

Interface: WAN
Source: LAN_net
Destination: !LAN_net
NAT Address: 2001:db8:b000::10

The above configuration allows a client connected to the LAN network to request for IPv6 address (IPv4 as well, but I'm skipping that part). Client gets two specific IPv6 addresses from DHCPv6: one of the "global temporary dynamic" type and the other of the "global dynamic mngtmpaddr" type.

This client has internet access over IPv6, but its traffic is NAT-ed to the WAN VIP CARP IP, which is not how it should work in the IPv6 world (at least that's what I think).

How should I properly configure the WAN/LAN interfaces and DHCPv6 so that LAN client hosts derive the higher 64 bits from the OPN DHCPv6 and generate the lower 64 bits themselves (because, as far as I know, this is how it should work "properly")?

I need your support to correctly configure WAN IPv6 (static), LAN IPv6 (static?), and DHCPv6 for the LAN network. Unfortunately, I lack experience in this area, and the documentation seems rather sparse, especially regarding owning an IPv6 prefix and configuring interfaces statically.

Any suggestion on this topic is welcome.

Regards
Borys

@peterwkc You should have something similar to this:

Code Select Expand

<45>1 2025-01-10T02:33:17+01:00 opnsense2 syslog-ng 28239 - [meta sequenceId="1"] syslog-ng starting up; version='4.8.1'
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="2"] Fatal trap 12: page fault while in kernel mode
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="3"] cpuid = 3; apic id = 03
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="4"] fault virtual address     = 0x0
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="5"] fault code                = supervisor write data, page not present
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="6"] instruction pointer       = 0x20:0xffffffff80f3c00f
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="7"] stack pointer             = 0x28:0xfffffe000edf1d10
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="8"] frame pointer             = 0x28:0xfffffe000edf1d50
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="9"] code segment              = base 0x0, limit 0xfffff, type 0x1b
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="10"]                  = DPL 0, pres 1, long 1, def32 0, gran 1
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="11"] processor eflags = interrupt enabled, resume, IOPL = 0
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="12"] current process          = 0 (thread taskq)
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="13"] rdi: fffffe008ea60400 rsi: 0000000000000000 rdx: 000000000000002e
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="14"] rcx: 0000000000000000  r8: 0000000000000000  r9: fffff80005c2f480
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="15"] rax: 0000000000000000 rbx: 0000000000000000 rbp: fffffe000edf1d50
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="16"] r10: fffff80005c2f480 r11: 00000000802e6e20 r12: fffff801c6694fe0
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="17"] r13: fffffe008ea60400 r14: fffff801c6694318 r15: fffff80005c2f540
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="18"] trap number              = 12
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="19"] panic: page fault
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="20"] cpuid = 3
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="21"] time = 1736472729
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="22"] KDB: stack backtrace:
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="23"] db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe000edf1a00
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="24"] vpanic() at vpanic+0x131/frame 0xfffffe000edf1b30
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="25"] panic() at panic+0x43/frame 0xfffffe000edf1b90
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="26"] trap_fatal() at trap_fatal+0x40b/frame 0xfffffe000edf1bf0
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="27"] trap_pfault() at trap_pfault+0x46/frame 0xfffffe000edf1c40
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="28"] calltrap() at calltrap+0x8/frame 0xfffffe000edf1c40
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="29"] --- trap 0xc, rip = 0xffffffff80f3c00f, rsp = 0xfffffe000edf1d10, rbp = 0xfffffe000edf1d50 ---
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="30"] zone_release() at zone_release+0x1df/frame 0xfffffe000edf1d50
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="31"] bucket_drain() at bucket_drain+0xb9/frame 0xfffffe000edf1d80
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="32"] bucket_cache_reclaim_domain() at bucket_cache_reclaim_domain+0x2ff/frame 0xfffffe000edf1de0
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="33"] zone_timeout() at zone_timeout+0x2eb/frame 0xfffffe000edf1e20
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="34"] uma_timeout() at uma_timeout+0x58/frame 0xfffffe000edf1e40
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="35"] taskqueue_run_locked() at taskqueue_run_locked+0x182/frame 0xfffffe000edf1ec0
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="36"] taskqueue_thread_loop() at taskqueue_thread_loop+0xc2/frame 0xfffffe000edf1ef0
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="37"] fork_exit() at fork_exit+0x7f/frame 0xfffffe000edf1f30
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="38"] fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000edf1f30
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="39"] --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="40"] KDB: enter: panic
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="41"] ---<<BOOT>>---

Copy it with to lines before "Fatal trap 12:..." and paste it here as a "code".

Now I'm waiting and seeing if the kernel panic happens again.

So far so good, no crash since Friday afternoon.

Check if your reboots are not related to kernel panic's, there have been several threads on this topic recently.

@Borys - your log looks the same as mine did. Do what I did a couple posts up and I would imagine you will be good to go. Check the kernel version now that you've updated and then

Code Select Expand

pkg install -f syslog-ng

I've already done that. Now I'm waiting and seeing if the kernel panic happens again.

@DocHodges can You show output from:

Code Select Expand

uname -a
Look at this thread: [SOLVED] Kernel Panic - box restarts every few hours

And @dedi #4 post

Regards
Borys

I'm experiencing exactly the same issue:

Code Select Expand

<45>1 2025-01-10T02:33:17+01:00 opnsense2 syslog-ng 28239 - [meta sequenceId="1"] syslog-ng starting up; version='4.8.1'
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="2"] Fatal trap 12: page fault while in kernel mode
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="3"] cpuid = 3; apic id = 03
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="4"] fault virtual address     = 0x0
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="5"] fault code                = supervisor write data, page not present
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="6"] instruction pointer       = 0x20:0xffffffff80f3c00f
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="7"] stack pointer             = 0x28:0xfffffe000edf1d10
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="8"] frame pointer             = 0x28:0xfffffe000edf1d50
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="9"] code segment              = base 0x0, limit 0xfffff, type 0x1b
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="10"]                  = DPL 0, pres 1, long 1, def32 0, gran 1
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="11"] processor eflags = interrupt enabled, resume, IOPL = 0
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="12"] current process          = 0 (thread taskq)
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="13"] rdi: fffffe008ea60400 rsi: 0000000000000000 rdx: 000000000000002e
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="14"] rcx: 0000000000000000  r8: 0000000000000000  r9: fffff80005c2f480
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="15"] rax: 0000000000000000 rbx: 0000000000000000 rbp: fffffe000edf1d50
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="16"] r10: fffff80005c2f480 r11: 00000000802e6e20 r12: fffff801c6694fe0
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="17"] r13: fffffe008ea60400 r14: fffff801c6694318 r15: fffff80005c2f540
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="18"] trap number              = 12
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="19"] panic: page fault
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="20"] cpuid = 3
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="21"] time = 1736472729
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="22"] KDB: stack backtrace:
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="23"] db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe000edf1a00
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="24"] vpanic() at vpanic+0x131/frame 0xfffffe000edf1b30
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="25"] panic() at panic+0x43/frame 0xfffffe000edf1b90
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="26"] trap_fatal() at trap_fatal+0x40b/frame 0xfffffe000edf1bf0
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="27"] trap_pfault() at trap_pfault+0x46/frame 0xfffffe000edf1c40
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="28"] calltrap() at calltrap+0x8/frame 0xfffffe000edf1c40
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="29"] --- trap 0xc, rip = 0xffffffff80f3c00f, rsp = 0xfffffe000edf1d10, rbp = 0xfffffe000edf1d50 ---
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="30"] zone_release() at zone_release+0x1df/frame 0xfffffe000edf1d50
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="31"] bucket_drain() at bucket_drain+0xb9/frame 0xfffffe000edf1d80
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="32"] bucket_cache_reclaim_domain() at bucket_cache_reclaim_domain+0x2ff/frame 0xfffffe000edf1de0
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="33"] zone_timeout() at zone_timeout+0x2eb/frame 0xfffffe000edf1e20
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="34"] uma_timeout() at uma_timeout+0x58/frame 0xfffffe000edf1e40
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="35"] taskqueue_run_locked() at taskqueue_run_locked+0x182/frame 0xfffffe000edf1ec0
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="36"] taskqueue_thread_loop() at taskqueue_thread_loop+0xc2/frame 0xfffffe000edf1ef0
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="37"] fork_exit() at fork_exit+0x7f/frame 0xfffffe000edf1f30
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="38"] fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000edf1f30
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="39"] --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="40"] KDB: enter: panic
<13>1 2025-01-10T02:33:17+01:00 opnsense2 kernel - - [meta sequenceId="41"] ---<<BOOT>>---
I just performed a manual kernel update (unfortunately, I didn't check what the previous version was :/), and we'll see if the situation improves for me as well.

Regards
Borys

Hi,

I also struggle with kernel panic on a backup machine in a cluster running as virtual machines on opnestack. In my case, the problems started after updating to 24.7.10, as far as I remember.

Code Select Expand

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address   = 0x0
fault code              = supervisor write data, page not present
instruction pointer     = 0x20:0xffffffff80f3c00f
stack pointer           = 0x28:0xfffffe000edf1d10
frame pointer           = 0x28:0xfffffe000edf1d50
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 0 (thread taskq)
rdi: fffffe008e859400 rsi: 0000000000000000 rdx: 000000000000002e
rcx: 0000000000000000  r8: 0000000000000000  r9: fffff80005bbe480
rax: 0000000000000000 rbx: 0000000000000000 rbp: fffffe000edf1d50
r10: fffff80005bbe480 r11: 00000000800a7d8e r12: fffff80156d6cfe0
r13: fffffe008e859400 r14: fffff80156d6ccb8 r15: fffff80005bbe540
trap number             = 12
panic: page fault
cpuid = 2
time = 1736260351
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe000edf1a00
vpanic() at vpanic+0x131/frame 0xfffffe000edf1b30
panic() at panic+0x43/frame 0xfffffe000edf1b90
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe000edf1bf0
trap_pfault() at trap_pfault+0x46/frame 0xfffffe000edf1c40
calltrap() at calltrap+0x8/frame 0xfffffe000edf1c40
--- trap 0xc, rip = 0xffffffff80f3c00f, rsp = 0xfffffe000edf1d10, rbp = 0xfffffe000edf1d50 ---
zone_release() at zone_release+0x1df/frame 0xfffffe000edf1d50
bucket_drain() at bucket_drain+0xb9/frame 0xfffffe000edf1d80
bucket_cache_reclaim_domain() at bucket_cache_reclaim_domain+0x2ff/frame 0xfffffe000edf1de0
zone_timeout() at zone_timeout+0x2eb/frame 0xfffffe000edf1e20
uma_timeout() at uma_timeout+0x58/frame 0xfffffe000edf1e40
taskqueue_run_locked() at taskqueue_run_locked+0x182/frame 0xfffffe000edf1ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xc2/frame 0xfffffe000edf1ef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe000edf1f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000edf1f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic


Code Select Expand

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address   = 0x0
fault code              = supervisor write data, page not present
instruction pointer     = 0x20:0xffffffff82785e61
stack pointer           = 0x28:0xfffffe0084263a40
frame pointer           = 0x28:0xfffffe0084263a70
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 12 (irq28: virtio_pci2)
rdi: fffff801245bc210 rsi: fffff801245bc210 rdx: 000000002ce7b27e
rcx: 0000000000000000  r8: 000000004150a7d2  r9: 0000000020510000
rax: 0000000000000000 rbx: fffff80018710b00 rbp: fffffe0084263a70
r10: 000000002c28d619 r11: 0000000000000301 r12: fffffe008ea5c000
r13: 000000000005625c r14: fffff801245bc210 r15: fffff80003aea000
trap number             = 12
panic: page fault
cpuid = 1
time = 1736353882
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0084263730
vpanic() at vpanic+0x131/frame 0xfffffe0084263860
panic() at panic+0x43/frame 0xfffffe00842638c0
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe0084263920
trap_pfault() at trap_pfault+0x46/frame 0xfffffe0084263970
calltrap() at calltrap+0x8/frame 0xfffffe0084263970
--- trap 0xc, rip = 0xffffffff82785e61, rsp = 0xfffffe0084263a40, rbp = 0xfffffe0084263a70 ---
pf_detach_state() at pf_detach_state+0x6c1/frame 0xfffffe0084263a70
pf_unlink_state() at pf_unlink_state+0x290/frame 0xfffffe0084263ab0
pfsync_in_del_c() at pfsync_in_del_c+0x6c/frame 0xfffffe0084263af0
pfsync_input() at pfsync_input+0x23a/frame 0xfffffe0084263b70
ip_input() at ip_input+0x268/frame 0xfffffe0084263bd0
netisr_dispatch_src() at netisr_dispatch_src+0x9e/frame 0xfffffe0084263c20
ether_demux() at ether_demux+0x149/frame 0xfffffe0084263c50
ether_nh_input() at ether_nh_input+0x36a/frame 0xfffffe0084263cb0
netisr_dispatch_src() at netisr_dispatch_src+0x9e/frame 0xfffffe0084263d00
ether_input() at ether_input+0x56/frame 0xfffffe0084263d50
vtnet_rxq_eof() at vtnet_rxq_eof+0x6e9/frame 0xfffffe0084263e20
vtnet_rx_vq_process() at vtnet_rx_vq_process+0xbc/frame 0xfffffe0084263e60
ithread_loop() at ithread_loop+0x257/frame 0xfffffe0084263ef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe0084263f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0084263f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic

The strange thing is that the master is in exactly the same version and there are no problems with it. Of course, it is running in a different location on a different compute node (but with the same parameters for both virtual machines and compute nodes).

Regards,
Borys

Looks good.

Would you share final interfaces configuration on Opnsense (ifconfig) as well as Driver used on the NIcs?

Thanks & regards

Yes Sure.

Code Select Expand

root@admins-opn-1:~ # ifconfig
ixl0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9214
    options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
    ether xx:xx:xx:xx:11:b4
    media: Ethernet autoselect (25GBase-LR <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
ixl1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9214
    options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
    ether xx:xx:xx:xx:11:b4
    hwaddr xx:xx:xx:xx:11:b5
    media: Ethernet autoselect (25GBase-CR <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
igb0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
    ether xx:xx:xx:xx:22:24
    media: Ethernet autoselect
    status: no carrier
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
igb1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
    ether xx:xx:xx:xx:22:25
    media: Ethernet autoselect
    status: no carrier
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0 metric 0 mtu 1536
    options=0
    groups: enc
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
pfsync0: flags=1000041<UP,RUNNING,LOWER_UP> metric 0 mtu 1500
    options=0
    syncdev: lagg0_vlan2 syncpeer: 192.168.2.249 maxupd: 128 defer: off version: 1400
    syncok: 1
    groups: pfsync
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33152
    options=0
    groups: pflog
lagg0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9214
    options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
    ether xx:xx:xx:xx:11:b4
    hwaddr 00:00:00:00:00:00
    inet6 xx:xx:xx:xx%lagg0 prefixlen 64 scopeid 0x9
    laggproto lacp lagghash l2,l3
    laggport: ixl0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: ixl1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lagg0_vlanXX: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: lagg0_vlanXX (opt2)
    options=4000000<MEXTPG>
    ether xx:xx:xx:xx:11:b4
    inet x.x.x.x netmask 0xfffffff0 broadcast x.x.x.x
    inet6 xx:xx:xx:xx%lagg0_vlanXX prefixlen 64 scopeid 0xa
    groups: vlan
    vlan: XX vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
    media: Ethernet autoselect
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0_vlan2: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: lagg0_vlan2 (opt1)
    options=4000000<MEXTPG>
    ether xx:xx:xx:xx:11:b4
    inet 192.168.2.241 netmask 0xffffff00 broadcast 192.168.2.255
    inet6 xx:xx:xx:xx%lagg0_vlan2 prefixlen 64 scopeid 0xb
    groups: vlan
    vlan: 2 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
    media: Ethernet autoselect
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0_vlan3: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 8800
    description: lagg0_vlan3 (opt6)
    options=4000000<MEXTPG>
    ether xx:xx:xx:xx:11:b4
    inet 10.255.5.2 netmask 0xffffff00 broadcast 10.255.5.255
    inet 10.255.5.1 netmask 0xffffff00 broadcast 10.255.5.255 vhid 91
    inet6 xx:xx:xx:xx%lagg0_vlan3 prefixlen 64 scopeid 0xc
    inet6 xx.xx.xx.xx::2 prefixlen 64
    inet6 xx.xx.xx.xx::1 prefixlen 64 vhid 92
    groups: vlan
    carp: MASTER vhid 91 advbase 1 advskew 0
          peer 224.0.0.18 peer6 ff02::12
    carp: MASTER vhid 92 advbase 1 advskew 0
          peer 224.0.0.18 peer6 ff02::12
    vlan: 3 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
    media: Ethernet autoselect
    status: active
    nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
lagg0_vlan4: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 8800
    options=4000000<MEXTPG>
    ether xx:xx:xx:xx:11:b4
    inet6 xx:xx:xx:xx%lagg0_vlan4 prefixlen 64 scopeid 0xd
    groups: vlan
    vlan: 4 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
    media: Ethernet autoselect
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lagg0_vlan52: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 8800
    description: lagg0_vlan52 (wan)
    options=4000000<MEXTPG>
    ether xx:xx:xx:xx:11:b4
    inet x.x.x.201 netmask 0xffffffe0 broadcast x.x.x.223
    inet x.x.x.200 netmask 0xffffffe0 broadcast x.x.x.223 vhid 98
    inet6 xx:xx:xx:xx%lagg0_vlan52 prefixlen 64 scopeid 0xf
    inet6 xxxx:yyyy:zzzz::11 prefixlen 64
    inet6 xxxx:yyyy:zzzz::10 prefixlen 64 vhid 99
    groups: vlan
    carp: MASTER vhid 98 advbase 1 advskew 0
          peer 224.0.0.18 peer6 ff02::12
    carp: MASTER vhid 99 advbase 1 advskew 0
          peer 224.0.0.18 peer6 ff02::12
    vlan: 52 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
    media: Ethernet autoselect
    status: active
    nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
root@admins-opn-1:~ #


Code Select Expand

FW: 9.30 default OPNsense ixl driver


Code Select Expand

root@admins-opn-1:~ # sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'
dev.ixl.1.%desc: Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k
dev.ixl.0.%desc: Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k


More tuning&testing

Code Select Expand

hw.ibrs_disable=1
kern.ipc.maxsockbuf=614400000
net.isr.maxthreads=-1
net.isr.bindthreads=1
net.inet.rss.enabled=1
net.inet.tcp.soreceive_stream=1
net.inet.tcp.delayed_ack=1
net.inet.tcp.blackhole=0
net.inet.udp.blackhole=0


Code Select Expand

root@admins-opn-1:~ # iperf3 -c x.x.x.2
Connecting to host x.x.x.2, port 5201
[  5] local x.x.x.1 port 20279 connected to x.x.x.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.20 GBytes  18.9 Gbits/sec    0   2.01 MBytes       
[  5]   1.00-2.03   sec  2.30 GBytes  19.2 Gbits/sec    0   2.01 MBytes       
[  5]   2.03-3.03   sec  2.26 GBytes  19.3 Gbits/sec    0   2.01 MBytes       
[  5]   3.03-4.01   sec  2.22 GBytes  19.5 Gbits/sec    0   2.01 MBytes       
[  5]   4.01-5.00   sec  2.22 GBytes  19.2 Gbits/sec    0   2.01 MBytes       
[  5]   5.00-6.00   sec  2.22 GBytes  19.1 Gbits/sec    0   2.01 MBytes       
[  5]   6.00-7.00   sec  2.24 GBytes  19.2 Gbits/sec    0   2.01 MBytes       
[  5]   7.00-8.00   sec  2.24 GBytes  19.3 Gbits/sec    0   2.01 MBytes       
[  5]   8.00-9.01   sec  2.27 GBytes  19.4 Gbits/sec    0   2.01 MBytes       
[  5]   9.01-10.00  sec  2.25 GBytes  19.5 Gbits/sec    0   2.01 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  22.4 GBytes  19.3 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  22.4 GBytes  19.3 Gbits/sec                  receiver

iperf Done.


Code Select Expand

root@admins-opn-1:~ # iperf3 -c x.x.x.2 -P4
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.24 GBytes  6.22 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  7.24 GBytes  6.21 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  6.68 GBytes  5.74 Gbits/sec    0             sender
[  7]   0.00-10.00  sec  6.68 GBytes  5.73 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  7.24 GBytes  6.22 Gbits/sec    0             sender
[  9]   0.00-10.00  sec  7.24 GBytes  6.22 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  7.24 GBytes  6.22 Gbits/sec    0             sender
[ 11]   0.00-10.00  sec  7.24 GBytes  6.21 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  28.4 GBytes  24.4 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  28.4 GBytes  24.4 Gbits/sec                  receiver


Code Select Expand

root@admins-opn-1:~ # iperf3 -c x.x.x.2 -P8
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.82 GBytes  1.56 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  1.81 GBytes  1.56 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  1.81 GBytes  1.55 Gbits/sec    0             sender
[  7]   0.00-10.00  sec  1.81 GBytes  1.55 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  3.61 GBytes  3.10 Gbits/sec    0             sender
[  9]   0.00-10.00  sec  3.61 GBytes  3.10 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  7.24 GBytes  6.22 Gbits/sec    0             sender
[ 11]   0.00-10.00  sec  7.24 GBytes  6.21 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  6.82 GBytes  5.85 Gbits/sec    0             sender
[ 13]   0.00-10.00  sec  6.82 GBytes  5.85 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  1.82 GBytes  1.56 Gbits/sec    0             sender
[ 15]   0.00-10.00  sec  1.81 GBytes  1.56 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  3.63 GBytes  3.12 Gbits/sec    0             sender
[ 17]   0.00-10.00  sec  3.63 GBytes  3.12 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  1.81 GBytes  1.56 Gbits/sec    0             sender
[ 19]   0.00-10.00  sec  1.81 GBytes  1.56 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  28.6 GBytes  24.5 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  28.5 GBytes  24.5 Gbits/sec                  receiver

iperf Done.


So called jumbo frames were a bad idea to begin with.

Yep.

And what did you change?

I'm ashamed to admit it, but it seems that the main problem was the MTU settings. Currently, I achieved these results on a completely clean installation (couple of firewall rules).

On the switch side, I had the MTU set "fixed" to 9214 by my network team and I set the same on the OPNs side. Unfortunately, the effect of this was as you saw earlier. After setting the MTU to 8800 on the OPNsense side, everything started working except for pfSync (on unicast). This connection started working when I set the MTU to 1500 on the interface dedicated to HA and synchronization.

I still want to do some fine tuning and see if I can achieve even better results. When I have some free time, I will try to test the same settings for Broadcom cards as well.