"Recent posts
#1
General Discussion / Re: What is the Purpose of 'op...
Last post by Al Muckart - Today at 07:57:56 AMRight, that makes perfect sense, thank you.
#2
26.1, 26,4 Series / Re: Business Edition pf CVE-20...
Last post by wirehire - Today at 07:28:29 AMthanks franco and the team. we are now updated all our firewall to 26.4 Business!
Work like a charme! very good work, thanks for your software !
Work like a charme! very good work, thanks for your software !
#3
Hardware and Performance / Re: TOPTON Mini PC Running OPN...
Last post by BrandyWine - Today at 05:21:52 AMVerify uCode is loading in.
Verify the PSU is capable in both volts & amps under rated load.
Verify the PSU is capable in both volts & amps under rated load.
#4
Hardware and Performance / Re: cpu-microcode-intel: no ma...
Last post by BrandyWine - Today at 05:11:57 AMSomething perhaps relevant on org site.
https://forums.freebsd.org/threads/cpu-microcode-not-updated-during-boot.95641/
Having proper uCode is interesting, for some cpu, for other reasons too.
https://forum.opnsense.org/index.php?topic=48343.0
https://lists.freebsd.org/archives/freebsd-current/2025-January/006984.html
But at this writing it's unclear to me where the issues and fixes are at.
https://forums.freebsd.org/threads/cpu-microcode-not-updated-during-boot.95641/
Having proper uCode is interesting, for some cpu, for other reasons too.
https://forum.opnsense.org/index.php?topic=48343.0
https://lists.freebsd.org/archives/freebsd-current/2025-January/006984.html
But at this writing it's unclear to me where the issues and fixes are at.
#5
General Discussion / Re: losing internet connection...
Last post by stanps - Today at 05:02:04 AMAre you sure you're losing internet access, and it's not just DNS failing?
I ask because I noticed that after 5 to 8 days, and it ended up being Unbound needing to be restarted. Which btw, didn't work properly once Unbound was in a bad state. So I set up a cron job to restart Unbound every 5 days or so.
Just a thought....hope it helps.
-S
I ask because I noticed that after 5 to 8 days, and it ended up being Unbound needing to be restarted. Which btw, didn't work properly once Unbound was in a bad state. So I set up a cron job to restart Unbound every 5 days or so.
Just a thought....hope it helps.
-S
#6
26.1, 26,4 Series / Moved from ISC-DHCPv4 to KeaDH...
Last post by SpikeyGG - Today at 04:03:02 AMWith the release of OPNsense 26 and announcement of deprecating ISC-DHCP (which I was using) I decided to move to Kea DHCP. I went through the complete migration and everything was working great for a few weeks. Then I needed to do some maintenance in my wiring closet so I took the whole network down, performed the maintenance, and brought everything back up. All of it seemed to be working but then I realized that all my DNS records for dynamic DHCP addresses were pointing at the wrong host!
I'm using Kea DHCP with Unbound DNS. Previously, with ISC-DHCP taking the network down and bringing it back online would not cause any issues with static nor dynamic DHCP hosts. I was hoping that eventually the bad records would "work themselves out" but it has been over a month and they're still wrong. I can see the correct dynamic leases in Kea's "Leases DHCPv4" list but nslookups for the name always give a different IP. In fact, I even turned on the Kea DDNS Agent at port 53001 and when querying that service, it too has the wrong IP!
I tried a couple of times already to flush the Unbound cache and force it to rebuild but somehow it always finds/restores (or never removes) the bad records. I also have mDNS deployed and it seems to always be working great -- I'm not sure why but I also don't know how to use nslookup to query it (if that is even possible).
Here's an example the device "media-equipment" which is actually at 172.27.80.29 (see Lease screenshot), however, the .mynet address shows 172.27.80.19:
Here is an nslookup issued to both port 53 (Unbound DNS) and port 53001 (Kea DDNS Agent) showing that they both report the wrong address:
I had tried manually setting the Kea DHCP options for:
Anyone have suggestions of how to get my Unbound DNS (or even Kea DDNS Agent) fixed without resorting to use static IPs for all my dynamic DHCP devices?
I'm using Kea DHCP with Unbound DNS. Previously, with ISC-DHCP taking the network down and bringing it back online would not cause any issues with static nor dynamic DHCP hosts. I was hoping that eventually the bad records would "work themselves out" but it has been over a month and they're still wrong. I can see the correct dynamic leases in Kea's "Leases DHCPv4" list but nslookups for the name always give a different IP. In fact, I even turned on the Kea DDNS Agent at port 53001 and when querying that service, it too has the wrong IP!
I tried a couple of times already to flush the Unbound cache and force it to rebuild but somehow it always finds/restores (or never removes) the bad records. I also have mDNS deployed and it seems to always be working great -- I'm not sure why but I also don't know how to use nslookup to query it (if that is even possible).
Here's an example the device "media-equipment" which is actually at 172.27.80.29 (see Lease screenshot), however, the .mynet address shows 172.27.80.19:
Code Select
Microsoft Windows [Version 10.0.19045.7184]
(c) Microsoft Corporation. All rights reserved.
C:\Users\SpikeyGG>ping media-equipment.mynet
Pinging media-equipment.mynet [172.27.80.19] with 32 bytes of data:
Reply from 172.27.80.19: bytes=32 time=2ms TTL=254
Reply from 172.27.80.19: bytes=32 time=1ms TTL=254
Ping statistics for 172.27.80.19:
Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 1ms, Maximum = 2ms, Average = 1ms
Control-C
^C
C:\Users\SpikeyGG>ping media-equipment.local
Pinging media-equipment.local [172.27.80.29] with 32 bytes of data:
Reply from 172.27.80.29: bytes=32 time=2ms TTL=254
Reply from 172.27.80.29: bytes=32 time=2ms TTL=254
Ping statistics for 172.27.80.29:
Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 2ms, Maximum = 2ms, Average = 2ms
Control-C
^C
C:\Users\SpikeyGG>Here is an nslookup issued to both port 53 (Unbound DNS) and port 53001 (Kea DDNS Agent) showing that they both report the wrong address:
Code Select
C:\Users\SpikeyGG>nslookup -port=53001 media-equipment.mynet 172.27.20.1
Server: UnKnown
Address: 172.27.20.1
Name: media-equipment.mynet
Address: 172.27.80.19
C:\Users\SpikeyGG>nslookup -port=53 media-equipment.mynet 172.27.20.1
Server: UnKnown
Address: 172.27.20.1
Name: media-equipment.mynet
Address: 172.27.80.19
C:\Users\SpikeyGG>
I had tried manually setting the Kea DHCP options for:
- ddns-send-updates
- ddns-override-no-update
- ddns-override-client-update
Anyone have suggestions of how to get my Unbound DNS (or even Kea DDNS Agent) fixed without resorting to use static IPs for all my dynamic DHCP devices?
#7
General Discussion / NAT64 Dropping Return Traffic:...
Last post by meangarp - Today at 03:28:03 AMI have exhausted all my troubleshooting, someone please tell me why I'm stupid.
I have DNS64 setup correctly, clients are resolving the virtual IPv6 address, however they have been timing out and falling back to ipv4 (when they have an IPv4 address).
Tayga config:
IPv4 Address 192.168.255.1
IPv4 NAT64 Interface Address 172.20.0.1
IPv6 Address _blank_
IPv6 NAT64 Interface Address fd01::a:172:20:0:1
IPv6 Prefix 64:ff9b::/96
IPv4 Pool 192.168.255.0/24
"Only used for ICMP."
This phrase in the tayga config is doing *a lot* of heavy lifting. (It makes it seem like its essentially useless if you dont care about icmp, which for a pure netcat TCP test, I dont)
NAT Oubound rule:
Interface Source Source Port Destination Destination Port NAT Address NAT Port Static Port Description
WAN 192.168.255.0/24 * * * Interface address * NO NAT64 Tayga Outbound NAT
Tayga Interface rule (allow all):
Pass IN Tayga IPv4+IPv6 * * * * * *
Looks like I've hit all the points in the setup wiki https://docs.opnsense.org/manual/how-tos/tayga.html
And from my troubleshooting below, it seems like the outbound nat, firewall rule, and tayga itself are all operating properly.
I think I have it narrowed down to the internal IPv6 return traffic being dropped by the kernel.
My tcpdumps are showing:
WAN Interface
[root@EFW ~]# tcpdump -vvvniigb0 host 64:ff9b::44f:8ec8 or host 4.79.142.200
tcpdump: listening on igb0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:01:18.926683 IP (tos 0x0, ttl 88, id 25707, offset 0, flags [none], proto TCP (6), length 52)
publicIPv4.42260 > 4.79.142.200.443: Flags [ S ], cksum 0x35d7 (correct), seq 1104216988, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
18:01:18.960649 IP (tos 0x0, ttl 121, id 16200, offset 0, flags [DF], proto TCP (6), length 52)
4.79.142.200.443 > publicIPv4.42260: Flags [S.], cksum 0xf1e9 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:21.965648 IP (tos 0x0, ttl 121, id 17602, offset 0, flags [DF], proto TCP (6), length 52)
4.79.142.200.443 > publicIPv4.42260: Flags [S.], cksum 0xf1e9 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:27.971536 IP (tos 0x0, ttl 121, id 18655, offset 0, flags [DF], proto TCP (6), length 48)
4.79.142.200.443 > publicIPv4.42260: Flags [S.], cksum 0x25f9 (correct), seq 1881125064, ack 1104216989, win 65535, options [mss 1460,nop,nop,sackOK], length 0
18:01:39.967678 IP (tos 0x0, ttl 121, id 20842, offset 0, flags [DF], proto TCP (6), length 40)
4.79.142.200.443 > publicIPv4.42260: Flags [R], cksum 0x52c9 (correct), seq 1881125065, win 0, length 0
5 packets captured
4058 packets received by filter
0 packets dropped by kernel
Tayga Interface
[root@EFW ~]# tcpdump -vvvninat64 host 64:ff9b::44f:8ec8 or host 4.79.142.200
tcpdump: listening on nat64, link-type NULL (BSD loopback), snapshot length 262144 bytes
18:01:18.926619 IP6 (flowlabel 0x6a58e, hlim 90, next-header TCP (6) payload length: 32) fd01::1:172:20:20:10.39346 > 64:ff9b::44f:8ec8.443: Flags [ S ], cksum 0xff4d (correct), seq 1104216988, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
18:01:18.926638 IP (tos 0x0, ttl 89, id 25707, offset 0, flags [none], proto TCP (6), length 52)
192.168.255.195.39346 > 4.79.142.200.443: Flags [ S ], cksum 0x3da6 (correct), seq 1104216988, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
18:01:18.960684 IP (tos 0x0, ttl 120, id 16200, offset 0, flags [DF], proto TCP (6), length 52)
4.79.142.200.443 > 192.168.255.195.39346: Flags [S.], cksum 0xf9b8 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:18.960718 IP6 (hlim 119, next-header TCP (6) payload length: 32) 64:ff9b::44f:8ec8.443 > fd01::1:172:20:20:10.39346: Flags [S.], cksum 0xbb60 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:21.965711 IP (tos 0x0, ttl 120, id 17602, offset 0, flags [DF], proto TCP (6), length 52)
4.79.142.200.443 > 192.168.255.195.39346: Flags [S.], cksum 0xf9b8 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:21.965742 IP6 (hlim 119, next-header TCP (6) payload length: 32) 64:ff9b::44f:8ec8.443 > fd01::1:172:20:20:10.39346: Flags [S.], cksum 0xbb60 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:27.971559 IP (tos 0x0, ttl 120, id 18655, offset 0, flags [DF], proto TCP (6), length 48)
4.79.142.200.443 > 192.168.255.195.39346: Flags [S.], cksum 0x2dc8 (correct), seq 1881125064, ack 1104216989, win 65535, options [mss 1460,nop,nop,sackOK], length 0
18:01:27.971590 IP6 (hlim 119, next-header TCP (6) payload length: 28) 64:ff9b::44f:8ec8.443 > fd01::1:172:20:20:10.39346: Flags [S.], cksum 0xef6f (correct), seq 1881125064, ack 1104216989, win 65535, options [mss 1460,nop,nop,sackOK], length 0
18:01:39.967745 IP (tos 0x0, ttl 120, id 20842, offset 0, flags [DF], proto TCP (6), length 40)
4.79.142.200.443 > 192.168.255.195.39346: Flags [R], cksum 0x5a98 (correct), seq 1881125065, win 0, length 0
18:01:39.967789 IP6 (hlim 119, next-header TCP (6) payload length: 20) 64:ff9b::44f:8ec8.443 > fd01::1:172:20:20:10.39346: Flags [R], cksum 0x1c40 (correct), seq 1881125065, win 0, length 0
10 packets captured
10 packets received by filter
0 packets dropped by kernel
Internal Interface
[root@EFW ~]# tcpdump -vvvnivlan01 host 64:ff9b::44f:8ec8 or host 4.79.142.200
tcpdump: listening on vlan01, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:01:18.926584 IP6 (flowlabel 0x6a58e, hlim 91, next-header TCP (6) payload length: 32) fd01::1:172:20:20:10.39346 > 64:ff9b::44f:8ec8.443: Flags [ S ], cksum 0xff4d (correct), seq 1104216988, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
1 packet captured
2694 packets received by filter
0 packets dropped by kernel
I think this one counter `failures of source address selection` is the symptom, as it tends to increase ~60seconds after each test
[root@EFW ~]# netstat -s -p ip6
ip6:
34443359 total packets received
0 with size smaller than minimum
0 with data size < data length
0 with bad options
276 with incorrect version number
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 fragments that exceeded limit
0 atomic fragments
0 packets reassembled ok
1864797 packets for this host
31895889 packets forwarded
0 packets not forwardable
0 redirects sent
3043173 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
2 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
4 packets that violated scope rules
62 multicast packets which we don't join
Input histogram:
hop by hop: 699
TCP: 32210783
UDP: 1981356
ICMP6: 250233
PIM: 12
Mbuf statistics:
16835632 one mbuf
two or more mbuf:
lo0= 2124
wg1= 377865
17227738 one ext mbuf
0 two or more ext mbuf
0 packets whose headers are not contiguous
0 tunneling packets that can't find gif
0 packets discarded because of too many headers
2648 failures of source address selection
source addresses on an outgoing I/F
53783 link-locals
77070 globals
source addresses on a non-outgoing I/F
82 globals
2648 addresses scope=0xf
source addresses of same scope
53780 link-locals
77152 globals
source addresses of a different scope
3 link-locals
Source addresses selection rule applied:
130935 first candidate
15095 same address
53708 appropriate scope
48685 outgoing interface
82 matching label
42921 longest match
Fw info: (OPNsense 26.1.8_5-amd64)
routes:
[root@EFW ~]# netstat -rn
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default publicipv4gw UGS igb0
1.1.1.1 publicipv4gw UGHS igb0
10.10.0.0/16 172.20.0.2 UGS ixl2
publicipv4block/24 link#1 U igb0
publicipv4 link#14 UHS lo0
127.0.0.1 link#14 UH lo0
172.20.0.0/30 link#12 U ixl2
172.20.0.1 link#14 UHS lo0
172.20.19.0/29 link#5 U igb4
172.20.19.1 link#14 UHS lo0
172.20.20.0/24 link#18 U vlan01
172.20.20.1 link#14 UHS lo0
172.20.21.4/30 link#2 U igb1
172.20.21.5 link#14 UHS lo0
172.20.22.0/24 link#19 U vlan02
172.20.22.1 link#14 UHS lo0
172.20.24.0/24 link#20 U vlan03
172.20.24.1 link#14 UHS lo0
172.20.253.0/30 link#22 U wg1
172.20.253.1 link#14 UHS lo0
192.168.12.0/24 link#4 U igb3
192.168.12.1 link#14 UHS lo0
192.168.255.0/24 link#24 US nat64
192.168.255.1 link#24 UH nat64
Internet6:
Destination Gateway Flags Netif Expire
default fe80::256:2bff:fe76:b022%igb0 UG igb0
::1 link#14 UHS lo0
64:ff9b::/96 link#24 US nat64
publicipv6 link#14 UHS lo0
publicipv6prefix::/60 link#14 USB lo0
publicipv6 fe80::256:2bff:fe76:b022%igb0 UGHS igb0
fd01:0:0:1::/64 link#18 U vlan01
fd01::1:172:20:20:1 link#14 UHS lo0
fd01::1:172:20:20:10 link#18 UHS vlan01
fd01:0:0:2::/64 link#19 U vlan02
fd01::2:172:20:22:1 link#14 UHS lo0
fd01:0:0:3::/64 link#20 U vlan03
fd01::3:172:20:24:1 link#14 UHS lo0
fd01:0:0:4::/64 link#5 U igb4
fd01::4:172:20:19:1 link#14 UHS lo0
fd01:0:0:8::/64 fd01::a:172:20:0:0 UGS ixl2
fd01::a:10:10:0:0/126 fd01::a:172:20:0:0 UGS ixl2
fd01::a:172:20:0:0/127 link#12 U ixl2
fd01::a:172:20:0:1 link#14 UHS lo0
fd01::a:172:20:21:0 link#14 UHS lo0
fd01::a:172:20:21:0/127 link#13 U ixl3
fd01::a:172:20:21:4/127 link#2 U igb1
fd01::a:172:20:21:5 link#14 UHS lo0
fd01::a:172:20:25:0 link#14 UHS lo0
fd01::a:172:20:25:0/127 link#3 U igb2
fd01::a:172:20:253:2/127 link#22 U wg1
fd01::a:172:20:253:3 link#14 UHS lo0
fd01:0:0:f::/64 link#4 U igb3
fd01:0:0:f::1 link#14 UHS lo0
fe80::%igb0/64 link#1 U igb0
fe80::a236:9fff:fe89:60e7%lo0 link#14 UHS lo0
fe80::%igb1/64 link#2 U igb1
fe80::7ec2:55ff:fe2e:2c71%lo0 link#14 UHS lo0
fe80::%igb2/64 link#3 U igb2
fe80::7ec2:55ff:fe2e:2c72%lo0 link#14 UHS lo0
fe80::%igb3/64 link#4 U igb3
fe80::7ec2:55ff:fe2e:2c73%lo0 link#14 UHS lo0
fe80::%igb4/64 link#5 U igb4
fe80::7ec2:55ff:fe2e:2c74%lo0 link#14 UHS lo0
fe80::%igb6/64 link#7 U igb6
fe80::7ec2:55ff:fe2e:2c76%lo0 link#14 UHS lo0
fe80::%ixl2/64 link#12 U ixl2
fe80::7ec2:55ff:fe25:88%lo0 link#14 UHS lo0
fe80::%ixl3/64 link#13 U ixl3
fe80::7ec2:55ff:fe25:89%lo0 link#14 UHS lo0
fe80::%lo0/64 link#14 U lo0
fe80::1%lo0 link#14 UHS lo0
fe80::%vlan01/64 link#18 U vlan01
fe80::7ec2:55ff:fe25:89%lo0 link#14 UHS lo0
fe80::%vlan02/64 link#19 U vlan02
fe80::7ec2:55ff:fe25:89%lo0 link#14 UHS lo0
fe80::%vlan03/64 link#20 U vlan03
fe80::7ec2:55ff:fe25:89%lo0 link#14 UHS lo0
fe80::%vlan04/64 link#21 U vlan04
fe80::7ec2:55ff:fe2e:2c76%lo0 link#14 UHS lo0
Interfaces:
nat64: flags=1008051<UP,POINTOPOINT,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4080000<LINKSTATE,MEXTPG>
inet 172.20.0.1 --> 192.168.255.1 netmask 0xffffffff
groups: tun tayga
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Opened by PID 85915
drivername: tun0
vlan01: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: EdgeNet (opt7)
options=4000000<MEXTPG>
ether 7c:c2:55:25:00:89
inet 172.20.20.1 netmask 0xffffff00 broadcast 172.20.20.255
inet6 fd01::1:172:20:20:1 prefixlen 64
inet6 fe80::7ec2:55ff:fe25:89%vlan01 prefixlen 64 scopeid 0x12
groups: vlan
vlan: 120 vlanproto: 802.1q vlanpcp: 0 parent interface: ixl3
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active
nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
drivername: vlan0
ixl3: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: DMZSRV (opt2)
options=48500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,HWSTATS,MEXTPG>
ether 7c:c2:55:25:00:89
inet6 fd01::a:172:20:21:0 prefixlen 127
inet6 fe80::7ec2:55ff:fe25:89%ixl3 prefixlen 64 scopeid 0xd
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active
nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
drivername: ixl3
igb0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN (wan)
options=48520b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,HWSTATS,MEXTPG>
ether a0:36:9f:89:60:e7
hwaddr 7c:c2:55:2e:2c:70
inet publicipv4 netmask 0xffffff00 broadcast 255.255.255.255
inet6 fe80::a236:9fff:fe89:60e7%igb0 prefixlen 64 scopeid 0x1
inet6 publicipv6 prefixlen 128 pltime 86400 vltime 86400
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
drivername: igb0
Client test : (
$ nc -6 -w 1 -vz grc.com 443
Ncat: Version 7.95 ( https://nmap.org/ncat )
Ncat: TIMEOUT.
I have DNS64 setup correctly, clients are resolving the virtual IPv6 address, however they have been timing out and falling back to ipv4 (when they have an IPv4 address).
Tayga config:
IPv4 Address 192.168.255.1
IPv4 NAT64 Interface Address 172.20.0.1
IPv6 Address _blank_
IPv6 NAT64 Interface Address fd01::a:172:20:0:1
IPv6 Prefix 64:ff9b::/96
IPv4 Pool 192.168.255.0/24
"Only used for ICMP."
This phrase in the tayga config is doing *a lot* of heavy lifting. (It makes it seem like its essentially useless if you dont care about icmp, which for a pure netcat TCP test, I dont)
NAT Oubound rule:
Interface Source Source Port Destination Destination Port NAT Address NAT Port Static Port Description
WAN 192.168.255.0/24 * * * Interface address * NO NAT64 Tayga Outbound NAT
Tayga Interface rule (allow all):
Pass IN Tayga IPv4+IPv6 * * * * * *
Looks like I've hit all the points in the setup wiki https://docs.opnsense.org/manual/how-tos/tayga.html
And from my troubleshooting below, it seems like the outbound nat, firewall rule, and tayga itself are all operating properly.
I think I have it narrowed down to the internal IPv6 return traffic being dropped by the kernel.
My tcpdumps are showing:
WAN Interface
[root@EFW ~]# tcpdump -vvvniigb0 host 64:ff9b::44f:8ec8 or host 4.79.142.200
tcpdump: listening on igb0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:01:18.926683 IP (tos 0x0, ttl 88, id 25707, offset 0, flags [none], proto TCP (6), length 52)
publicIPv4.42260 > 4.79.142.200.443: Flags [ S ], cksum 0x35d7 (correct), seq 1104216988, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
18:01:18.960649 IP (tos 0x0, ttl 121, id 16200, offset 0, flags [DF], proto TCP (6), length 52)
4.79.142.200.443 > publicIPv4.42260: Flags [S.], cksum 0xf1e9 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:21.965648 IP (tos 0x0, ttl 121, id 17602, offset 0, flags [DF], proto TCP (6), length 52)
4.79.142.200.443 > publicIPv4.42260: Flags [S.], cksum 0xf1e9 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:27.971536 IP (tos 0x0, ttl 121, id 18655, offset 0, flags [DF], proto TCP (6), length 48)
4.79.142.200.443 > publicIPv4.42260: Flags [S.], cksum 0x25f9 (correct), seq 1881125064, ack 1104216989, win 65535, options [mss 1460,nop,nop,sackOK], length 0
18:01:39.967678 IP (tos 0x0, ttl 121, id 20842, offset 0, flags [DF], proto TCP (6), length 40)
4.79.142.200.443 > publicIPv4.42260: Flags [R], cksum 0x52c9 (correct), seq 1881125065, win 0, length 0
5 packets captured
4058 packets received by filter
0 packets dropped by kernel
Tayga Interface
[root@EFW ~]# tcpdump -vvvninat64 host 64:ff9b::44f:8ec8 or host 4.79.142.200
tcpdump: listening on nat64, link-type NULL (BSD loopback), snapshot length 262144 bytes
18:01:18.926619 IP6 (flowlabel 0x6a58e, hlim 90, next-header TCP (6) payload length: 32) fd01::1:172:20:20:10.39346 > 64:ff9b::44f:8ec8.443: Flags [ S ], cksum 0xff4d (correct), seq 1104216988, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
18:01:18.926638 IP (tos 0x0, ttl 89, id 25707, offset 0, flags [none], proto TCP (6), length 52)
192.168.255.195.39346 > 4.79.142.200.443: Flags [ S ], cksum 0x3da6 (correct), seq 1104216988, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
18:01:18.960684 IP (tos 0x0, ttl 120, id 16200, offset 0, flags [DF], proto TCP (6), length 52)
4.79.142.200.443 > 192.168.255.195.39346: Flags [S.], cksum 0xf9b8 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:18.960718 IP6 (hlim 119, next-header TCP (6) payload length: 32) 64:ff9b::44f:8ec8.443 > fd01::1:172:20:20:10.39346: Flags [S.], cksum 0xbb60 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:21.965711 IP (tos 0x0, ttl 120, id 17602, offset 0, flags [DF], proto TCP (6), length 52)
4.79.142.200.443 > 192.168.255.195.39346: Flags [S.], cksum 0xf9b8 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:21.965742 IP6 (hlim 119, next-header TCP (6) payload length: 32) 64:ff9b::44f:8ec8.443 > fd01::1:172:20:20:10.39346: Flags [S.], cksum 0xbb60 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:27.971559 IP (tos 0x0, ttl 120, id 18655, offset 0, flags [DF], proto TCP (6), length 48)
4.79.142.200.443 > 192.168.255.195.39346: Flags [S.], cksum 0x2dc8 (correct), seq 1881125064, ack 1104216989, win 65535, options [mss 1460,nop,nop,sackOK], length 0
18:01:27.971590 IP6 (hlim 119, next-header TCP (6) payload length: 28) 64:ff9b::44f:8ec8.443 > fd01::1:172:20:20:10.39346: Flags [S.], cksum 0xef6f (correct), seq 1881125064, ack 1104216989, win 65535, options [mss 1460,nop,nop,sackOK], length 0
18:01:39.967745 IP (tos 0x0, ttl 120, id 20842, offset 0, flags [DF], proto TCP (6), length 40)
4.79.142.200.443 > 192.168.255.195.39346: Flags [R], cksum 0x5a98 (correct), seq 1881125065, win 0, length 0
18:01:39.967789 IP6 (hlim 119, next-header TCP (6) payload length: 20) 64:ff9b::44f:8ec8.443 > fd01::1:172:20:20:10.39346: Flags [R], cksum 0x1c40 (correct), seq 1881125065, win 0, length 0
10 packets captured
10 packets received by filter
0 packets dropped by kernel
Internal Interface
[root@EFW ~]# tcpdump -vvvnivlan01 host 64:ff9b::44f:8ec8 or host 4.79.142.200
tcpdump: listening on vlan01, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:01:18.926584 IP6 (flowlabel 0x6a58e, hlim 91, next-header TCP (6) payload length: 32) fd01::1:172:20:20:10.39346 > 64:ff9b::44f:8ec8.443: Flags [ S ], cksum 0xff4d (correct), seq 1104216988, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
1 packet captured
2694 packets received by filter
0 packets dropped by kernel
I think this one counter `failures of source address selection` is the symptom, as it tends to increase ~60seconds after each test
[root@EFW ~]# netstat -s -p ip6
ip6:
34443359 total packets received
0 with size smaller than minimum
0 with data size < data length
0 with bad options
276 with incorrect version number
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 fragments that exceeded limit
0 atomic fragments
0 packets reassembled ok
1864797 packets for this host
31895889 packets forwarded
0 packets not forwardable
0 redirects sent
3043173 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
2 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
4 packets that violated scope rules
62 multicast packets which we don't join
Input histogram:
hop by hop: 699
TCP: 32210783
UDP: 1981356
ICMP6: 250233
PIM: 12
Mbuf statistics:
16835632 one mbuf
two or more mbuf:
lo0= 2124
wg1= 377865
17227738 one ext mbuf
0 two or more ext mbuf
0 packets whose headers are not contiguous
0 tunneling packets that can't find gif
0 packets discarded because of too many headers
2648 failures of source address selection
source addresses on an outgoing I/F
53783 link-locals
77070 globals
source addresses on a non-outgoing I/F
82 globals
2648 addresses scope=0xf
source addresses of same scope
53780 link-locals
77152 globals
source addresses of a different scope
3 link-locals
Source addresses selection rule applied:
130935 first candidate
15095 same address
53708 appropriate scope
48685 outgoing interface
82 matching label
42921 longest match
Fw info: (OPNsense 26.1.8_5-amd64)
routes:
[root@EFW ~]# netstat -rn
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default publicipv4gw UGS igb0
1.1.1.1 publicipv4gw UGHS igb0
10.10.0.0/16 172.20.0.2 UGS ixl2
publicipv4block/24 link#1 U igb0
publicipv4 link#14 UHS lo0
127.0.0.1 link#14 UH lo0
172.20.0.0/30 link#12 U ixl2
172.20.0.1 link#14 UHS lo0
172.20.19.0/29 link#5 U igb4
172.20.19.1 link#14 UHS lo0
172.20.20.0/24 link#18 U vlan01
172.20.20.1 link#14 UHS lo0
172.20.21.4/30 link#2 U igb1
172.20.21.5 link#14 UHS lo0
172.20.22.0/24 link#19 U vlan02
172.20.22.1 link#14 UHS lo0
172.20.24.0/24 link#20 U vlan03
172.20.24.1 link#14 UHS lo0
172.20.253.0/30 link#22 U wg1
172.20.253.1 link#14 UHS lo0
192.168.12.0/24 link#4 U igb3
192.168.12.1 link#14 UHS lo0
192.168.255.0/24 link#24 US nat64
192.168.255.1 link#24 UH nat64
Internet6:
Destination Gateway Flags Netif Expire
default fe80::256:2bff:fe76:b022%igb0 UG igb0
::1 link#14 UHS lo0
64:ff9b::/96 link#24 US nat64
publicipv6 link#14 UHS lo0
publicipv6prefix::/60 link#14 USB lo0
publicipv6 fe80::256:2bff:fe76:b022%igb0 UGHS igb0
fd01:0:0:1::/64 link#18 U vlan01
fd01::1:172:20:20:1 link#14 UHS lo0
fd01::1:172:20:20:10 link#18 UHS vlan01
fd01:0:0:2::/64 link#19 U vlan02
fd01::2:172:20:22:1 link#14 UHS lo0
fd01:0:0:3::/64 link#20 U vlan03
fd01::3:172:20:24:1 link#14 UHS lo0
fd01:0:0:4::/64 link#5 U igb4
fd01::4:172:20:19:1 link#14 UHS lo0
fd01:0:0:8::/64 fd01::a:172:20:0:0 UGS ixl2
fd01::a:10:10:0:0/126 fd01::a:172:20:0:0 UGS ixl2
fd01::a:172:20:0:0/127 link#12 U ixl2
fd01::a:172:20:0:1 link#14 UHS lo0
fd01::a:172:20:21:0 link#14 UHS lo0
fd01::a:172:20:21:0/127 link#13 U ixl3
fd01::a:172:20:21:4/127 link#2 U igb1
fd01::a:172:20:21:5 link#14 UHS lo0
fd01::a:172:20:25:0 link#14 UHS lo0
fd01::a:172:20:25:0/127 link#3 U igb2
fd01::a:172:20:253:2/127 link#22 U wg1
fd01::a:172:20:253:3 link#14 UHS lo0
fd01:0:0:f::/64 link#4 U igb3
fd01:0:0:f::1 link#14 UHS lo0
fe80::%igb0/64 link#1 U igb0
fe80::a236:9fff:fe89:60e7%lo0 link#14 UHS lo0
fe80::%igb1/64 link#2 U igb1
fe80::7ec2:55ff:fe2e:2c71%lo0 link#14 UHS lo0
fe80::%igb2/64 link#3 U igb2
fe80::7ec2:55ff:fe2e:2c72%lo0 link#14 UHS lo0
fe80::%igb3/64 link#4 U igb3
fe80::7ec2:55ff:fe2e:2c73%lo0 link#14 UHS lo0
fe80::%igb4/64 link#5 U igb4
fe80::7ec2:55ff:fe2e:2c74%lo0 link#14 UHS lo0
fe80::%igb6/64 link#7 U igb6
fe80::7ec2:55ff:fe2e:2c76%lo0 link#14 UHS lo0
fe80::%ixl2/64 link#12 U ixl2
fe80::7ec2:55ff:fe25:88%lo0 link#14 UHS lo0
fe80::%ixl3/64 link#13 U ixl3
fe80::7ec2:55ff:fe25:89%lo0 link#14 UHS lo0
fe80::%lo0/64 link#14 U lo0
fe80::1%lo0 link#14 UHS lo0
fe80::%vlan01/64 link#18 U vlan01
fe80::7ec2:55ff:fe25:89%lo0 link#14 UHS lo0
fe80::%vlan02/64 link#19 U vlan02
fe80::7ec2:55ff:fe25:89%lo0 link#14 UHS lo0
fe80::%vlan03/64 link#20 U vlan03
fe80::7ec2:55ff:fe25:89%lo0 link#14 UHS lo0
fe80::%vlan04/64 link#21 U vlan04
fe80::7ec2:55ff:fe2e:2c76%lo0 link#14 UHS lo0
Interfaces:
nat64: flags=1008051<UP,POINTOPOINT,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4080000<LINKSTATE,MEXTPG>
inet 172.20.0.1 --> 192.168.255.1 netmask 0xffffffff
groups: tun tayga
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Opened by PID 85915
drivername: tun0
vlan01: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: EdgeNet (opt7)
options=4000000<MEXTPG>
ether 7c:c2:55:25:00:89
inet 172.20.20.1 netmask 0xffffff00 broadcast 172.20.20.255
inet6 fd01::1:172:20:20:1 prefixlen 64
inet6 fe80::7ec2:55ff:fe25:89%vlan01 prefixlen 64 scopeid 0x12
groups: vlan
vlan: 120 vlanproto: 802.1q vlanpcp: 0 parent interface: ixl3
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active
nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
drivername: vlan0
ixl3: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: DMZSRV (opt2)
options=48500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,HWSTATS,MEXTPG>
ether 7c:c2:55:25:00:89
inet6 fd01::a:172:20:21:0 prefixlen 127
inet6 fe80::7ec2:55ff:fe25:89%ixl3 prefixlen 64 scopeid 0xd
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active
nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
drivername: ixl3
igb0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN (wan)
options=48520b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,HWSTATS,MEXTPG>
ether a0:36:9f:89:60:e7
hwaddr 7c:c2:55:2e:2c:70
inet publicipv4 netmask 0xffffff00 broadcast 255.255.255.255
inet6 fe80::a236:9fff:fe89:60e7%igb0 prefixlen 64 scopeid 0x1
inet6 publicipv6 prefixlen 128 pltime 86400 vltime 86400
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
drivername: igb0
Client test : (
$ nc -6 -w 1 -vz grc.com 443
Ncat: Version 7.95 ( https://nmap.org/ncat )
Ncat: TIMEOUT.
#8
26.1, 26,4 Series / Re: Unable to get IPv6 Traffic...
Last post by space_cadet - Today at 02:23:33 AMHey, sorry to hear your IPv6 settings are still not working.
1. My RA settings are blank. WireGuard assigns the IPv6 addresses, and I'm using Dnsmasq for DHCPv6 address assignment on my LAN.
2. NAT Outbound is set to Automatic Outbound NAT Rule Generation.
3. NAT NPTv6 is also blank.
What settings do you have set up for your WireGuard Server Instance? I know you have requested a /56 prefix from your ISP, but are you sure it's honoring it? When I tried to use a /56 the ISP kicked me back to a /64. The lowest it would let me go was a /60. See my last post regarding the Interface > Overview, to ensure your ISP is honoring the /56 prefix.
I also see you have 2a01:xxxx:xxxx:xx03::9001 listed as your DNS. When you entered this into your tunnel address, did you list it as 2a01:xxxx:xxxx:xx03::9001/64? This is just like setting the subnet mask for IPv4. The /64 in the tunnel address tells WireGuard that you've set the prefix for your VPN to 2a01:xxxx:xxxx:xx03. You should also make sure that your LAN is not using the same prefix as your VPN.
1. My RA settings are blank. WireGuard assigns the IPv6 addresses, and I'm using Dnsmasq for DHCPv6 address assignment on my LAN.
2. NAT Outbound is set to Automatic Outbound NAT Rule Generation.
3. NAT NPTv6 is also blank.
What settings do you have set up for your WireGuard Server Instance? I know you have requested a /56 prefix from your ISP, but are you sure it's honoring it? When I tried to use a /56 the ISP kicked me back to a /64. The lowest it would let me go was a /60. See my last post regarding the Interface > Overview, to ensure your ISP is honoring the /56 prefix.
I also see you have 2a01:xxxx:xxxx:xx03::9001 listed as your DNS. When you entered this into your tunnel address, did you list it as 2a01:xxxx:xxxx:xx03::9001/64? This is just like setting the subnet mask for IPv4. The /64 in the tunnel address tells WireGuard that you've set the prefix for your VPN to 2a01:xxxx:xxxx:xx03. You should also make sure that your LAN is not using the same prefix as your VPN.
#9
General Discussion / Re: What is the Purpose of 'op...
Last post by Patrick M. Hausen - Today at 12:43:03 AMThe OS names change when the hardware does.
Switch from 1G to 10G by adding a new 10G NIC? Simply reassign the logical interface to the new device - done. IP configuration, DHCP, firewall rules, NAT, ... all will just follow with a single click.
Switch from 1G to 10G by adding a new 10G NIC? Simply reassign the logical interface to the new device - done. IP configuration, DHCP, firewall rules, NAT, ... all will just follow with a single click.
#10
26.1, 26,4 Series / DEC840 gets stuck on boot if a...
Last post by feld - Today at 12:07:55 AMOPNsense 26.1.8_5-amd64
I have Quantum Fiber for my internet service and my CPE equipment is a C5500XK. Unfortunately I don't have a GPON SFP that will let me bypass this equipment at this time.
I have a 10GBASE-T SFP+ 30m https://www.fs.com/products/111919.html
If I attempt to boot my DEC840 with this physical link up it gets stuck and fails to complete booting. I have tried replacing this SFP as I have many on hand, but they all produce the same behavior. Changing this out for an 1G transceiver works fine and it boots noticeably faster than the 10G.
I am attaching the boot log captured over serial console at boot via script+screen. I had to clean this up a bit as it had all the terminal control characters and color sequences littered throughout but I think it's fully intact.
What I have observed is that there are some additional i2c devices detected when the 10G module is used, and it seems to fail to configure/communicate with the SFP based on the timeouts it displays in the boot log.
If I boot the firewall without the physical link up (unplug ethernet), and then plug in after fully booted it works fine -- or at least I don't notice any issues with the link or its stability.
edit: I want to mention that this hardware configuration worked fine for a couple years, but I can't be certain when this problem started happening. I believe it started somewhere within the 25.x series.
I have Quantum Fiber for my internet service and my CPE equipment is a C5500XK. Unfortunately I don't have a GPON SFP that will let me bypass this equipment at this time.
I have a 10GBASE-T SFP+ 30m https://www.fs.com/products/111919.html
If I attempt to boot my DEC840 with this physical link up it gets stuck and fails to complete booting. I have tried replacing this SFP as I have many on hand, but they all produce the same behavior. Changing this out for an 1G transceiver works fine and it boots noticeably faster than the 10G.
I am attaching the boot log captured over serial console at boot via script+screen. I had to clean this up a bit as it had all the terminal control characters and color sequences littered throughout but I think it's fully intact.
What I have observed is that there are some additional i2c devices detected when the 10G module is used, and it seems to fail to configure/communicate with the SFP based on the timeouts it displays in the boot log.
If I boot the firewall without the physical link up (unplug ethernet), and then plug in after fully booted it works fine -- or at least I don't notice any issues with the link or its stability.
edit: I want to mention that this hardware configuration worked fine for a couple years, but I can't be certain when this problem started happening. I believe it started somewhere within the 25.x series.
