Messages

Great, thank you for the documentation and the PR. Perhaps it is also worth mentioning that an ULA IPv6 address can be used. This is a working configuration:

Code Select Expand


IPv4 Address: 10.14.0.1                         # from IPv4 Pool
IPv4 NAT64 Interface Address: 10.15.0.1         # not from IPv4 Pool
IPv6 Address: <GUA IPv6 address>                # one of your "public" IPv6 adresses
IPv6 NAT64 Interface Address: fd00:14::1        # not used by other interfaces
IPv6 Prefix: 64:ff9b::/96                       # well-known prefix
IPv4 Pool: 10.14.0.0/16

Don't forget DNS64 configuration and firewall/nat rules as documented by Maurice.

[For simplicity, you may reuse an address of another OPNsense interface.]

After a more detailed analysis in the last few days we realized that this setence from the Tayga plugin documentation (https://docs.opnsense.org/manual/how-tos/tayga.html) is our problem:

IPv6 NAT64 Interface Address: ... For simplicity, you may reuse an address of another OPNsense interface.

Looking at the start script /usr/local/etc/rc.d/opnsense-tayga you can see, that it tries to configure the IPv6 NAT64 Interface Address on the nat64 interface. If the IP is already used by another interface, this configuration will fail and ifconfig nat64 shows only an IPv4 but not an IPv6 address. The result ist, that tayga will not work (completely) as reported. Changing the IPv6 NAT64 Interface Address leads to a stable working in our environment.

Anyone? Perhaps tips for further debugging? Is there any (network interface, memory, buffer, cache, ... related) command that shows what happens to the packet after leaving the nat64 interface? Or kind of command/filter that allows to observe the packages while traveling through all interfaces? Thanks again + Best

[Thank you!!!]

Hello, to connect our IPv6-only local network (vlan2) to the IPv4 internet, we configured NAT64 as described in the Tayga tutorial. But at first the connection did not work. Tcpdump shows an almost functioning routing path, but from the last "NAT64 hop" the packet is not forwarded back to vlan2. After hours, however, it suddenly worked, even though we had not made any configuration changes (we only changed logging settings during that time, but that does not seem to be related, we could not reproduce the behavior)... Here the excerpt from the tcpdump traces:

Code Select Expand

# THE ENVIRONMENT
foo.example.org = 1.2.3.4 = A public server with only an IPv4 address
14141 = TCP Test Port that is listening on 1.2.3.4
64:ff9b::aaaa:bbbb = the (anonymised) IPv6 translation of 1.2.3.4 address
2001:db8:0000:...  = the (anonymised) IPv6 prefix (assigned by ISP)
2001:db8:0000:2::1 = OPNsense incl. Tayga and Unbound DNS
100.100.100.136/29 = the (anonymised) public IPv4 network (assigned by ISP)


# CONNECTING FROM VLAN2 HOST
# server1 (2001:db8:0000:2::10, vlan2)
$ nc foo.example.org 14141
<hangs until timeout>


# TCPDUMPS AT OPNSENSE
opnsense$ tcpdump -n -i bge0_vlan2
IP6 2001:db8:0000:2::10.52218 > 2001:db8:0000:2::1.53: 45473+ A? foo.example.org. (35)
IP6 2001:db8:0000:2::10.52218 > 2001:db8:0000:2::1.53: 44454+ AAAA? foo.example.org. (35)
IP6 2001:db8:0000:2::1.53 > 2001:db8:0000:2::10.52218: 45473 1/0/0 A 1.2.3.4 (51)
IP6 2001:db8:0000:2::1.53 > 2001:db8:0000:2::10.52218: 44454 1/0/0 AAAA 64:ff9b::aaaa:bbbb (63)

opnsense$ tcpdump -n -i bge0_vlan2
IP6 2001:db8:0000:2::10.51300 > 64:ff9b::aaaa:bbbb.14141: Flags [S],...

opnsense$ tcpdump -n -i nat64
IP6 2001:db8:0000:2::10.51314 > 64:ff9b::aaaa:bbbb.14141: Flags [S], ...
IP 10.64.120.219.51314 > 1.2.3.4.14141: Flags [S], ...

opnsense$ tcpdump -n -i bnxt1 port 14141   # WAN interface
IP 100.100.100.140.38343 > 1.2.3.4.14141: Flags [S], ...
IP 1.2.3.4.14141 > 100.100.100.140.38343: Flags [S.], ...

opnsense$ tcpdump -n -i nat64
IP 1.2.3.4.14141 > 10.64.120.219.51314: Flags [S.], ...
IP6 64:ff9b::aaaa:bbbb.14141 > 2001:db8:0000:2::10.51314: ...

opnsense$ tcpdump -n -i bge0_vlan2
IP6 2001:db8:0000:2::10.51300 > 64:ff9b::aaaa:bbbb.14141: Flags [S], ...
IP6 2001:db8:0000:2::10.51300 > 64:ff9b::aaaa:bbbb.14141: Flags [S], ...
IP6 2001:db8:0000:2::10.51300 > 64:ff9b::aaaa:bbbb.14141: Flags [S], ...

As you can see: NAT64 sends "64:ff9b::aaaa:bbbb.14141 > 2001:db8:0000:2:10" but it does not reach the bge0_vlan2 interface (or any other, we checked all). Firewall rules do not block the connection, despite activating the entire logging (incl. default rules), the log does not show any related entries. And as I wrote: After hours it suddenly worked and the final vlan2 tcpdump showed the packages and the nc command got the response from 1.2.3.4's listening nc. But since we had little confidence in the situation, we restarted the opnsense server and unfortunately the old state was immediately restored: The connection failed again. So, we have the following questions and would appreciate your help:

• Are there obvious configuration errors? We have attached our settings below.
• Since there seems to be a time dependency: Are there any caches/buffers that we should flush?
• Is it normal that the nat64 interface has only an ipv4 but not an ipv6 address?
• Are there (FreeBSD) command/tools we can use for deeper troubleshooting?

Thank you!!!


Our setup:

Code Select Expand

##### ISP Gateway:
Static IP 2001:db8:0000::1
Upstream Gateway = Yes
Disable Gateway Monitoring = Yes
Disable reply-to on WAN rules  = No   # also tried Yes with same result


##### Tayga and Unbound DNS:
IPv4 Address                            10.64.0.1
IPv4 NAT64 Interface Address    10.65.64.1
IPv6 Address                            2001:db8:0000:5001:64::1
IPv6 NAT64 Interface Address    2001:db8:0000::4
IPv6 Prefix                                64:ff9b::/96
IPv4 Pool                                  10.64.0.0/16
Enable DNS64 Support =           Yes
DNS64 Prefix = Not set             # to use default 64:ff9b::/96


##### NAT 64 interface and Routing Table:
opnsense$ ifconfig nat64
nat64: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
inet 10.65.64.1 --> 10.64.0.1 netmask 0xffffffff
groups: tun
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Opened by PID 85229

opnsense$ netstat -rn
Destination        Gateway            Flags     Netif Expire
default            100.100.100.137     UGS       bnxt1
10.64.0.0/16       link#16            US        nat64
10.64.0.1          link#16            UH        nat64
10.65.64.1         link#16            UHS         lo0
100.100.100.136/29 link#4             U         bnxt1
100.100.100.140    link#4             UHS         lo0
127.0.0.1          link#5             UH          lo0

Internet6:
Destination               Gateway            Flags     Netif Expire
default                   2001:db8:0000::1   UGS       bnxt1
::1                       link#5             UHS         lo0
64:ff9b::/96              link#16            US        nat64
2001:db8:0000::/48        link#4             U         bnxt1
2001:db8:0000::4          link#4             UHS         lo0
2001:db8:0000:1::/64      link#10            U      bge0_vlan1
2001:db8:0000:1::1        link#10            UHS         lo0
2001:db8:0000:2::/64      link#9             U      bge0_vlan2
2001:db8:0000:2::1        link#9             UHS         lo0
2001:db8:0000:3::/64      link#11            U      bnxt0_vlan3
2001:db8:0000:3::1        link#11            UHS         lo0


##### pf/NAT rules:
Tayga Interface:
  pass IPv4 from 10.64.0.0/16 to any  # also tried any to any with same result
Vlan 2 Interface:
  pass IPv6 from any to any
NAT Outbound:
  IPv4, Source = 10.64.0.0/16, Destination = any, Translation = WAN Address (100.100.100.140)

... or let the driver do it as soon as it has VLANs is exactly the same from a hardware standpoint once VLAN is used in your network: promiscuous mode is enabled.

Thanks for the explanation, i had not thought of this point, but it seems logical of course.

Also thanks to gege29 for the experience report, what is always helpful when making a decision: We will also enable the promisc mode - Now, with a (reasonably) good feeling :)

Good explanation! We will discuss this point in the team. Thank you.

[Are there any security impacts due to we should not permanently enable the promisc mode until the bug is fixed?]

Hi, after the successful integration of the bnxt network interface driver [1], we realized that the driver has an error in the processing of vlan packets [2]. To fix this issue, the associated FreeBSD ticket mentions the possibility of activating the promiscuous mode for bnxt0 as a workaround. Away from performance drawbacks and under the assumption that only privileged administrators have access to the OPNsense instance...

Are there any security impacts due to we should not permanently enable the promisc mode until the bug is fixed?

Thank you for your assessment.


[1] Enabling bnxt driver: https://forum.opnsense.org/index.php?topic=27168.0
[2] FreeBSD bnxt vlan bugzilla ticket: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236983

To answer my newbie question finally: I failed to activate the module in the early startup phase (FreeBSD Boot-Stage3). Therefore the bnxt NICs were not available during the initial OPNsense installation steps (within the console). Instead, the module must be activated via OPNsense configuration after installation and then the network interfaces must be reconfigured. Instructions for loading modules can be found in the relevant FreeBSD file /boot/loader.conf:

Code Select Expand

# OPNsense console menu: 8->Shell

root$  ls /boot/kernel/*bnxt*        # check if module exists
/boot/kernel/if_bnxt.ko

root$ cat /boot/loader.conf
...
# In order to deploy a custom change to this installation,
# please use /boot/loader.conf.local as it is not rewritten,
# or better yet use System:Settings:Tunables from the GUI.
...

So to load bnxt, the System/Settings/Tunables entry if_bnxt_load with the value YES is added in the web GUI. After a subsequent reboot, all my network interfaces were available.

[Do I miss a (kernel) boot flag to force read-write access?]

Ah, thank you. I tried to load the module but without loading the kernel before (I didn't realize that I can enter more commands after kernel loading). Ok, your commands work: They load the kernel and the module and then start the kernel. But unfortunately the boot process aborts at a later point due to read-only file system:

Code Select Expand


Press any key to start the configuration importer: ........
mkdir: /conf/backup: Read-only file system
...: Read-only file system
override r--r--r-- root/wheel for /var/run/ld-elf32.so.hints?


Do I miss a (kernel) boot flag to force read-write access?


FYI: I also tried:

Code Select Expand

Boot Menu -> 3 (Escape to loader prompt)
OK  unload
OK  set if_bnxt_load="YES"
OK  boot-conf
Loading kernel...

But I doesn't load the bnxt module.

Thanks again

[bnxt]

Hello,

I am installing OPNsense for the first time and not sure if I have a OPNsense or FreeBSD question:

The first (console based) installation step doesn't list all network interfaces. It is missing two bnxt interfaces. I tried to enable the bnxt module in FreeBSD Boot-Stage-Three:

Boot Menu -> 3 (Escape to loader prompt)
OK  set if_bnxt_load="YES"
OK  boot
Loading kernel...

But the two nics are still missing. Is it possible to load the bnxt module before installation?

Thank you and best