Hey there,
this is my first post, so please be kind.
I tried to find my answers in the documentation, in this forum and on the web.
I am struggling to get One to One NAT to work in my setup.
ContextI have recently rebuilt my firewall setup and switched to OPNsense on a Sophos SG 135 rev3 box.
I was running a PFsense installation inside a VM previously.
So far, most things run fine, but I cannot get 1:1 NAT to work.
Glossary
Term | Description |
mLAN | The main LAN of my uplink router (not OPNsense) |
oWAN | The WAN interface of OPNsense (part of mLAN) |
oLAN | The LAN of OPNsense. |
My SetupI use OPNsense in a home lab environment with the goal of separating my lab from the rest of my main LAN (mLAN).
mLAN: 192.168.2.0/23 (yes, 23 not 24) - Uplink gateway: 192.168.2.1
oWAN configuration: 192.168.2.2/23 - Gateway 192.168.2.1
oLAN configuration: 10.10.0.1/16
Behind OPNsense I run a Proxmox host with several containers and VMs, assigned different VLANs.
The full setup is connected via a managed L2 switch.
The switch is connected to the upstream router of the LAN, the firewall, mLAN devices and proxmox.
I separated the devices via port based VLANs.
What I am trying to achieveI wish to expose one of the containers inside oLAN to oWAN (consequently to mLAN) to a WAN IP.
oLAN IP: 10.10.2.101
oWAN IP: 192.168.3.101
One thing to note: the container runs in a VLAN with ID 20 on a separate Interface on SN 10.10.2.0/24 (GW 10.10.2.1)
The container has internet access and uses Gateway 10.10.2.1.
The problem of my current setupWhen pinging the container from oWAN (192.168.2.11 to be exact), the container receives the ICMP requests, sends out an echo, but the echo does not find its way back to 192.168.2.11.
My configurationCreated Virtual IP Address: 192.168.3.101/32
Interface: WAN
Type: IP Alias
One-to-One NAT Interface: WAN
Type: BINAT
External network: 192.168.3.101/32
Source/Internal: 10.10.2.101
Destination: any
NAT reflection: default
Outbound NATMode: Automatic
I unsuccessfully tried Hybrid mode with a manual rule:
Interface: WAN
Source: 10.10.2.101/32
Destination: *
NAT Address: 192.168.3.101
Static Port: NO
Advanced Firewall SettingsNetwork Address Translation
Reflection for port forwards: on
Reflection for 1:1: on
Automatic outbound NAT for Reflection: on
I am running on the latest OPNsense version 24.7.8-amd64.
Thank you for all offered help. I will gladly provide more information.
Best Regards,
Jannick
Consider hostname.int.example.com the hostname of the container
The ICMP tcpdump output shows that the mLAN gateway advises to send packets to 192.168.3.11 directly.
I am unsure if this is a misconfiguration in OPNsense or somewhere else. Imo the request should never pass through the Uplink gateway at all...
19:14:27.705347 IP 192.168.3.11 > hostname.int.example.com: ICMP echo request, id 29, seq 1, length 64
19:14:27.705358 IP hostname.int.example.com > 192.168.3.11: ICMP echo reply, id 29, seq 1, length 64
19:14:27.706141 IP 192.168.2.1 > hostname.int.example.com: ICMP redirect 192.168.3.11 to host 192.168.3.11, length 92
Quote from: joursin on November 13, 2024, 04:32:02 PM
oLAN configuration: 10.10.0.1/16
[snip]
One thing to note: the container runs in a VLAN with ID 20 on a separate Interface on SN 10.10.2.0/24 (GW 10.10.2.1)
You realise that 10.10.2.0/24 is a subnet within 10.10.0.0/16, right? Is this "separate interface" on OPNsense or something else?
Quote
You realise that 10.10.2.0/24 is a subnet within 10.10.0.0/16, right? Is this "separate interface" on OPNsense or something else?
You are of course correct that 10.10.2.0/24 is a different subnet than 10.10.0.0/16.
I reserve the entire subnet 10.10.0.0/16 for the oLAN interface.
The oLAN is split up into multiple VLANs.
1. A management VLAN 10 on 10.10.0.0/24 (should not be relevant here)
2. A services VLAN 20 on 10.10.2.0/24
3. Another VLAN for VPN connectivity (should not be relevant here)
OPNsense is aware of these VLANs and uses one interface per VLAN.
OK, so long as your VLAN 10 interface is configured with 10.10.0.1/24 and not /16 (as your original post suggested).
I don't see anything obvious wrong with your OPNsense setup as described, so I'd tend to suspect something with the container or proxmox setup. I'd probably get a shell on OPNsense an run tcpdump on the VLAN 20 interface to see if the ping response is coming back, and look towards proxmox if not....
Quote
OK, so long as your VLAN 10 interface is configured with 10.10.0.1/24 and not /16 (as your original post suggested).
Sorry for the misleading information. As I mentioned in the beginning, this is my first post. I'm trying my best to be clear.
QuoteI'd probably get a shell on OPNsense and run tcpdump on the VLAN 20 interface to see if the ping response is coming back, and look towards Proxmox if not....
I ran a packet capture on the WAN interface. The results show that the ping responses are leaving OPNsense with the correct target address, but the gateway still responds with an ICMP redirect. I think that proxmox is in the clear here.
21:03:22.053809 IP (tos 0x0, ttl 64, id 52340, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.3.11 > 192.168.3.101: ICMP echo request, id 38, seq 4, length 64
21:03:22.053925 IP (tos 0x0, ttl 63, id 53210, offset 0, flags [none], proto ICMP (1), length 84)
192.168.3.101 > 192.168.3.11: ICMP echo reply, id 38, seq 4, length 64
21:03:22.054586 IP (tos 0xc0, ttl 64, id 16302, offset 0, flags [none], proto ICMP (1), length 112)
192.168.2.1 > 192.168.3.101: ICMP redirect 192.168.3.11 to host 192.168.3.11, length 92
IP (tos 0x0, ttl 62, id 53210, offset 0, flags [none], proto ICMP (1), length 84)
192.168.3.101 > 192.168.3.11: ICMP echo reply, id 38, seq 4, length 64
I should emphasize the subnet mask on the mLAN: 192.168.2.0/23, covering the range 192.168.2.0 - 192.168.3.255.
I intended to reserve 192.168.2.0/24 for homelab devices and services exposed to the mLAN, while 192.168.3.0/24 is reserved for all mLAN devices not part of the homelab. (DHCP on the uplink router is configured so only IPs in the 192.168.3.0/24 subnet are assigned).
Could the issue be related to the subnet mask on the mLAN?
Maybe OPNsense routes the responses to the router first because it considers it part of the 192.168.2.0/23 subnet? I double-checked the WAN interface, which is configured as 192.168.2.2/23, so it should include the 192.168.3.0/24 subnet.
My assumption is that, if true, this may be caused by the usage of a virtual IP. (tho it should not have a subnet, should it?)
Just to make sure, I pinged 192.168.3.11 from OPNsense directly via shell, which worked perfectly fine.
root@OPNsense:~ # ping 192.168.3.11
PING 192.168.3.11 (192.168.3.11): 56 data bytes
64 bytes from 192.168.3.11: icmp_seq=0 ttl=64 time=0.263 ms
64 bytes from 192.168.3.11: icmp_seq=1 ttl=64 time=0.176 ms
64 bytes from 192.168.3.11: icmp_seq=2 ttl=64 time=0.214 ms
64 bytes from 192.168.3.11: icmp_seq=3 ttl=64 time=0.180 ms
Quote from: joursin on November 13, 2024, 09:30:10 PM
21:03:22.053809 IP (tos 0x0, ttl 64, id 52340, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.3.11 > 192.168.3.101: ICMP echo request, id 38, seq 4, length 64
21:03:22.053925 IP (tos 0x0, ttl 63, id 53210, offset 0, flags [none], proto ICMP (1), length 84)
192.168.3.101 > 192.168.3.11: ICMP echo reply, id 38, seq 4, length 64
21:03:22.054586 IP (tos 0xc0, ttl 64, id 16302, offset 0, flags [none], proto ICMP (1), length 112)
192.168.2.1 > 192.168.3.101: ICMP redirect 192.168.3.11 to host 192.168.3.11, length 92
IP (tos 0x0, ttl 62, id 53210, offset 0, flags [none], proto ICMP (1), length 84)
192.168.3.101 > 192.168.3.11: ICMP echo reply, id 38, seq 4, length 64
I should emphasize the subnet mask on the mLAN: 192.168.2.0/23, covering the range 192.168.2.0 - 192.168.3.255.
I intended to reserve 192.168.2.0/24 for homelab devices and services exposed to the mLAN, while 192.168.3.0/24 is reserved for all mLAN devices not part of the homelab. (DHCP on the uplink router is configured so only IPs in the 192.168.3.0/24 subnet are assigned).
Could the issue be related to the subnet mask on the mLAN?
I think so, yes. If the echo reply to 192.168.3.101 is being send to your "mLAN" gateway (192.168.2.1), it suggests that the netmask on OPNsense's WAN interface is not /23, but probably /24 ?
Thank you for your help so far!
The WAN interface uses the 23 Subnet mask, so the error must be elsewhere.
What kind of information can help debugging in this case?
Hmm. It might be interesting to see the output of `route get 192.168.3.11` in an OPNsense shell...
**Output:**
root@OPNsense:~ # route get 192.168.3.11
route to: 192.168.3.11
destination: 192.168.2.0
mask: 255.255.254.0
fib: 0
interface: ix1
flags: <UP,DONE,PINNED>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1500 1 0
The `ix1` interface represents the WAN.
**Interface Summary:**
0010_MANAGEMENT (vlan0.10) -> v4: 10.10.0.1/24
0015_OPENVPN (vlan0.15) -> v4: 10.10.1.1/30
0020_GAMESERVERS (vlan0.20) -> v4: 10.10.2.1/24
0040_CLOUD (vlan0.40) -> v4: 10.10.4.1/24
LAN (ix0) ->
WAN (ix1) -> v4: 192.168.2.2/23
WG_JUMP01 (wg0) -> v4: 10.0.0.1/24
LAN is not in use. All VLANs above use the ix0 (LAN) interface.
WG_JUMP01 is a WireGuard tunnel for incoming connections from a public VPS (used as a public IP). Multiple port forwards on WG_JUMP01 work perfectly.
Interestingly, regular port forwards on oWAN have the same issue. When attempting to SSH into a system NATed on oWAN by OPNsense, the request reaches the destination, but the response is redirected via 192.168.2.1.
Do you have any firewall rules which specifically use the (o)WAN gateway? Even then, I don't think it should cause this, as it's the response packets for sessions that should be already tracked ... but I'm running out of ideas :/
Sorry, I need a second to take a look at my configuration.
Right now, I seem to have broken something that prevents ping requests from reaching the container at all.
Needless to say, I am pretty frustrated at this point, but also very grateful that you still take your time to try and support me.
Quote from: dseven on November 14, 2024, 02:09:34 PM
Do you have any firewall rules which specifically use the (o)WAN gateway? Even then, I don't think it should cause this, as it's the response packets for sessions that should be already tracked ... but I'm running out of ideas :/
I found the low level explanation. The L2 Frames are routed to the Gateway rather than the target host directly:
| IP Address | MAC Address | Description |
| 192.168.2.1 | 18:82:8c:05:be:20 | Gateway |
| 192.168.3.11 | dc:a6:32:0c:da:28 | Accessing Host |
| 192.168.2.103 | 7c:5a:1c:7c:f9:cc | Exposed Host |
13:49:41.518261 dc:a6:32:0c:da:28 (oui Unknown) > 7c:5a:1c:7c:f9:cc (oui Unknown), ethertype IPv4 (0x0800), length 74: (tos 0x48, ttl 64, id 38546, offset 0, flags [DF], proto TCP (6), length 60)
192.168.3.11.53974 > 192.168.3.101.ssh: Flags [S], cksum 0xe6ad (correct), seq 2350197043, win 64240, options [mss 1460,sackOK,TS val 821306743 ecr 0,nop,wscale 7], length 0
13:49:41.518448 7c:5a:1c:7c:f9:cc (oui Unknown) > 18:82:8c:05:be:20 (oui Unknown), ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
192.168.3.101.ssh > 192.168.3.11.53974: Flags [S.], cksum 0xdc32 (correct), seq 865488568, ack 2350197044, win 65160, options [mss 1460,sackOK,TS val 265254068 ecr 821306743,nop,wscale 7], length 0
The configuration of the WAN interface is as such:
192.168.2.2/23Manually pinging
192.168.3.11 from
10.10.2.101 (the host mapped to
192.168.3.101) works perfectly fine.
I am starting to doubt that this is a configuration error and suspect a bug of some sort. Any ideas? :-(
I was able to resolve the problem.
I have enabled the option "Disable reply-to" under "Firewall > Settings > Advanced".
I am unsure if this is the correct way to go, but I will go on like this and see if anything else breaks.
I was pondering reply-to, but I wasn't able to reproduce what you were seeing. Do you have any inbound firewall rules (aside from your NAT ones) on your WAN interface?
Yes, I have these rules active on the WAN interface:
1. Allow Traffic from PiKVM (from WAN) to Firewall Web interface
2. Floating Rule: Block Traffic to Management Lan unless coming from Management Lan (sn 10.10.0.0/24)
3. Block mDNS and netBIOS traffic from Uplink router to oLAN network 10.10.0.0/16. (unnecessary traffic)
The last rule is just for experimentation, but disabling it had not effect on the issue discussed.