Hi folks,
I'm pulling my hair out on this one.
I'm trying to route traffic from a specific network to a VPN gateway.
It works great for ping initiated from my host (ovpnc5 is the VPN interface I should go through):
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ovpnc5, link-type NULL (BSD loopback), capture size 262144 bytes
08:54:14.110777 IP 10.101.0.100 > 1.1.1.1: ICMP echo request, id 68, seq 1, length 64
08:54:14.124537 IP 1.1.1.1 > 10.101.0.100: ICMP echo reply, id 68, seq 1, length 64
But for traffic the host replies to, it goes through igb2 and immediatly goes out igb1 instead of the matching ovpnc5 from the gateway rule.
# igb2 is the interface for the 10.101.0.100 network
root@opnsense:~ # tcpdump -i igb2 -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb2, link-type EN10MB (Ethernet), capture size 262144 bytes
08:45:35.525754 IP xxx.xxx.xxx.xxx.xxx > 10.101.0.100: ICMP echo request, id 50180, seq 220, length 64
08:45:35.525947 IP 10.101.0.100 > xxx.xxx.xxx.xxx.xxx: ICMP echo reply, id 50180, seq 220, length 64
# igb1 is the interface for the default route network
root@opnsense:~ # tcpdump -i igb1 -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 262144 bytes
08:46:30.715195 IP 10.101.0.100 > xxx.xxx.xxx.xxx.xxx: ICMP echo reply, id 50180, seq 275, length 64
08:46:31.719935 IP 10.101.0.100 > xxx.xxx.xxx.xxx.xxx: ICMP echo reply, id 50180, seq 276, length 64
08:46:32.723703 IP 10.101.0.100 > xxx.xxx.xxx.xxx.xxx: ICMP echo reply, id 50180, seq 277, length 64
Now the replies goes a different route and never works.
Does that makes sense ?
maybe affected by this one?
https://github.com/opnsense/docs/commit/ec62439d9ceeaed8e236cb2b834dfda34c0ada38
Thank you, the issue certainly looks similar.
But I think I disabled reply-to pretty much across the board.
I'm wondering if my setup can be called "multiple wan", as in some way I'm making another WAN through VPN...
It is, yes, as it is used as an own uplink
I'll get some traces and report
That's the trace from OPNsense LAN interface, facing the machine that replies to ping
1 2020-12-11 21:18:18.753424 xxx.xxx.xxx.xxx.xxx 10.101.0.100 ICMP 98 Echo (ping) request id=0xa25f, seq=33/8448, ttl=51 (reply in 2)
Ethernet II, Src: IskraTra_d0:9b:e3 (00:0e:c4:d0:9b:e3), Dst: VMware_a4:72:a2 (00:50:56:a4:72:a2)
2 2020-12-11 21:18:18.753621 10.101.0.100 xxx.xxx.xxx.xxx.xxx ICMP 98 Echo (ping) reply id=0xa25f, seq=33/8448, ttl=64 (request in 1)
Ethernet II, Src: VMware_a4:72:a2 (00:50:56:a4:72:a2), Dst: IskraTra_d0:9b:e3 (00:0e:c4:d0:9b:e3)
That's full packet 2, the ping reply :
Frame 2: 98 bytes on wire (784 bits), 98 bytes captured (784 bits)
Encapsulation type: Ethernet (1)
Arrival Time: Dec 11, 2020 21:18:18.753621000 CET
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1607717898.753621000 seconds
[Time delta from previous captured frame: 0.000197000 seconds]
[Time delta from previous displayed frame: 0.000197000 seconds]
[Time since reference or first frame: 0.000197000 seconds]
Frame Number: 2
Frame Length: 98 bytes (784 bits)
Capture Length: 98 bytes (784 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:ethertype:ip:icmp:data]
[Coloring Rule Name: ICMP]
[Coloring Rule String: icmp || icmpv6]
Ethernet II, Src: VMware_a4:72:a2 (00:50:56:a4:72:a2), Dst: IskraTra_d0:9b:e3 (00:0e:c4:d0:9b:e3)
Destination: IskraTra_d0:9b:e3 (00:0e:c4:d0:9b:e3)
Address: IskraTra_d0:9b:e3 (00:0e:c4:d0:9b:e3)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Source: VMware_a4:72:a2 (00:50:56:a4:72:a2)
Address: VMware_a4:72:a2 (00:50:56:a4:72:a2)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 10.101.0.100, Dst: xxx.xxx.xxx.xxx.xxx
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
0000 00.. = Differentiated Services Codepoint: Default (0)
.... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
Total Length: 84
Identification: 0x9097 (37015)
Flags: 0x00
0... .... = Reserved bit: Not set
.0.. .... = Don't fragment: Not set
..0. .... = More fragments: Not set
Fragment Offset: 0
Time to Live: 64
Protocol: ICMP (1)
Header Checksum: 0xdf41 [validation disabled]
[Header checksum status: Unverified]
Source Address: 10.101.0.100
Destination Address: xxx.xxx.xxx.xxx.xxx
Internet Control Message Protocol
Type: 0 (Echo (ping) reply)
Code: 0
Checksum: 0xd1b5 [correct]
[Checksum Status: Good]
Identifier (BE): 41567 (0xa25f)
Identifier (LE): 24482 (0x5fa2)
Sequence Number (BE): 33 (0x0021)
Sequence Number (LE): 8448 (0x2100)
[Request frame: 1]
[Response time: 0.197 ms]
Timestamp from icmp data: Dec 11, 2020 21:18:18.748765000 CET
[Timestamp from icmp data (relative): 0.004856000 seconds]
And for some reason, this packet ends up on the default route interface, even though I have a rule (reply-to disabled) that says anything "in" this interface (igb2) set Gateway to OpenVPN gateway :
76:ac:b9:d9:17:ae is my internet router
Frame 1: 98 bytes on wire (784 bits), 98 bytes captured (784 bits)
Encapsulation type: Ethernet (1)
Arrival Time: Dec 11, 2020 21:20:44.110994000 CET
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1607718044.110994000 seconds
[Time delta from previous captured frame: 0.000000000 seconds]
[Time delta from previous displayed frame: 0.000000000 seconds]
[Time since reference or first frame: 0.000000000 seconds]
Frame Number: 1
Frame Length: 98 bytes (784 bits)
Capture Length: 98 bytes (784 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:ethertype:ip:icmp:data]
[Coloring Rule Name: ICMP]
[Coloring Rule String: icmp || icmpv6]
Ethernet II, Src: IskraTra_d0:9b:e2 (00:0e:c4:d0:9b:e2), Dst: 76:ac:b9:d9:17:ae (76:ac:b9:d9:17:ae)
Destination: 76:ac:b9:d9:17:ae (76:ac:b9:d9:17:ae)
Address: 76:ac:b9:d9:17:ae (76:ac:b9:d9:17:ae)
.... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Source: IskraTra_d0:9b:e2 (00:0e:c4:d0:9b:e2)
Address: IskraTra_d0:9b:e2 (00:0e:c4:d0:9b:e2)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 10.101.0.100, Dst: xxx.xxx.xxx.xxx.xxx
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
0000 00.. = Differentiated Services Codepoint: Default (0)
.... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
Total Length: 84
Identification: 0xd562 (54626)
Flags: 0x00
0... .... = Reserved bit: Not set
.0.. .... = Don't fragment: Not set
..0. .... = More fragments: Not set
Fragment Offset: 0
Time to Live: 63
Protocol: ICMP (1)
Header Checksum: 0x9b76 [validation disabled]
[Header checksum status: Unverified]
Source Address: 10.101.0.100
Destination Address: xxx.xxx.xxx.xxx.xxx
Internet Control Message Protocol
Type: 0 (Echo (ping) reply)
Code: 0
Checksum: 0x9e6b [correct]
[Checksum Status: Good]
Identifier (BE): 41567 (0xa25f)
Identifier (LE): 24482 (0x5fa2)
Sequence Number (BE): 178 (0x00b2)
Sequence Number (LE): 45568 (0xb200)
Timestamp from icmp data: Dec 11, 2020 21:20:44.106254000 CET
[Timestamp from icmp data (relative): 0.004740000 seconds]
PBR only handle packets through the firewall, not local initiated
Sorry if I wasn't clear.
None of the pings or locally initiated. "host" in the topic subject designates a Linux OS on the LAN.
(https://forum.opnsense.org/index.php?action=dlattach;topic=20417.0;attach=14030)
So you want to achieve a port-forward inside a VPN with already PBR in use?
In that case I'm redirection the whole a.b.c.d IP address to this Test Linux. It's a simple DNAT to route packers through VPN down to Linux. That part works and Test linux has a ping request coming from some public address.
It replies but the problem is OPNsense sends it down the default route instead of the VPN.
But when Test Linux pings some public IP, all is fine with PBR
Finally found someone with the same problem !
Apparently reply traffic always go through default route.
I might just configure a dedicated OPNsense for this use case which is fine.
https://forum.netgate.com/topic/149740/policy-based-routing-of-return-traffic/3
If you let me on the system I can have a look.
CEST during workdays in IRC
I had a strange situation like that on an OPNsense yesterday. For some reasons packets were not leaving the way they should. Ended up in doing a reboot because everything looked like it should be. Reboot then seems to have set the interfaces and routes correctly. It just worked.
Maybe it's worth a try.
It might be worth setting up a lab for this. I'll see if I can make some time but I have everything working on a separate OPNsense.
Another thread for that same issue https://forum.netgate.com/topic/146704/policy-routing-via-openvpn-uplink/3?_=1607793462725
Basically, "reply-to" that make traffic go back the interface it came in does not work for virtual interfaces and it's a known OpenBSD problem. It will always go back default route.
Maybe we can add a Nat option via ipfw?
What NAT would that be you think ?
I understand one of the solution that is often explained is to NAT on the other side, so the tunnel address is put in place of the actual source IP but that's not something I'm wanting to do becaause the service would then only see one IP address for all the users instead of actual clients, in this case, that's a mail server so it is important to have the actual client IP for anti spam.
Then you have to get rid of assigning OpenVPN interfaces?
If you're talking about assigning interface per VPN configuration (client or server) then no, it doesn't seem to be related. I have the same issue whether or not I use that.
In the final config with a dedicated OPNsense server I keep using it.
Them you have a faulty setup. Maybe screenshots would help
So the posts I found are inaccurate you think ?
Especially the one saying :
QuoteThis happens because pfsense is also not working option "reply-to" for virtual interfaces (e.g., openvpn,vti ...)
The traffic that came through the virtual tunnel will never return through the same gateway back.
The solution to this problem is to use outbound NAT for this traffic on the other end of the tunnel
The Link you quoted is only for Port forwards inside VPN. What you have is OPNsense as OpenVPN Client and OpenVPN server outside using as Site 2 Site. This is a common setup and works in both directions when correctly configured
I don't think my configuration is that common, and they do port forward like I do SNAT, exactly the same thing in the end.
The point is Site B of the VPN gets connections from internet through the tunnel, and replies *must* go through the same path, if it goes back Site B's internet it won't do.
FreeBSD has apparently that limitation that reply-to for virtual interfaces (VPNs) isn't functional so unless there is a route that send the packets through the VPN, it doesn't seem to be possible to achieve. And I cannot setup such route because the reply to address is any kind of public IP : it has to be the default route which means a dedicated OPNsense has to be setup.
This user has another shot at it :
Quote
[Internet] ----(WAN)---- [Firewall DC1]
|
|
(IPSEC VTI)
|
|
[Internet] ----(WAN)---- [Firewall DC2]
|
|
(Transit Network)
|
|
[Internal LAN] ------- [Web Servers]
Hi
This is a known PF problem, and it has been discussed here many times. Via virtual interfaces (VTI, GRE,OpenVPn,...) the function "Reply-to" does not work. Therefore, all external traffic will ALWAYS return through the default gateway DC2 (WAN interface DC2). To solve this problem you need to:
Use NAT Outbound on the interface VTI DC1 for all external traffic that is forwarded for the WEB server (DC2 side)
or change the default gateway to VTI DC1 (DC2 side)
I'm sorry if it looks my I'm trying to be the smart a**, I'm not, but I'm pretty confident now and that would be a lot of my time to recreate the configuration and it's just not worth it !