Messages

I don't think my configuration is that common, and they do port forward like I do SNAT, exactly the same thing in the end.

The point is Site B of the VPN gets connections from internet through the tunnel, and replies *must* go through the same path, if it goes back Site B's internet it won't do.

FreeBSD has apparently that limitation that reply-to for virtual interfaces (VPNs) isn't functional so unless there is a route that send the packets through the VPN, it doesn't seem to be possible to achieve. And I cannot setup such route because the reply to address is any kind of public IP : it has to be the default route which means a dedicated OPNsense has to be setup.

This user has another shot at it :

Code Select Expand


[Internet] ----(WAN)---- [Firewall DC1]
                      |
                      |
                 (IPSEC VTI)
                      |                       
                      |
[Internet] ----(WAN)---- [Firewall DC2]
                               |
                               |
                       (Transit Network)
                               |
                               |
                         [Internal LAN]   -------  [Web Servers]

Hi
This is a known PF problem, and it has been discussed here many times. Via virtual interfaces (VTI, GRE,OpenVPn,...) the function "Reply-to" does not work. Therefore, all external traffic will ALWAYS return through the default gateway DC2 (WAN interface DC2). To solve this problem you need to:
Use NAT Outbound on the interface VTI DC1 for all external traffic that is forwarded for the WEB server (DC2 side)
or change the default gateway to VTI DC1 (DC2 side)

I'm sorry if it looks my I'm trying to be the smart a**, I'm not, but I'm pretty confident now and that would be a lot of my time to recreate the configuration and it's just not worth it !

So the posts I found are inaccurate you think ?

Especially the one saying :

This happens because pfsense is also not working option "reply-to" for virtual interfaces (e.g., openvpn,vti ...)
The traffic that came through the virtual tunnel will never return through the same gateway back.
The solution to this problem is to use outbound NAT for this traffic on the other end of the tunnel

If you're talking about assigning interface per VPN configuration (client or server) then no, it doesn't seem to be related. I have the same issue whether or not I use that.

In the final config with a dedicated OPNsense server I keep using it.

What NAT would that be you think ?

I understand one of the solution that is often explained is to NAT on the other side, so the tunnel address is put in place of the actual source IP but that's not something I'm wanting to do becaause the service would then only see one IP address for all the users instead of actual clients, in this case, that's a mail server so it is important to have the actual client IP for anti spam.

Another thread for that same issue https://forum.netgate.com/topic/146704/policy-routing-via-openvpn-uplink/3?_=1607793462725

Basically, "reply-to" that make traffic go back the interface it came in does not work for virtual interfaces and it's a known OpenBSD problem. It will always go back default route.

It might be worth setting up a lab for this. I'll see if I can make some time but I have everything working on a separate OPNsense.

Finally found someone with the same problem !
Apparently reply traffic always go through default route.

I might just configure a dedicated OPNsense for this use case which is fine.

https://forum.netgate.com/topic/149740/policy-based-routing-of-return-traffic/3

In that case I'm redirection the whole a.b.c.d IP address to this Test Linux. It's a simple DNAT to route packers through VPN down to Linux. That part works and Test linux has a ping request coming from some public address.

It replies but the problem is OPNsense sends it down the default route instead of the VPN.

But when Test Linux pings some public IP, all is fine with PBR

Sorry if I wasn't clear.

None of the pings or locally initiated. "host" in the topic subject designates a Linux OS on the LAN.

That's the trace from OPNsense LAN interface, facing the machine that replies to ping

Code Select Expand


1 2020-12-11 21:18:18.753424 xxx.xxx.xxx.xxx.xxx 10.101.0.100 ICMP 98 Echo (ping) request  id=0xa25f, seq=33/8448, ttl=51 (reply in 2)
Ethernet II, Src: IskraTra_d0:9b:e3 (00:0e:c4:d0:9b:e3), Dst: VMware_a4:72:a2 (00:50:56:a4:72:a2)

2 2020-12-11 21:18:18.753621 10.101.0.100 xxx.xxx.xxx.xxx.xxx ICMP 98 Echo (ping) reply    id=0xa25f, seq=33/8448, ttl=64 (request in 1)
Ethernet II, Src: VMware_a4:72:a2 (00:50:56:a4:72:a2), Dst: IskraTra_d0:9b:e3 (00:0e:c4:d0:9b:e3)


That's full packet 2, the ping reply :

Code Select Expand


Frame 2: 98 bytes on wire (784 bits), 98 bytes captured (784 bits)
    Encapsulation type: Ethernet (1)
    Arrival Time: Dec 11, 2020 21:18:18.753621000 CET
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1607717898.753621000 seconds
    [Time delta from previous captured frame: 0.000197000 seconds]
    [Time delta from previous displayed frame: 0.000197000 seconds]
    [Time since reference or first frame: 0.000197000 seconds]
    Frame Number: 2
    Frame Length: 98 bytes (784 bits)
    Capture Length: 98 bytes (784 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:icmp:data]
    [Coloring Rule Name: ICMP]
    [Coloring Rule String: icmp || icmpv6]
Ethernet II, Src: VMware_a4:72:a2 (00:50:56:a4:72:a2), Dst: IskraTra_d0:9b:e3 (00:0e:c4:d0:9b:e3)
    Destination: IskraTra_d0:9b:e3 (00:0e:c4:d0:9b:e3)
        Address: IskraTra_d0:9b:e3 (00:0e:c4:d0:9b:e3)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: VMware_a4:72:a2 (00:50:56:a4:72:a2)
        Address: VMware_a4:72:a2 (00:50:56:a4:72:a2)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 10.101.0.100, Dst: xxx.xxx.xxx.xxx.xxx
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
        0000 00.. = Differentiated Services Codepoint: Default (0)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 84
    Identification: 0x9097 (37015)
    Flags: 0x00
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..0. .... = More fragments: Not set
    Fragment Offset: 0
    Time to Live: 64
    Protocol: ICMP (1)
    Header Checksum: 0xdf41 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 10.101.0.100
    Destination Address: xxx.xxx.xxx.xxx.xxx
Internet Control Message Protocol
    Type: 0 (Echo (ping) reply)
    Code: 0
    Checksum: 0xd1b5 [correct]
    [Checksum Status: Good]
    Identifier (BE): 41567 (0xa25f)
    Identifier (LE): 24482 (0x5fa2)
    Sequence Number (BE): 33 (0x0021)
    Sequence Number (LE): 8448 (0x2100)
    [Request frame: 1]
    [Response time: 0.197 ms]
    Timestamp from icmp data: Dec 11, 2020 21:18:18.748765000 CET
    [Timestamp from icmp data (relative): 0.004856000 seconds]


And for some reason, this packet ends up on the default route interface, even though I have a rule (reply-to disabled) that says anything "in" this interface (igb2) set Gateway to OpenVPN gateway :

76:ac:b9:d9:17:ae is my internet router

Code Select Expand


Frame 1: 98 bytes on wire (784 bits), 98 bytes captured (784 bits)
    Encapsulation type: Ethernet (1)
    Arrival Time: Dec 11, 2020 21:20:44.110994000 CET
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1607718044.110994000 seconds
    [Time delta from previous captured frame: 0.000000000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 0.000000000 seconds]
    Frame Number: 1
    Frame Length: 98 bytes (784 bits)
    Capture Length: 98 bytes (784 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:icmp:data]
    [Coloring Rule Name: ICMP]
    [Coloring Rule String: icmp || icmpv6]
Ethernet II, Src: IskraTra_d0:9b:e2 (00:0e:c4:d0:9b:e2), Dst: 76:ac:b9:d9:17:ae (76:ac:b9:d9:17:ae)
    Destination: 76:ac:b9:d9:17:ae (76:ac:b9:d9:17:ae)
        Address: 76:ac:b9:d9:17:ae (76:ac:b9:d9:17:ae)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: IskraTra_d0:9b:e2 (00:0e:c4:d0:9b:e2)
        Address: IskraTra_d0:9b:e2 (00:0e:c4:d0:9b:e2)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 10.101.0.100, Dst: xxx.xxx.xxx.xxx.xxx
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
        0000 00.. = Differentiated Services Codepoint: Default (0)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 84
    Identification: 0xd562 (54626)
    Flags: 0x00
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..0. .... = More fragments: Not set
    Fragment Offset: 0
    Time to Live: 63
    Protocol: ICMP (1)
    Header Checksum: 0x9b76 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 10.101.0.100
    Destination Address: xxx.xxx.xxx.xxx.xxx
Internet Control Message Protocol
    Type: 0 (Echo (ping) reply)
    Code: 0
    Checksum: 0x9e6b [correct]
    [Checksum Status: Good]
    Identifier (BE): 41567 (0xa25f)
    Identifier (LE): 24482 (0x5fa2)
    Sequence Number (BE): 178 (0x00b2)
    Sequence Number (LE): 45568 (0xb200)
    Timestamp from icmp data: Dec 11, 2020 21:20:44.106254000 CET
    [Timestamp from icmp data (relative): 0.004740000 seconds]


I'll get some traces and report

Thank you, the issue certainly looks similar.

But I think I disabled reply-to pretty much across the board.

I'm wondering if my setup can be called "multiple wan", as in some way I'm making another WAN through VPN...

Hi folks,

I'm pulling my hair out on this one.

I'm trying to route traffic from a specific network to a VPN gateway.

It works great for ping initiated from my host (ovpnc5 is the VPN interface I should go through):

Code Select Expand


tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ovpnc5, link-type NULL (BSD loopback), capture size 262144 bytes
08:54:14.110777 IP 10.101.0.100 > 1.1.1.1: ICMP echo request, id 68, seq 1, length 64
08:54:14.124537 IP 1.1.1.1 > 10.101.0.100: ICMP echo reply, id 68, seq 1, length 64


But for traffic the host replies to, it goes through igb2 and immediatly goes out igb1 instead of the matching ovpnc5 from the gateway rule.

Code Select Expand


# igb2 is the interface for the 10.101.0.100 network
root@opnsense:~ # tcpdump -i igb2 -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb2, link-type EN10MB (Ethernet), capture size 262144 bytes
08:45:35.525754 IP xxx.xxx.xxx.xxx.xxx > 10.101.0.100: ICMP echo request, id 50180, seq 220, length 64
08:45:35.525947 IP 10.101.0.100 > xxx.xxx.xxx.xxx.xxx: ICMP echo reply, id 50180, seq 220, length 64


Code Select Expand


# igb1 is the interface for the default route network
root@opnsense:~ # tcpdump -i igb1 -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 262144 bytes
08:46:30.715195 IP 10.101.0.100 > xxx.xxx.xxx.xxx.xxx: ICMP echo reply, id 50180, seq 275, length 64
08:46:31.719935 IP 10.101.0.100 > xxx.xxx.xxx.xxx.xxx: ICMP echo reply, id 50180, seq 276, length 64
08:46:32.723703 IP 10.101.0.100 > xxx.xxx.xxx.xxx.xxx: ICMP echo reply, id 50180, seq 277, length 64


Now the replies goes a different route and never works.

Does that makes sense ?

I have an extremely bizarre behaviour.

I configured VAN 101 of my switches, and created an interface on OPNsense
Configured DHCP server, all good.

Now, I added a rule for that VLAN interface "in" so that everything is ACCEPT, through the Gateway : (My VPN client interface).

I noticed that instead of listing the remote x.x.x.1 IP address, it lists the local x.x.x.2 IP address, seems odd to me.

Now to the bizarre part : anything I ping from the PC gets a ping reply from OPNsense itself. It could be 1.1.1.1 or any other IP, for some reason, OPNsense things the ping is for itself.

If I disable the rule that does the Gateway option, that behaviour stops.

OPNsense replying to any request (I described ping, but if I actually go to http://1.2.3.4/ i end up on OPNsense page.

What could cause a "catch-all" behavior ?

This is captured from OPNsense

Code Select Expand

23:25:42.513942 IP 10.101.0.100 > 1.1.1.1: ICMP echo request, id 29, seq 53, length 64
23:25:42.513976 IP 10.101.0.1 > 10.101.0.100: ICMP echo reply, id 29, seq 53, length 64


Now I also noticed that if on the VPN server I push push "redirect-gateway def1" then :
- The behaviour stops (but packets go through default interface instead of VPN
- Now I get the .1 IP address in the Gateway menu.

Thanks for your help !

OPNsense 20.7.5-amd64