[SOLVED] Issue with nat/portforward from openvpn connection since the upgrade

[xupetas]

Hello all,

I have this configuration, that was working before the upgrade:

PIA --> opnsense (openvpn) --> host.

PIA forwards a port into opnsense via a openvpn interface that get's natted (portforwared) to host.
Since the upgrade, i lost the ability to do a tcp connection from outside the PIA, thru opnsense, to the host on the port that it specified. I am able to exit the host via the pia tunnel without any issue

However i see traffic inside the host as coming from PIA, so it appears that something is working. Just not able to to a proper tcp connection /syn/synack.
There are rules allowing that all traffic from pia reach the host.

This is an extract of my unfiltered tcpdump. Code is reaching and appears to be leaving the host, thru openvpn into pia:

Code: [Select]

07:28:32.892295 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.892326 IP 172.16.3.7.documentum > host-95-237-116-82.retail.telecomitalia.it.61700: UDP, length 20
07:28:32.893405 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.893455 IP 172.16.3.7.documentum > host-95-237-116-82.retail.telecomitalia.it.61700: UDP, length 20
07:28:32.893469 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.893499 IP 172.16.3.7.documentum > host-95-237-116-82.retail.telecomitalia.it.61700: UDP, length 20
07:28:32.894146 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.894180 IP 172.16.3.7.documentum > host-95-237-116-82.retail.telecomitalia.it.61700: UDP, length 20
07:28:32.894192 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.894216 IP 172.16.3.7.documentum > host-95-237-116-82.retail.telecomitalia.it.61700: UDP, length 20
07:28:32.894420 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.894451 IP 172.16.3.7.documentum > host-95-237-116-82.retail.telecomitalia.it.61700: UDP, length 20
07:28:32.894613 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.894632 IP 172.16.3.7.documentum > host-95-237-116-82.retail.telecomitalia.it.61700: UDP, length 20
07:28:32.895104 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.895126 IP 172.16.3.7.documentum > host-95-237-116-82.retail.telecomitalia.it.61700: UDP, length 20
07:28:32.897229 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.897284 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.897285 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.897370 IP 172.16.3.7.documentum > host-95-237-116-82.retail.telecomitalia.it.61700: UDP, length 20
07:28:32.897938 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.897993 IP 172.16.3.7.documentum > host-95-237-116-82.retail.telecomitalia.it.61700: UDP, length 20
07:28:32.898007 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.898040 IP 172.16.3.7.documentum > host-95-237-116-82.retail.telecomitalia.it.61700: UDP, length 20
07:28:32.899480 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.899526 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.899602 IP 172.16.3.7.documentum > host-95-237-116-82.retail.telecomitalia.it.61700: UDP, length 20
07:28:32.900198 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.900239 IP 172.16.3.7.documentum > host-95-237-116-82.retail.telecomitalia.it.61700: UDP, length 20
07:28:32.904513 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.904567 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.904615 IP host-95-237-116-82.retail.telecomitalia.it.61700 > 172.16.3.7.documentum: UDP, length 849
07:28:32.904627 IP 172.16.3.7.documentum > host-95-237-116-82.retail.telecomitalia.it.61700: UDP, length 20
07:28:32.904692 IP 172.16.3.7.documentum > host-95-237-116-82.retail.telecomitalia.it.61700: UDP, length 20


This is an extract from a tcpdump from a tcp connection connecting from a public ip, into the front of the PIA vpn endpoint, on the port specified. It does reach the host vm, but is unable to do a proper tcp connection, and yes the port on the destination is listening and there is no local firewall on that particular host

Code: [Select]


tcpdump: listening on veth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
07:29:50.612214 IP (tos 0x0, ttl 54, id 11967, offset 0, flags [DF], proto TCP (6), length 60)
    bing.unammed.isp.telecom.7960 > 172.16.3.7.documentum-s: Flags [S], cksum 0x00df (correct), seq 4120766767, win 64240, options [mss 1238,sackOK,TS val 205320036 ecr 0,nop,wscale 7], length 0
07:29:50.612456 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.3.7.documentum-s > bing.unammed.isp.telecom.7960: Flags [S.], cksum 0x2ca3 (incorrect -> 0xe663), seq 2016981066, ack 4120766768, win 65160, options [mss 1460,sackOK,TS val 46328494 ecr 205320036,nop,wscale 7], length 0
07:29:51.621210 IP (tos 0x0, ttl 54, id 11968, offset 0, flags [DF], proto TCP (6), length 60)
    bing.unammed.isp.telecom.7960 > 172.16.3.7.documentum-s: Flags [S], cksum 0xfcf4 (correct), seq 4120766767, win 64240, options [mss 1238,sackOK,TS val 205321038 ecr 0,nop,wscale 7], length 0
07:29:51.621271 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.3.7.documentum-s > bing.unammed.isp.telecom.7960: Flags [S.], cksum 0x2ca3 (incorrect -> 0xe272), seq 2016981066, ack 4120766768, win 65160, options [mss 1460,sackOK,TS val 46329503 ecr 205320036,nop,wscale 7], length 0
07:29:52.630136 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.3.7.documentum-s > bing.unammed.isp.telecom.7960: Flags [S.], cksum 0x2ca3 (incorrect -> 0xde81), seq 2016981066, ack 4120766768, win 65160, options [mss 1460,sackOK,TS val 46330512 ecr 205320036,nop,wscale 7], length 0
07:29:53.646538 IP (tos 0x0, ttl 54, id 11969, offset 0, flags [DF], proto TCP (6), length 60)
    bing.unammed.isp.telecom.7960 > 172.16.3.7.documentum-s: Flags [S], cksum 0xf514 (correct), seq 4120766767, win 64240, options [mss 1238,sackOK,TS val 205323054 ecr 0,nop,wscale 7], length 0
07:29:53.646565 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.3.7.documentum-s > bing.unammed.isp.telecom.7960: Flags [S.], cksum 0x2ca3 (incorrect -> 0xda89), seq 2016981066, ack 4120766768, win 65160, options [mss 1460,sackOK,TS val 46331528 ecr 205320036,nop,wscale 7], length 0
07:29:55.670131 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.3.7.documentum-s > bing.unammed.isp.telecom.7960: Flags [S.], cksum 0x2ca3 (incorrect -> 0xd2a1), seq 2016981066, ack 4120766768, win 65160, options [mss 1460,sackOK,TS val 46333552 ecr 205320036,nop,wscale 7], length 0
07:29:57.771595 IP (tos 0x0, ttl 54, id 11970, offset 0, flags [DF], proto TCP (6), length 60)
    bing.unammed.isp.telecom.7960 > 172.16.3.7.documentum-s: Flags [S], cksum 0xe4f4 (correct), seq 4120766767, win 64240, options [mss 1238,sackOK,TS val 205327182 ecr 0,nop,wscale 7], length 0
07:29:57.771636 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.3.7.documentum-s > bing.unammed.isp.telecom.7960: Flags [S.], cksum 0x2ca3 (incorrect -> 0xca6c), seq 2016981066, ack 4120766768, win 65160, options [mss 1460,sackOK,TS val 46335653 ecr 205320036,nop,wscale 7], length 0
07:30:01.910112 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.3.7.documentum-s > bing.unammed.isp.telecom.7960: Flags [S.], cksum 0x2ca3 (incorrect -> 0xba41), seq 2016981066, ack 4120766768, win 65160, options [mss 1460,sackOK,TS val 46339792 ecr 205320036,nop,wscale 7], length 0
 


What is wrong with this picture?
Was there any change that needs to be done to the openvpn client to allow this configuration? Or any mandatory new configuration on opnsense's interface to allow this again?

Thanks for your help

[xupetas]

I think that its openvpn (binary) related. Tried the same configuration using wireguard and no issues were found.
I have been reading the changelog of the product and i do not see any breaking changes regarding hairpinning on the alerts... is anyone aware?

[franco]

You haven't provided a version number reference of good and bad version. Impossible to guess.


Cheers,
Franco

[xupetas]

Hi Franco,

This started with the latest production version of opnsense:    OPNsense 23.1.11-amd64
The versions are the ones bundled with that:

openvpn:

OpenVPN 2.6.5 amd64-portbld-freebsd13.1 [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [MH/RECVDA] [AEAD]
library versions: OpenSSL 1.1.1u  30 May 2023, LZO 2.10

[franco]

Thanks, and before you were using which version?


Cheers,
Franco

[xupetas]

Hi

I was running version 23.1.8.

Thanks

[xupetas]

Hi @Franco.

Any ideas on where the problem might me?

Kindly,
Nuno

[franco]

Hi Nuno,

Not sure but I think it has to do with the OpenVPN 2.6 update. There was a bit of discussion and workarounds for this topic on the forum.


Cheers,
Franco

[xupetas]

Hi Franco

I think that there is a mix between the openvpn configuration and something nat related with bsd itself of even with what is building the rules on the background and then feeding them into pf. I have now noticed that i have the a similar issue with another wireguard connection (so no openvpn here).

HTTP request nated ip --> internal ip (sinkhole) --> porfrwrd to transparent SQUID --> Wireguard --> destination.

I am getting many RST on the portfrwd part. And the request never reaches the entrance the wireguard tunnel. The corruption appears to be happening in the NAT part of this configuration.

I have also noticed that there are reports on github regarding this:

https://github.com/opnsense/core/issues/6662
https://github.com/opnsense/core/issues/6650

Do you know if this is a bug and will be fixed on the next release?
Also, asking a dumb question, i am having issues understanding regarding the workaround proposed. What needs to be done with the snat configuration. Do you have anything (or know a post) where there is a picture of an snat example configuration?

I am really struggling here to understand how this solve my issue.
I presently have this enabled on firewall advanced settings:

Use sticky connections
Use shared forwarding between packet filter, traffic shaper and captive portal


And all the return ip paths from the portfwrds, have an outbound rule from the destination (in this case the vm) via the interface from where the requests originated and nated with the ip of said interface

Thanks for your help!

Edit: been testing some different configurations on advanced settings, and i want my text to reflect what i have presently

[xupetas]

Hi @Franco.

Installed a new vanilla opnsense and the results are the same.

I see the package coming thru from the openvpn, passing the firewall, reaching the VM, getting returned to the firewall but not leaving the FW via the openvpn.

One thing i did not remember saying before, but the VM that is the recipient of the portfwrd has the gw forcefully set on opnsense to the gateway provided by the openvpn and i am reaching the internet with the exit IP from that openvpn connection.

Will continue to dig on this.

Thanks for your help

[xupetas]

Solved my issue:

I reconfigured my openvpn client like this:



My redirection started working again as expected.

@Franco, do you have any idea on why is that? What does that flag do that breaks RDR?

Thanks so much for your help

[franco]

Sorry, I don't know. It could have still been an incompatibility with OpenVPN 2.6.


Cheers,
Franco

[funfuck1337]

@xupetas, I have encountered the same problem as you after creating an OVPN server. Port reflection is not working when a user is connected to OVPN. However, the workaround is working like a charm. Thanks for the information.

The 'do not add/remove route' option in the new instances setup method is under the 'Miscellaneous' section > 'Option' > 'route-noexec'.