IPSec VPN seems to work, but no communication?

[ibrewster]

I'm trying to set up a IPsec VPN for my remote laptop (mobile client). I chose IPsec because it is built-in to both OPNSense and MacOS, so no third-party or add-on software is needed on either end.

I configured everything following this guide: https://docs.opnsense.org/manual/how-tos/ipsec-rw-srv-ikev1xauth.html, and as far as I can tell all settings - other than IP addresses - are identical to what is shown there.

My LAN has a subnet of 10.27.81.0/24, and I gave the IPSec clients an address pool of 10.27.82.0/24.

From my laptop I can connect to the VPN without difficulty, and it says it gets an IP of 10.27.82.1, which is as expected.

The OPNsense screens all seem to indicate a successful connection, and when I try to ping or SSH a host on the inside network, the firewall log seems to indicate the packets are being passed (I was going to provide screen shots, but can't figure out how). However, nothing works: no ping responses, SSH can't connect, I can't get a web page, etc.

So the OPNsense seems to indicate everything is working fine, and the remote client seems to indicate everything is working fine, but no traffic actually passes.

What am I missing here? What diagnostics can I perform/what logs can I look at to figure out why it is not working?

[Monviech (Cedrik)]

Have you created a firewall rule in "IPsec" that allows traffic to your LAN?

[ibrewster]

Yes. Not only was that in the guide I followed, but as I mentioned the firewall shows packets being allowed, both in and out, when attempting to ping, ssh, or load a webpage over the VPN.

It sort of looks like the VPN is getting packets from the remote, routing them properly to the destination, and then routing the destination back to the LAN, but then loosing the packet. Perhaps a NAT issue?

[Monviech (Cedrik)]

Hey I just try to help. I really didn't see a focus on firewall rules in the doc you linked.

Could you verify your firewall and nat rules with the document I wrote? I have an extra section in my guide, and I know that it works since I use it myself and with customers.

https://docs.opnsense.org/manual/how-tos/ipsec-swanctl-rw-ikev2-eap-mschapv2.html

[ibrewster]

Hey I just try to help. I really didn't see a focus on firewall rules in the doc you linked.

Sorry about that. This whole situation is frustrating me, and I kinda took it out on you there. That was inappropriate of me. Please accept my apologies.

Could you verify your firewall and nat rules with the document I wrote? I have an extra section in my guide, and I know that it works since I use it myself and with customers.

https://docs.opnsense.org/manual/how-tos/ipsec-swanctl-rw-ikev2-eap-mschapv2.html

Yeah, I actually spent about an hour this morning ripping out the Legacy settings and following that guide you just linked to - specifically the "static IP address per roadwarrior" method, using a lets encrypt certificate rather than a self-signed one, and following the iOS native VPN client setup (though on my MacOS Sonoma install, it *appeared* to be identical). Unfortunately I couldn't even get that one to connect - the logs just kept saying NO PROPOSAL CHOSEN or something similar. So I put things back to the "legacy" connection where at least I can get connected.

Regarding firewall rules, I have the following on the WAN interface:

IPv4+6 ESP   *   *   WAN address   *   *   *      IPsec ESP
IPv4+6 UDP   *   *   WAN address   ipsec_ports    *   *      Allow IPsec UDP ports to this firewall

and on the IPsec Interface I have this:

IPv4+6 *   *   *   LAN net   *   *   *      Allow access to LAN net

I have also tried setting that LAN net in the IPsec rule to "any".

Under NAT, I don't have anything specific to the IPsec interface, though the IPsec subnet is listed in the Auto created rule on the WAN interface.

EDIT: I also realized that I read the firewall log wrong. When I attempt to, for example, SSH to a machine on the LAN, I see the traffic go out the IPsec interface and IN the LAN interface, as expected. I *don't* see any return traffic, at least not with a filter of "address" and the IPsec net.

[Monviech (Cedrik)]

I understand that ipsec gets very frustrating and I accept your apology.

What you write is totally unexpected and should not happen. Is there something that makes your setup more complicated than the norm, maybe a HA setup?

Is your outer tunnel address IPv4 or IPv6?

Do you SSH to an internal IP address or do you use a hostname that resolves to an external IP and expect NAT Reflection to happen?

[ibrewster]

Is there something that makes your setup more complicated than the norm, maybe a HA setup?

It is quite possible I caused the issue by turning on some feature I thought "sounded neat" or the like, not realizing the implications, however the overall setup is pretty basic: single router with two physical ports WAN and LAN, cable modem on the WAN, switch on the LAN. Nothing special or fancy there. I am using DNSBL on unbound, with a firewall rule set up to force all DNS queries through the local Unbound, but unless I *really* messed that up it should have no effect. The only plugins I am using are for LetsEncrypt and Dynamic DNS. System is fully updated (last checked this morning).

Is your outer tunnel address IPv4 or IPv6?

IPv4. Apparently my ISP doesn't offer IPv6 (at least, not to me), though I do have IPv6 support enabled with the idea that if/when it is offered I could use it. Could that be confusing things?

Do you SSH to an internal IP address or do you use a hostname that resolves to an external IP and expect NAT Reflection to happen?

Using IP addresses at the moment. My hope is that the VPN tunnel will push the local DNS server to the client, so internal addresses will resolve when connected to VPN, but since I can't get to even the LAN address of the router at the moment, that's obviously not working, so I'm just using IP addresses.

[Monviech (Cedrik)]

Please post the output of
"pfctl -s rules" and
"pfctl -s nat".

Your routing table on the opnsens:
"netstat -r"

Then post the output of
"swanctl --list-sas" while your client is connected. And you have already pinged a few times.

Also post "/var/log/ipsec/latest.log" the snippet where your client connects and the first few ike keepalives after it.

I also want the output of tcpdump while your client pings an internal IP address in your LAN. Try to make sure the icmp echo has the same ID so tcpdump in all shells at the same time.
tcpdump -i enc0 proto ICMP -n
tcpdump -i lan proto ICMP -n

On the client you try to SSH please also tcpdump, if it's linux:
tcpdump -i any proto ICMP -n

If you need to anonymize public IPs, please use the IPv4 test address range of IANA 203.0.113.0/24.

Please post it all as text and not as screenshots. Thank you~

[ibrewster]

Thanks for those tips. This...makes no sense to me. the TCP dumps seems to show both the request and reply flowing smoothly, both through my router and through the local hosts I am pinging (see below). It is just the remote host that doesn't seem to be getting the memo. Perhaps interestingly (perhaps expectedly) running a `tcpdump -i any proto ICMP -n` on the remote host shows nothing whatsoever.

Do you see anything I am missing here? Because to me it looks like it should be working...

Latest log attached in zip form so it will be small enough for an attachment, rest of the commands below (hopefully in a readable format)

Remote host IP: 137.229.32.97
Local host IP: 10.27.81.60

Code Select Expand


pfctl -s rules

scrub on em0 all fragment reassemble
scrub on em1 all fragment reassemble
block drop in log on ! em0 inet from 10.27.81.0/24 to any
block drop in log inet from 10.27.81.1 to any
block drop in log on ! em1 inet from 69.178.80.0/20 to any
block drop in log inet from 69.178.88.174 to any
block drop in log on em0 inet6 from fe80::222:4dff:fe9d:7b76 to any
block drop in log on em1 inet6 from fe80::222:4dff:fe9d:7b72 to any
block drop in log inet all label "02f4bab031b57d1e30553ce08e0ec131"
block drop in log inet6 all label "02f4bab031b57d1e30553ce08e0ec131"
pass in log quick inet6 proto ipv6-icmp all icmp6-type unreach keep state label "1d245529367b2e34eeaff16086aeafe9"
pass in log quick inet6 proto ipv6-icmp all icmp6-type toobig keep state label "1d245529367b2e34eeaff16086aeafe9"
pass in log quick inet6 proto ipv6-icmp all icmp6-type neighbrsol keep state label "1d245529367b2e34eeaff16086aeafe9"
pass in log quick inet6 proto ipv6-icmp all icmp6-type neighbradv keep state label "1d245529367b2e34eeaff16086aeafe9"
pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type echoreq keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type echoreq keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type echorep keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type echorep keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type routersol keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type routersol keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type routeradv keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type routeradv keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type neighbrsol keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type neighbrsol keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type neighbradv keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type neighbradv keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type echoreq keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type echoreq keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routersol keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routersol keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routeradv keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routeradv keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type neighbrsol keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type neighbrsol keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type neighbradv keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type neighbradv keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type echoreq keep state label "8752fca75c6be992847ea984161bd3f1"
pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type routersol keep state label "8752fca75c6be992847ea984161bd3f1"
pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type routeradv keep state label "8752fca75c6be992847ea984161bd3f1"
pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type neighbrsol keep state label "8752fca75c6be992847ea984161bd3f1"
pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type neighbradv keep state label "8752fca75c6be992847ea984161bd3f1"
pass in log quick inet6 proto ipv6-icmp from :: to ff02::/16 icmp6-type echoreq keep state label "71dd196398b3f1da265dbd9dcad00e70"
pass in log quick inet6 proto ipv6-icmp from :: to ff02::/16 icmp6-type routersol keep state label "71dd196398b3f1da265dbd9dcad00e70"
pass in log quick inet6 proto ipv6-icmp from :: to ff02::/16 icmp6-type routeradv keep state label "71dd196398b3f1da265dbd9dcad00e70"
pass in log quick inet6 proto ipv6-icmp from :: to ff02::/16 icmp6-type neighbrsol keep state label "71dd196398b3f1da265dbd9dcad00e70"
pass in log quick inet6 proto ipv6-icmp from :: to ff02::/16 icmp6-type neighbradv keep state label "71dd196398b3f1da265dbd9dcad00e70"
block drop in log quick inet proto tcp from any port = 0 to any label "7b5bdc64d7ae74be1932f6764a591da5"
block drop in log quick inet proto udp from any port = 0 to any label "7b5bdc64d7ae74be1932f6764a591da5"
block drop in log quick inet6 proto tcp from any port = 0 to any label "7b5bdc64d7ae74be1932f6764a591da5"
block drop in log quick inet6 proto udp from any port = 0 to any label "7b5bdc64d7ae74be1932f6764a591da5"
block drop in log quick inet proto tcp from any to any port = 0 label "ae69f581dc429e3484a65f8ecd63baa5"
block drop in log quick inet proto udp from any to any port = 0 label "ae69f581dc429e3484a65f8ecd63baa5"
block drop in log quick inet6 proto tcp from any to any port = 0 label "ae69f581dc429e3484a65f8ecd63baa5"
block drop in log quick inet6 proto udp from any to any port = 0 label "ae69f581dc429e3484a65f8ecd63baa5"
pass log quick inet6 proto carp from any to ff02::12 keep state label "cf439d72ef4d245e8ad4a1405df1f665"
pass log quick inet proto carp from any to 224.0.0.18 keep state label "2ffa978d51f7b3fbc9000c2895106ee7"
block drop in log quick proto tcp from <sshlockout> to (self) port = ssh label "669143f420c3ab4118bcb0bf4b5fd823"
block drop in log quick proto tcp from <sshlockout> to (self) port = https label "6baefc2a9cf2536834c092a51134a45c"
block drop in log quick from <virusprot> to any label "8e367e2f9944d93137ae56d788c5d5e1"
pass in log quick on em0 inet proto udp from any port = bootpc to 255.255.255.255 port = bootps keep state label "5168be2cca1e130b1ef2ac18161356a8"
pass in log quick on em0 proto udp from any port = bootpc to (self) port = bootps keep state label "0b032d1bab91fc97e4a7faf03a7f17c3"
pass out log quick on em0 proto udp from (self) port = bootps to any port = bootpc keep state label "5039e43005a9aa50eb032af274cc9aad"
pass in log quick on em0 inet6 proto udp from fe80::/10 to fe80::/10 port = dhcpv6-client keep state label "fef3d333d96a8d3558956de1fffc61cc"
pass in log quick on em0 inet6 proto udp from fe80::/10 to ff02::/16 port = dhcpv6-client keep state label "fef3d333d96a8d3558956de1fffc61cc"
pass in log quick on em0 inet6 proto udp from fe80::/10 to ff02::/16 port = dhcpv6-server keep state label "d2bd536587a9f5680c1f850b2d346839"
pass in log quick on em0 inet6 proto udp from ff02::/16 to fe80::/10 port = dhcpv6-server keep state label "3420206ced96c01ef73fbc4ac9deb745"
pass in log quick on em0 inet6 proto udp from fe80::/10 to (self) port = dhcpv6-client keep state label "0fd202708c326aebbe44ab710b6d3652"
pass out log quick on em0 inet6 proto udp from (self) port = dhcpv6-server to fe80::/10 keep state label "83f6c28de8efae9b444094e4a5bf898c"
pass in log quick on em1 inet6 proto udp from fe80::/10 port = dhcpv6-client to fe80::/10 port = dhcpv6-client keep state label "a6cd2cce1bc1d912f6258ef1f3fb07e1"
pass in log quick on em1 proto udp from any port = dhcpv6-server to any port = dhcpv6-client keep state label "f7e4334c3e7dc4ba900c5780b828d4a3"
pass out log quick on em1 proto udp from any port = dhcpv6-client to any port = dhcpv6-server keep state label "5ba1258fcaf073eff4060b40ff63044d"
pass in log quick on em1 proto udp from any port = bootps to any port = bootpc keep state label "f994f615e00b8be0042263f86c79913f"
pass out log quick on em1 proto udp from any port = bootpc to any port = bootps keep state label "5cf7ab808da1fcbca1ddb9ba9b46b669"
block drop in log quick on em1 inet from <bogons> to any label "b7cd97a164650b538506fb551a0369e7"
block drop in log quick on em1 inet6 from <bogonsv6> to any label "f140a48ddade668b9d6f5259669a1d5c"
pass in quick on lo0 all no state label "7535c94082e72e2207679aadb26afd92"
pass out log all flags S/SA keep state allow-opts label "fae559338f65e11c53669fc3642c93c2"
pass out log on enc0 all flags S/SA keep state label "c1eff64cbafdd6b80448f92cd4aff7e5"
pass in log quick on em0 proto tcp from any to (self) port = ssh flags S/SA keep state label "bb72618316fdf630cdf15f33ae3d699f"
pass in log quick on em0 proto tcp from any to (self) port = http flags S/SA keep state label "bb72618316fdf630cdf15f33ae3d699f"
pass in log quick on em0 proto tcp from any to (self) port = https flags S/SA keep state label "bb72618316fdf630cdf15f33ae3d699f"
pass out log route-to (em1 69.178.80.1) inet from (em1) to ! (em1:network) flags S/SA keep state allow-opts label "ae4923441933901a90e7d9997f83b9f6"
pass in log quick on enc0 inet all flags S/SA keep state label "e3742b5a34e16197f4feafb15b3c6264"
pass in log quick on enc0 inet6 all flags S/SA keep state label "e3742b5a34e16197f4feafb15b3c6264"
pass in quick on openvpn inet all flags S/SA keep state label "f9bd0646952d6451270a554876a7fc2f"
pass in log quick on em1 reply-to (em1 69.178.80.1) inet proto esp from any to (em1) keep state label "21df42ed9a2f63199a7b34caf3b368ce"
pass in log quick on em1 inet6 proto esp from any to (em1) keep state label "21df42ed9a2f63199a7b34caf3b368ce"
pass in quick on em1 reply-to (em1 69.178.80.1) inet proto udp from any to (em1) port = isakmp keep state label "6db69ea380b5d3568fff18d9f2613591"
pass in quick on em1 reply-to (em1 69.178.80.1) inet proto udp from any to (em1) port = ipsec-nat-t keep state label "6db69ea380b5d3568fff18d9f2613591"
pass in quick on em1 inet6 proto udp from any to (em1) port = isakmp keep state label "6db69ea380b5d3568fff18d9f2613591"
pass in quick on em1 inet6 proto udp from any to (em1) port = ipsec-nat-t keep state label "6db69ea380b5d3568fff18d9f2613591"
pass in quick on em1 reply-to (em1 69.178.80.1) inet proto icmp from any to (em1) keep state label "814aa8aefcc1293a6d38e5fb227d4f66"
pass in quick on em1 inet6 proto ipv6-icmp from any to (em1) keep state label "814aa8aefcc1293a6d38e5fb227d4f66"
pass in quick on em1 reply-to (em1 69.178.80.1) inet proto tcp from any to 10.27.81.60 port = imaps flags S/SA keep state label "fe9c419bba801dc8ae9171b46d403a70"
pass in quick on em1 reply-to (em1 69.178.80.1) inet proto tcp from any to 10.27.81.60 port = http flags S/SA keep state label "639d2f3b29418248181069c17215d193"
pass in quick on em1 reply-to (em1 69.178.80.1) inet proto tcp from any to 10.27.81.60 port = smtp flags S/SA keep state label "c8715002cd9f2473b9d5b2a6f1bd932a"
pass in quick on em1 reply-to (em1 69.178.80.1) inet proto tcp from any to 10.27.81.60 port = imap flags S/SA keep state label "95c26eef2e3e4b10ab24410b49607a7f"
pass in quick on em1 reply-to (em1 69.178.80.1) inet proto tcp from any to 10.27.81.60 port = smtps flags S/SA keep state label "f567acd4ea8dd59eee904a15e2179395"
pass in quick on em1 reply-to (em1 69.178.80.1) inet proto udp from any to 10.27.81.60 port = openvpn keep state label "4ba2235140039a651483e748534ee3a1"
pass in quick on em1 reply-to (em1 69.178.80.1) inet proto tcp from any to 10.27.81.60 port = 943 flags S/SA keep state label "d972c26bcf562b5f300bf0eb8df65d02"
pass in log quick on em1 reply-to (em1 69.178.80.1) inet proto tcp from any to 10.27.81.60 port = https flags S/SA keep state label "d730ade7f168ca2b6f67ed20cf3f48bc"
pass in log quick on em0 inet proto tcp from any to 127.0.0.1 port = domain flags S/SA keep state label "47c04110e1336fa111ad5190bcd73b41"
pass in log quick on em0 inet proto udp from any to 127.0.0.1 port = domain keep state label "47c04110e1336fa111ad5190bcd73b41"
pass in quick on em0 inet from (em0:network) to any flags S/SA keep state label "4bbc875dd7dcb4ad450d0f9d0f91b692"
pass in quick on em0 inet6 from (em0:network) to any flags S/SA keep state label "133a07584c296ef77d9a309f604315ea"
pass in quick on em0 inet6 from fe80::/10 to any flags S/SA keep state label "133a07584c296ef77d9a309f604315ea"
pass out log on em1 proto udp from any to any port = isakmp keep state label "022363a8f0ee1cceef8d18eeef536433"
pass in log on em1 reply-to (em1 69.178.80.1) inet proto udp from any to any port = isakmp keep state label "3dd371c341d2544829ba5926b472378c"
pass out log on em1 proto udp from any to any port = ipsec-nat-t keep state label "bb9ef419eca4a7ddbcce91fc7054fc38"
pass in log on em1 reply-to (em1 69.178.80.1) inet proto udp from any to any port = ipsec-nat-t keep state label "b1a75ea5c4b603f2889008a64e2c1149"
pass out log on em1 proto esp all keep state label "a247580377c9b6716c340d6b4a471b55"
pass in log on em1 reply-to (em1 69.178.80.1) inet proto esp all keep state label "a4a2bf39bff388e10337dafeee3828e0"
anchor "acme-client/*" all


pfctl -s nat

no nat proto carp all
nat log on em1 inet from (em0:network) to any port = isakmp -> (em1:0) static-port
nat log on em1 inet from (lo0:network) to any port = isakmp -> (em1:0) static-port
nat log on em1 inet from 127.0.0.0/8 to any port = isakmp -> (em1:0) static-port
nat log on em1 inet from 10.27.84.0/24 to any port = isakmp -> (em1:0) static-port
nat log on em1 inet from (em0:network) to any -> (em1:0) port 1024:65535
nat log on em1 inet from (lo0:network) to any -> (em1:0) port 1024:65535
nat log on em1 inet from 127.0.0.0/8 to any -> (em1:0) port 1024:65535
nat log on em1 inet from 10.27.84.0/24 to any -> (em1:0) port 1024:65535
nat-anchor "acme-client/*" all
no rdr proto carp all
no rdr on em0 proto tcp from any to (em0) port = ssh
no rdr on em0 proto tcp from any to (em0) port = http
no rdr on em0 proto tcp from any to (em0) port = https
rdr on em1 inet proto tcp from any to (em1) port = http -> 10.27.81.60 port 80
rdr on em0 inet proto tcp from any to (em1) port = http -> 10.27.81.60 port 80
rdr on lo0 inet proto tcp from any to (em1) port = http -> 10.27.81.60 port 80
rdr log on em1 inet proto tcp from any to (em1) port = https -> 10.27.81.60 port 443
rdr on em1 inet proto tcp from any to (em1) port = smtp -> 10.27.81.60 port 25
rdr on em0 inet proto tcp from any to (em1) port = smtp -> 10.27.81.60 port 25
rdr on lo0 inet proto tcp from any to (em1) port = smtp -> 10.27.81.60 port 25
rdr on em1 inet proto tcp from any to (em1) port = imap -> 10.27.81.60 port 143
rdr on em0 inet proto tcp from any to (em1) port = imap -> 10.27.81.60 port 143
rdr on lo0 inet proto tcp from any to (em1) port = imap -> 10.27.81.60 port 143
rdr on em1 inet proto tcp from any to (em1) port = imaps -> 10.27.81.60 port 993
rdr on em1 inet proto tcp from any to (em1) port = smtps -> 10.27.81.60 port 465
rdr on em0 inet proto tcp from any to (em1) port = smtps -> 10.27.81.60 port 465
rdr on lo0 inet proto tcp from any to (em1) port = smtps -> 10.27.81.60 port 465
rdr on em1 inet proto udp from any to (em1) port = openvpn -> 10.27.81.60 port 1194
rdr on em1 inet proto tcp from any to (em1) port = 943 -> 10.27.81.60 port 943
rdr log on em0 inet proto tcp from any to ! (em0) port = domain -> 127.0.0.1 port 53
rdr log on em0 inet proto udp from any to ! (em0) port = domain -> 127.0.0.1 port 53
rdr-anchor "acme-client/*" all


Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            1-80-178-69.gci.ne UGS         em1
10.27.81.0/24      link#1             U           em0
gatekeeper         link#1             UHS         lo0
69.178.80.0/20     link#2             U           em1
174-88-178-69.gci. link#2             UHS         lo0
localhost          link#4             UH          lo0

Internet6:
Destination        Gateway            Flags     Netif Expire
localhost          link#4             UHS         lo0
fe80::%em0/64      link#1             U           em0
fe80::222:4dff:fe9 link#1             UHS         lo0
fe80::%em1/64      link#2             U           em1
fe80::222:4dff:fe9 link#2             UHS         lo0
fe80::%lo0/64      link#4             U           lo0
fe80::1%lo0        link#4             UHS         lo0


swanctl --list-sas

no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
con1: #15, ESTABLISHED, IKEv1, 7b75fd54d458c5f7_i 4be91184ab849282_r*
  local  'gatekeeper.brewstersoft.net' @ 69.178.88.174[500]
  remote '137.229.32.97' @ 137.229.32.97[500] XAuth: 'israel' [10.27.84.1]
  AES_CBC-256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
  established 248s ago, rekeying in 13354s
  con1: #12, reqid 1, INSTALLED, TUNNEL, ESP:AES_CBC-256/HMAC_SHA1_96
    installed 247s ago, rekeying in 2751s, expires in 3353s
    in  c8b17443,   3208 bytes,    40 packets,     0s ago
    out 0334cb6e,   7352 bytes,    45 packets,     0s ago
    local  10.27.81.0/24
    remote 10.27.84.1/32
   
TCP Dump host on LAN (destination client) during ping:

root@watchman:~# tcpdump -i br0 proto ICMP -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on br0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:05:30.372294 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 3, length 64
08:05:30.372356 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 3, length 64
08:05:31.375614 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 4, length 64
08:05:31.375652 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 4, length 64
08:05:32.378598 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 5, length 64
08:05:32.378657 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 5, length 64
08:05:33.378304 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 6, length 64
08:05:33.378374 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 6, length 64
08:05:34.382842 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 7, length 64
08:05:34.382921 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 7, length 64
08:05:35.385426 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 8, length 64
08:05:35.385468 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 8, length 64
08:05:36.390296 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 9, length 64
08:05:36.390333 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 9, length 64
08:05:37.390469 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 10, length 64
08:05:37.390504 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 10, length 64
08:05:38.392235 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 11, length 64
08:05:38.392307 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 11, length 64
08:05:39.399218 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 12, length 64
08:05:39.399248 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 12, length 64
08:05:40.401661 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 13, length 64
08:05:40.401700 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 13, length 64
08:05:41.406685 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 14, length 64
08:05:41.406748 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 14, length 64

Firewall LAN interface:

root@gatekeeper:/home/israel # tcpdump -i em0 proto ICMP -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
08:05:31.377214 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 4, length 64
08:05:31.377442 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 4, length 64
08:05:31.409149 IP 10.27.81.250 > 142.250.217.78: ICMP echo request, id 18156, seq 1, length 14
08:05:31.414584 IP 10.27.81.250 > 216.239.38.120: ICMP echo request, id 18155, seq 1, length 14
08:05:31.430467 IP 10.27.81.250 > 104.193.88.77: ICMP echo request, id 18157, seq 1, length 14
08:05:31.450488 IP 142.250.217.78 > 10.27.81.250: ICMP echo reply, id 18156, seq 1, length 14
08:05:31.457189 IP 216.239.38.120 > 10.27.81.250: ICMP echo reply, id 18155, seq 1, length 14
08:05:31.503578 IP 104.193.88.77 > 10.27.81.250: ICMP echo reply, id 18157, seq 1, length 14
08:05:31.669138 IP 10.27.81.251 > 104.193.88.123: ICMP echo request, id 22986, seq 1, length 14
08:05:31.671761 IP 10.27.81.251 > 142.250.217.110: ICMP echo request, id 22985, seq 1, length 14
08:05:31.677814 IP 10.27.81.251 > 216.239.38.120: ICMP echo request, id 22984, seq 1, length 14
08:05:31.720092 IP 142.250.217.110 > 10.27.81.251: ICMP echo reply, id 22985, seq 1, length 14
08:05:31.720110 IP 216.239.38.120 > 10.27.81.251: ICMP echo reply, id 22984, seq 1, length 14
08:05:31.741902 IP 104.193.88.123 > 10.27.81.251: ICMP echo reply, id 22986, seq 1, length 14
08:05:31.794959 IP 10.27.81.251 > 10.27.81.1: ICMP echo request, id 23157, seq 0, length 64
08:05:31.794986 IP 10.27.81.1 > 10.27.81.251: ICMP echo reply, id 23157, seq 0, length 64
08:05:32.380191 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 5, length 64
08:05:32.380364 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 5, length 64
08:05:32.409616 IP 10.27.81.250 > 142.250.217.78: ICMP echo request, id 18156, seq 2, length 14
08:05:32.414986 IP 10.27.81.250 > 216.239.38.120: ICMP echo request, id 18155, seq 2, length 14
08:05:32.430846 IP 10.27.81.250 > 104.193.88.77: ICMP echo request, id 18157, seq 2, length 14
08:05:32.451785 IP 142.250.217.78 > 10.27.81.250: ICMP echo reply, id 18156, seq 2, length 14
08:05:32.460504 IP 216.239.38.120 > 10.27.81.250: ICMP echo reply, id 18155, seq 2, length 14
08:05:32.502566 IP 104.193.88.77 > 10.27.81.250: ICMP echo reply, id 18157, seq 2, length 14
08:05:32.669463 IP 10.27.81.251 > 104.193.88.123: ICMP echo request, id 22986, seq 2, length 14
08:05:32.672046 IP 10.27.81.251 > 142.250.217.110: ICMP echo request, id 22985, seq 2, length 14
08:05:32.678109 IP 10.27.81.251 > 216.239.38.120: ICMP echo request, id 22984, seq 2, length 14
08:05:32.719773 IP 142.250.217.110 > 10.27.81.251: ICMP echo reply, id 22985, seq 2, length 14
08:05:32.719792 IP 216.239.38.120 > 10.27.81.251: ICMP echo reply, id 22984, seq 2, length 14
08:05:32.745302 IP 104.193.88.123 > 10.27.81.251: ICMP echo reply, id 22986, seq 2, length 14
08:05:32.872426 IP 10.27.81.201 > 1.1.1.1: ICMP echo request, id 41550, seq 0, length 64
08:05:32.922024 IP 1.1.1.1 > 10.27.81.201: ICMP echo reply, id 41550, seq 0, length 64
08:05:33.379910 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 6, length 64
08:05:33.380202 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 6, length 64
08:05:34.066848 IP 10.27.81.251 > 10.27.81.1: ICMP echo request, id 23587, seq 0, length 64
08:05:34.066873 IP 10.27.81.1 > 10.27.81.251: ICMP echo reply, id 23587, seq 0, length 64
08:05:34.384438 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 7, length 64
08:05:34.384748 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 7, length 64
08:05:35.387013 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 8, length 64
08:05:35.387175 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 8, length 64
08:05:35.610171 IP 10.27.81.250 > 216.239.38.120: ICMP echo request, id 18349, seq 0, length 14
08:05:35.614222 IP 10.27.81.250 > 142.251.211.238: ICMP echo request, id 18350, seq 0, length 14
08:05:35.619856 IP 10.27.81.250 > 104.193.88.123: ICMP echo request, id 18351, seq 0, length 14
08:05:35.651972 IP 216.239.38.120 > 10.27.81.250: ICMP echo reply, id 18349, seq 0, length 14
08:05:35.668621 IP 142.251.211.238 > 10.27.81.250: ICMP echo reply, id 18350, seq 0, length 14
08:05:35.692912 IP 104.193.88.123 > 10.27.81.250: ICMP echo reply, id 18351, seq 0, length 14
08:05:35.816110 IP 10.27.81.90 > 8.8.8.8: ICMP echo request, id 64088, seq 0, length 64
08:05:35.863191 IP 8.8.8.8 > 10.27.81.90: ICMP echo reply, id 64088, seq 0, length 64
08:05:35.884790 IP 10.27.81.251 > 216.239.38.120: ICMP echo request, id 23686, seq 0, length 14
08:05:35.894338 IP 10.27.81.251 > 142.251.215.238: ICMP echo request, id 23687, seq 0, length 14
08:05:35.894893 IP 10.27.81.251 > 104.193.88.77: ICMP echo request, id 23688, seq 0, length 14
08:05:35.926846 IP 216.239.38.120 > 10.27.81.251: ICMP echo reply, id 23686, seq 0, length 14
08:05:35.941411 IP 142.251.215.238 > 10.27.81.251: ICMP echo reply, id 23687, seq 0, length 14
08:05:35.971615 IP 104.193.88.77 > 10.27.81.251: ICMP echo reply, id 23688, seq 0, length 14
08:05:36.030861 IP 10.27.81.200 > 1.1.1.1: ICMP echo request, id 55886, seq 0, length 64
08:05:36.075680 IP 1.1.1.1 > 10.27.81.200: ICMP echo reply, id 55886, seq 0, length 64
08:05:36.391884 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 9, length 64
08:05:36.392131 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 9, length 64
08:05:36.610645 IP 10.27.81.250 > 216.239.38.120: ICMP echo request, id 18349, seq 1, length 14
08:05:36.614773 IP 10.27.81.250 > 142.251.211.238: ICMP echo request, id 18350, seq 1, length 14
08:05:36.620142 IP 10.27.81.250 > 104.193.88.123: ICMP echo request, id 18351, seq 1, length 14
08:05:36.653183 IP 216.239.38.120 > 10.27.81.250: ICMP echo reply, id 18349, seq 1, length 14
08:05:36.659589 IP 10.27.81.251 > 10.27.81.1: ICMP echo request, id 23792, seq 0, length 64
08:05:36.659607 IP 10.27.81.1 > 10.27.81.251: ICMP echo reply, id 23792, seq 0, length 64
08:05:36.666148 IP 142.251.211.238 > 10.27.81.250: ICMP echo reply, id 18350, seq 1, length 14
08:05:36.692978 IP 104.193.88.123 > 10.27.81.250: ICMP echo reply, id 18351, seq 1, length 14
08:05:36.820084 IP 10.27.81.90 > 8.8.8.8: ICMP echo request, id 64088, seq 1, length 64
08:05:36.866410 IP 8.8.8.8 > 10.27.81.90: ICMP echo reply, id 64088, seq 1, length 64
08:05:36.885163 IP 10.27.81.251 > 216.239.38.120: ICMP echo request, id 23686, seq 1, length 14
08:05:36.895265 IP 10.27.81.251 > 142.251.215.238: ICMP echo request, id 23687, seq 1, length 14
08:05:36.895753 IP 10.27.81.251 > 104.193.88.77: ICMP echo request, id 23688, seq 1, length 14
08:05:36.927345 IP 216.239.38.120 > 10.27.81.251: ICMP echo reply, id 23686, seq 1, length 14
08:05:36.942097 IP 142.251.215.238 > 10.27.81.251: ICMP echo reply, id 23687, seq 1, length 14
08:05:36.968220 IP 104.193.88.77 > 10.27.81.251: ICMP echo reply, id 23688, seq 1, length 14
08:05:37.392078 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 10, length 64
08:05:37.392249 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 10, length 64
08:05:37.611063 IP 10.27.81.250 > 216.239.38.120: ICMP echo request, id 18349, seq 2, length 14
08:05:37.616467 IP 10.27.81.250 > 142.251.211.238: ICMP echo request, id 18350, seq 2, length 14
08:05:37.620491 IP 10.27.81.250 > 104.193.88.123: ICMP echo request, id 18351, seq 2, length 14
08:05:37.655274 IP 216.239.38.120 > 10.27.81.250: ICMP echo reply, id 18349, seq 2, length 14
08:05:37.668079 IP 142.251.211.238 > 10.27.81.250: ICMP echo reply, id 18350, seq 2, length 14
08:05:37.694545 IP 104.193.88.123 > 10.27.81.250: ICMP echo reply, id 18351, seq 2, length 14
08:05:37.885657 IP 10.27.81.251 > 216.239.38.120: ICMP echo request, id 23686, seq 2, length 14
08:05:37.895698 IP 10.27.81.251 > 142.251.215.238: ICMP echo request, id 23687, seq 2, length 14
08:05:37.896072 IP 10.27.81.251 > 104.193.88.77: ICMP echo request, id 23688, seq 2, length 14
08:05:37.928228 IP 216.239.38.120 > 10.27.81.251: ICMP echo reply, id 23686, seq 2, length 14
08:05:37.942409 IP 142.251.215.238 > 10.27.81.251: ICMP echo reply, id 23687, seq 2, length 14
08:05:37.972267 IP 104.193.88.77 > 10.27.81.251: ICMP echo reply, id 23688, seq 2, length 14
08:05:38.055355 IP 10.27.81.201 > 1.1.1.1: ICMP echo request, id 63566, seq 0, length 64
08:05:38.098579 IP 1.1.1.1 > 10.27.81.201: ICMP echo reply, id 63566, seq 0, length 64
08:05:38.393834 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 11, length 64
08:05:38.394129 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 11, length 64
08:05:39.400829 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 12, length 64
08:05:39.401030 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 12, length 64
08:05:39.711356 IP 10.27.81.251 > 10.27.81.1: ICMP echo request, id 24534, seq 0, length 64
08:05:39.711375 IP 10.27.81.1 > 10.27.81.251: ICMP echo reply, id 24534, seq 0, length 64
08:05:40.403262 IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 13, length 64
08:05:40.403397 IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 13, length 64
08:05:40.799456 IP 10.27.81.250 > 216.239.38.120: ICMP echo request, id 18555, seq 0, length 14
08:05:40.805730 IP 10.27.81.250 > 142.251.33.110: ICMP echo request, id 18556, seq 0, length 14
08:05:40.812614 IP 10.27.81.250 > 104.193.88.77: ICMP echo request, id 18557, seq 0, length 14
08:05:40.841685 IP 216.239.38.120 > 10.27.81.250: ICMP echo reply, id 18555, seq 0, length 14
08:05:40.851763 IP 142.251.33.110 > 10.27.81.250: ICMP echo reply, id 18556, seq 0, length 14
^C
103 packets captured
6455 packets received by filter
0 packets dropped by kernel

Firewall enc0 interface:

root@gatekeeper:~ # tcpdump -i enc0 proto ICMP -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enc0, link-type ENC (OpenBSD encapsulated IP), capture size 262144 bytes
08:05:32.380179 (authentic,confidential): SPI 0xcff6d002: IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 5, length 64
08:05:32.380377 (authentic,confidential): SPI 0x0d12f320: IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 5, length 64
08:05:33.379898 (authentic,confidential): SPI 0xcff6d002: IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 6, length 64
08:05:33.380214 (authentic,confidential): SPI 0x0d12f320: IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 6, length 64
08:05:34.384425 (authentic,confidential): SPI 0xcff6d002: IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 7, length 64
08:05:34.384759 (authentic,confidential): SPI 0x0d12f320: IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 7, length 64
08:05:35.387000 (authentic,confidential): SPI 0xcff6d002: IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 8, length 64
08:05:35.387187 (authentic,confidential): SPI 0x0d12f320: IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 8, length 64
08:05:36.391873 (authentic,confidential): SPI 0xcff6d002: IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 9, length 64
08:05:36.392143 (authentic,confidential): SPI 0x0d12f320: IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 9, length 64
08:05:37.392064 (authentic,confidential): SPI 0xcff6d002: IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 10, length 64
08:05:37.392262 (authentic,confidential): SPI 0x0d12f320: IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 10, length 64
08:05:38.393821 (authentic,confidential): SPI 0xcff6d002: IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 11, length 64
08:05:38.394142 (authentic,confidential): SPI 0x0d12f320: IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 11, length 64
08:05:39.400815 (authentic,confidential): SPI 0xcff6d002: IP 10.27.84.1 > 10.27.81.60: ICMP echo request, id 26633, seq 12, length 64
08:05:39.401043 (authentic,confidential): SPI 0x0d12f320: IP 10.27.81.60 > 10.27.84.1: ICMP echo reply, id 26633, seq 12, length 64
^C
16 packets captured
16 packets received by filter
0 packets dropped by kernel


[ibrewster]

Given what I am seeing above, I'm beginning to think it's an issue with the built-in VPN client on MacOS Sonoma, not something wrong with my configuration, and as such I may have to throw in the towel and go with something else, like OpenVPN.  :-\

[Monviech (Cedrik)]

For IPsec I'm using NCP client on macOS productively with my customers. Wireguard works also great, but not using that as customer remote access.

[ibrewster]

For IPsec I'm using NCP client on macOS productively with my customers. Wireguard works also great, but not using that as customer remote access.

Yeah, I was hoping to avoid having yet another piece of software to install/manage on my client, but if the built-in VPN isn't functional, then so be it. Though I would prefer to go with something free (such as OpenVPN or Wireguard) if I can make it work, rather than paying for an IPsec client :)

[ibrewster]

aaaand OpenVPN doesn't like me either. Every connection attempt just fails with "Peer certificate verification failure". I don't even want to *use* peer certificates! I just want to be able to log into my home network with a username and password!

Sigh. On to Wireguard I guess...

[Patrick M. Hausen]

OpenVPN mandates using certificates.

[ibrewster]

OpenVPN mandates using certificates.

Well, I tried that too (that is, I created a certificate for my user and installed that on the client), still got the same error. So dunno