[Update] IPsec: wrong interface picked somehow (ping: sendto: Permission denied)

[arjen]

[Update below]

Hi

we have an on-premise OPNsense device (DEC4630) and a VM in Azure with OPNsense installed. Both are running with version 19.7.1:
OPNsense 19.7.1-amd64
FreeBSD 11.2-RELEASE-p11-HBSD
OpenSSL 1.0.2s 28 May 2019

I'm trying to set up a routed IPsec tunnel between the devices, which is partially working.
From a VM in Azure, I can ping some devices on the local network (not all, my computer is not working),

Code: [Select]

    +== 10.4.8.224/27 =================+                                                        +== 192.168.0.1/24 ================+
    |                                  |                                                        |                                  |
    |                 10.4.8.228 +-----------------+ 192.168.255.1       192.168.255.2  +-----------------+ 192.168.0.1            |
    |                 +----------+ OPNsense Azure  |-------------- IPsec ---------------| OPNsense Local  |-----------------+      |
    |                 |          +-----------------+                                    +-----------------+                 |      |
    |                 |                |                                                        |           192.168.0.100   |      |
    +=================|================+                                                        |       +---------------+   |      |
                      |                                                                         |       | Windows 10    |---+      |
    +== 10.4.4.0/22 ==|=====+                                                                   |       +---------------+   |      |
    |                 |     |                                                                   |                           |      |
    |        10.4.4.6 |     |                                                                   |                           |      |
    |   +---------------+   |                                                                   |            192.168.0.10   |      |
    |   | Debian VM     |   |                                                                   |       +---------------+   |      |
    |   +---------------+   |                                                                   |       | Other Device  |---+      |
    |                       |                                                                   |       +---------------+          |
    +=======================+                                                                   |                                  |
                                                                                                +==================================+

OPNsense local

Gateways:

Name	Interface	IP
AZUREGW	IPsec_ZRH	192.168.255.1

Routes:

Network	Gateway
10.4.8.224/27	AZUREGW - 192.168.255.1
10.4.4.0/24	AZUREGW - 192.168.255.1

OPNsense Azure

Gateways:

Name	Interface	IP
RouteGW	Route_AZURE	10.4.8.225
ZRHGW	IPsec_AZURE	192.168.255.2

Routes:

Network	Gateway
10.4.4.0/22	RouteGW - 10.4.8.225
192.168.0.0/24	ZRHGW - 192.168.255.2

The IPsec is up and running, I can successfully ping from the Debian VM on Azure to 192.168.0.10 on the local network.
But unfortunately, I cannot ping 192.168.0.100 nor vice versa. (On 192.168.0.10 I cannot test pinging back as it  is a Unifi Cloud Key and does not have such an option).

I have the following firewall rules on both OPNsense devices:

Floating - IPv4+6* * * * * * *
With interfaces IPsec, IPsec_AZURE, and the local interfaces (Route_AZURE or LAN)

In the firewall log, I noticed the following differences between the two devices:

When I ping 10.4.4.6 from 192.168.0.100 I get the following entries on OPNsense_local:




Whereas on OPNsense Azure when I ping 192.168.0.100 from 10.4.4.6 I get the following:




(see attachments)

Why does it use the "IPsec" interface on the local OPNsense and not on the one in Azure?

And what could be the issue that the connection is not working? Could it be a routing problem? Or firewall?
I'm really puzzled...

Thanks a lot for a hint!
Arjen
This issue seems to be related to the following forum post:
https://forum.opnsense.org/index.php?topic=12665.0
ping: sendto: Permission denied

When I try to ping 10.4.4.6 on the OPNsense Local machine (from the LAN interface), I get the following messages:

Code: [Select]

# /sbin/ping -S '192.168.0.1' -c '3' '10.4.4.6'
PING 10.4.4.6 (10.4.4.6) from 192.168.0.1: 56 data bytes

--- 10.4.4.6 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
ping: sendto: Permission denied
ping: sendto: Permission denied
ping: sendto: Permission denied
(I didn't pay attention enough before, thought it's just not working)

In fact, if I ping my local machine 192.168.0.100 from 10.4.4.6 and run Wireshark, I see that the echo request is hitting my machine, but the response doesn't make it back to the source...

[arjen]

Apparently, this issue has to do with the Auto-NAT-rules generation.
(see the issue here, for example: https://forum.opnsense.org/index.php?topic=13525.0)

Setting the Outbound NAT rule generation to manually solved the issue and allowed me to ping the machine in Azure.
Nevertheless, I somehow have to initiate the connection from Azure first. I need the ping my local machine from Azure first before I can ping the Azure machine from my local computer.