[solved] Problem with IPsec since 21.7.1 upgrade

[kosta]

Hello,

I hope it's fine if I post this here too, I did in the German forum also.
Since the upgrade to 21.7.1 (following the upgrade path via GUI), DNS doesn't resolve over IPsec Tunnel. Meaning: I have overrides on my OPNsense for certain domains in the company, to over company DC, which is behind another OPNsense (21.4.2). This has worked up to the upgrade to 21.7.1 on my own OPNsense without a problem.

This is an excerpt from the Level 4 Log in Unbound:
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: mesh_run: validator module exit state is module_finished   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: cannot validate non-answer, rcode SERVFAIL   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: attempt to get extra 3 targets   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: close fd 177   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: close of port 38016   
2021-08-08T11:56:57   unbound[49796]   [49796:2] notice: remote address is ip4 10.10.11.11 port 53 (len 16)   
2021-08-08T11:56:57   unbound[49796]   [49796:2] notice: send failed: Permission denied   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: comm point start listening 177 (-1 msec)   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: opened UDP if=0 port=38016   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: servselect ip4 10.10.11.11 port 53 (len 16)   
2021-08-08T11:56:57   unbound[49796]   [49796:2] info: DelegationPoint<internal.domain.com.>: 0 names (0 missing), 1 addrs (1 result, 0 avail) parentNS   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: close fd 177   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: close of port 50169   
2021-08-08T11:56:57   unbound[49796]   [49796:2] notice: remote address is ip4 10.10.11.11 port 53 (len 16)   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: dnssec status: not expected   
2021-08-08T11:56:57   unbound[49796]   [49796:2] info: DelegationPoint<internal.domain.com.>: 0 names (0 missing), 1 addrs (1 result, 0 avail) parentNS   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: iter_handle processing q with state QUERY TARGETS STATE   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: sending to target: <internal.domain.com.> 10.10.11.11#53   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: rtt=376   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: servselect ip4 10.10.11.11 port 53 (len 16)   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: attempt to get extra 3 targets   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: ip4 10.10.11.11 port 53 (len 16)   
2021-08-08T11:56:57   unbound[49796]   [49796:2] info: DelegationPoint<internal.domain.com.>: 0 names (0 missing), 1 addrs (1 result, 0 avail) parentNS   
2021-08-08T11:56:57   unbound[49796]   [49796:2] notice: remote address is ip4 10.10.11.11 port 53 (len 16)   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: opened UDP if=0 port=33175   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: rtt=376   
2021-08-08T11:56:57   unbound[49796]   [49796:2] info: error sending query to auth server ip4 10.10.11.11 port 53 (len 16)   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: close fd 177   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: close of port 61877   
2021-08-08T11:56:57   unbound[49796]   [49796:2] notice: remote address is ip4 10.10.11.11 port 53 (len 16)   
2021-08-08T11:56:57   unbound[49796]   [49796:2] notice: send failed: Permission denied   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: serviced query UDP timeout=376 msec   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: EDNS lookup known=0 vs=0   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: dnssec status: not expected   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: sending to target: <internal.domain.com.> 10.10.11.11#53   
2021-08-08T11:56:57   unbound[49796]   [49796:2] info: sending query: dc01.internal.domain.com. AAAA IN   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: selrtt 376   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: rtt=376   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: servselect ip4 10.10.11.11 port 53 (len 16)   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: attempt to get extra 3 targets   
2021-08-08T11:56:57   unbound[49796]   [49796:2] debug: ip4 10.10.11.11 port 53 (len 16)

I anonymized the domain.
I checked couple of things:
Access Lists, which only list local networks, seem fine. There is no entry for the remote network.
I tried setting All (Recommended) for Outgoing Interfaces and also Network Interfaces, that doesn't help.
Tried deactivating Suricata, Sensei and HAproxy just for test, no help (I suspected Suricata might play some role, but nothing in there really).
Checked the IPsec SAs, they seem to be established fine.
DNS Lookup in the Diagnostics doesn't work (??), simply does nothing.
Ping -> DC from PC OK, Ping from Firewall -> DC fails (what the...??).
--- 10.10.11.11 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
ping: sendto: Permission denied
ping: sendto: Permission denied
ping: sendto: Permission denied

It is quite obvious that it's since the update only, so either I am missing something or some new feature or there is a bug.

Can someone help please?

Thank you.

EDIT: it seems it's not only DNS. For instance, I can ping the remote server, but I can't access it, not even RDP via IP.
However, I do seem to be able to connect from the company to my work-computer at home.

EDIT2: the source of the problem seems to be the inability of the home firewall to reach the domain controller (or any other device) on the remote network(s).

[kosta]

The issue was one old Gateway entry in the GW-List, something I used to test Routed-IPsec. Either I by mistake activated it or it activated after the upgrade. Deleted static routes and the gateway, and all is as it used to be.