smtp (and other hosts) on a legacy IPsec tunnel unreachable from firewall itself

[BISI Sysadmin]

I am attempting to get monit to send me an alert when an IPsec [legacy] tunnel is down.

I am seeing some behaviours that I do not understand, I'm reluctant to apply the potential solutions I've found during my searching 'til I know better what's going on.

monit shows this in the logs when I disable the tunnel I am attempting to monitor

Code: [Select]

2023-10-29T21:13:30-07:00 Error monit Aborting event
2023-10-29T21:13:30-07:00 Error monit Mail: Delivery failed -- no mail server is available
2023-10-29T21:13:30-07:00 Error monit Cannot open a connection to the mailserver 192.168.254.9:25 -- Operation now in progress
2023-10-29T21:13:30-07:00 Error monit Cannot connect to [192.168.254.9]:25 -- Connection timed out
2023-10-29T21:13:00-07:00 Error monit 'restart_PCC_IPsec_tunnel' ping test failed
2023-10-29T21:13:00-07:00 Error monit Ping response for 192.168.78.254 5/5 timed out -- no response within 5 s
2023-10-29T21:12:55-07:00 Warning monit Ping response for 192.168.78.254 4/5 timed out -- no response within 5 s
2023-10-29T21:12:50-07:00 Warning monit Ping response for 192.168.78.254 3/5 timed out -- no response within 5 s
2023-10-29T21:12:45-07:00 Warning monit Ping response for 192.168.78.254 2/5 timed out -- no response within 5 s
2023-10-29T21:12:40-07:00 Warning monit Ping response for 192.168.78.254 1/5 timed out -- no response within 5 s
2023-10-29T21:12:35-07:00 Informational monit 'gw1.domain.tld' Monit reloaded

Additional information:
1st - the firewall running monit [OPNsense 23.7.7_3-amd64] is at 192.168.17.254
2nd - the mail server is at 192.168.254.9
3rd - the mail server at 192.168.254.9 is reachable from any other host on the 192.168.17.0/24 network

example from a normal host on the LAN:

Code: [Select]

root@ns1:~# ip addr show dev enp0s4 | grep inet
    inet 192.168.17.14/24 brd 192.168.17.255 scope global noprefixroute enp0s4
#
root@ns1:~# ping -q -c 3 192.168.254.9
PING 192.168.254.9 (192.168.254.9) 56(84) bytes of data.

--- 192.168.254.9 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2014ms
rtt min/avg/max/mdev = 14.848/16.489/18.878/1.727 ms
#
root@ns1:~# telnet 192.168.254.9 25
Trying 192.168.254.9...
Connected to 192.168.254.9.
Escape character is '^]'.
220 mx-backup.bisi.ca ESMTP Postfix (Ubuntu)

Example from the shell of the OPNsense box:

Code: [Select]

root@gw1:~ # ifconfig re0 | grep inet
inet 192.168.17.254 netmask 0xffffff00 broadcast 192.168.17.255
#
root@gw1:~ # ping -q -c 3 192.168.254.9
PING 192.168.254.9 (192.168.254.9): 56 data bytes

--- 192.168.254.9 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
#
root@gw1:~ # telnet 192.168.254.9 25
Trying 192.168.254.9...
^C

This description from the netgate docs seems to describe the situation, as well as a fix, but is this really what's going on here?
https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/access-firewall-over-ipsec.html
https://web.archive.org/web/20231030051250/https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/access-firewall-over-ipsec.html

I have obtained the same result from 4 other OPNsense firewalls of various generations, both from the shell and from Interfaces -> diagnostics -> ping.

All firewalls can ping (or telnet/ssh) to) hosts outside the firewall, and hosts on their own class C LAN (192.168.17.x in this example).

So - any feedback appreciated (other than to migrate the tunnels to "Connections" I've already incurred *way* too many unbillable hours for this client trying to do just that).  Sorely hoping there's a simpler fix to this issue so I can stop hearing from these clients