Problem with ping on one external host

[syshein]

Hi there,

i have a really strange problem. When i try to ping a specific host i don't get an answer but see the answer appear two times on the WAN interface. Here is my setup:
I have an Opnsense as a router with a business optical fibre internet connection connected to the WAN interface. The Opnsense runs as a VM and the WAN interface is accessed via PCI passthrough so only the Opnsense VM can access it. I have a bunch of LAN interfaces, e.g. one for my home net, one for my IoT devices, one for my business devices and one which is connected to another private network behind a consumer internet connection.
Everything seems to work fine, but i can't connect to or ping my cloud telephone system over my business connection.
I tried to debug it and run a ping from my PC in my home lan net (IP 10.11.0.5) to my telephone system (IP 212.XXX.XXX.XXX) while running a tcpdump on my Opnsense on the WAN interface (IP 87.XXX.XXX.XXX). My PC doesn't get an answer from my telephone system but i see the following on my WAN interface:
root@OPNsense:~ # tcpdump -i ix1 icmp -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ix1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:33:04.408713 IP 87.XXX.XXX.XXX > 212.XXX.XXX.XXX: ICMP echo request, id 19708, seq 57, length 64
09:33:04.417134 IP 212.XXX.XXX.XXX > 87.XXX.XXX.XXX: ICMP echo reply, id 19708, seq 57, length 64
09:33:04.417150 IP 212.XXX.XXX.XXX > 10.11.0.5: ICMP echo reply, id 61649, seq 57, length 64
09:33:05.432725 IP 87.XXX.XXX.XXX > 212.XXX.XXX.XXX: ICMP echo request, id 19708, seq 58, length 64
09:33:05.441115 IP 212.XXX.XXX.XXX > 87.XXX.XXX.XXX: ICMP echo reply, id 19708, seq 58, length 64
09:33:05.441137 IP 212.XXX.XXX.XXX > 10.11.0.5: ICMP echo reply, id 61649, seq 58, length 64
09:33:06.456799 IP 87.XXX.XXX.XXX > 212.XXX.XXX.XXX: ICMP echo request, id 19708, seq 59, length 64
09:33:06.465186 IP 212.XXX.XXX.XXX > 87.XXX.XXX.XXX: ICMP echo reply, id 19708, seq 59, length 64
09:33:06.465207 IP 212.XXX.XXX.XXX > 10.11.0.5: ICMP echo reply, id 61649, seq 59, length 64

For every request i see two replies, one with my public IP as DST and one with my private IP as DST. Whats even more strange, when i start the tcpdump with -Q out i see only my request and with -Q in i see the two replies. At first i thought the Opnsense gets the answer, replaces the DST with my private IP and then something went wrong while routing it but then it should be an outgoing package and not an incoming. it makes absolutely no sense that i see the second answer package as incoming because if its coming from the internet it shouldn't be routed to me and even if it is routed to me the WAN interface shouldn't accept it because its for another IP.

I think its a problem of the opnsense and not my internet provider because my private IP should not be known outside my network.

My Opnsense is also connected to a consumer conntection. When i create a route transmitting the pings via this consumer connection instead of my business connection i get an answer as expected and everything works.

Whats also strange and i don't know if its a part of the same or another problem: before i got my business internet connection two weeks ago my wan interface was connected to the private network with the consumer internet connection and i used this connection. Since i switched to my own connection and connected the wan interface to the modem (via a lancable, one direct connection, nothing else involved) two or three times after a reboot the wan interface got the IP it got when it was connected to the private network. When i go to Interfaces -> overview and click the "reload" button for the WAN interface it gets the right IP. That does not happen at every reboot but sometimes.

I can't explain how this problems can occure. it makes absolutely no sense. Maybe something is really broken in my opnsense installation, maybe i have to start from scratch but because of my many networks and firewall rules i don't wanna do it.

Thanks for your help
Adrian

[meyergru]

Do you operate the two connections from the same location? For multi-WAN setups, you must disable "Reply-To" on WAN connections to avoid assymetric routing.

Also, to avoid your RFC1918 IPs to the main gateway(s), you should set null routes to them: https://forum.opnsense.org/index.php?msg=259031

[syshein]

It's currently not a multi-WAN setup, i have a connection to this other private network so i can access my network from there. There are only routes for the IP range of that private network on that interface.
For a test i set a /32 route for the IP of my cloud telephone system over this other private network and that is working as expectet. But when i get the strange ping results, this route is disabled (otherwise i wouldn't see any of this traffic on the WAN interface).

I set a null route for 10.0.0.0/8 but get the same results as before...

[drosophila]

It probably is connected to your other issue, there seems to be some cruft in your config that creates a race condition for the IP config, and there seems to be a rules issue that may come from the same source. You could check the firewall live view if those ICMP replies are blocked. Also, your route configuration may be borked and have entries that blackhole these ICMP responses.

Like that a disabled but configured route blocks automatic generation of any route with an identical target network. IOW, if you configure a disabled static route to blackhole 192.168.0.0/16 and have this range on your LAN, then the LAN will get no route at all. This simple case is obviously not what you have, but there may be similar routing configurations left over, depending on the complexity of your set-up.

You could attempt a config export, followed by a config clear and a config import. Unless the remains are stored in the config file they should be gone after this withot affecting the intended configuration.

[lmoore]

09:33:06.465186 IP 212.XXX.XXX.XXX > 87.XXX.XXX.XXX: ICMP echo reply, id 19708, seq 59, length 64
09:33:06.465207 IP 212.XXX.XXX.XXX > 10.11.0.5: ICMP echo reply, id 61649, seq 59, length 64

It appears as if the packet has been reflected back out to ix1.

Just curious, what is the TTL in the echo-reply packets?

Code Select Expand

tcpdump -pnvi ix1 icmp and host 212.XXX.XXX.XXX
If you're not using a dual-WAN configuration, I would suggest you review _all_ of your rules which include 'Reply-to'.

To do this, in your 'Rules [new]' view, click on the column selection drop-down list, scroll down towards the end and select 'Reply-to'.

Ensure you have selected 'All rules' and then expand 'Automatically generated rules' so that you see them too. Identify any rule which is using 'Reply-to' and review if it may be interfering with the pings.

Have you tried pinging your phone system from another device on the home LAN and also from one of your other networks, which access the Internet through OPNsense?

One suggestion, disable the OPNsense interface which is connected to another private network behind a consumer internet connection and see if there is any change in behaviour.

Also review your Source and Destination NAT rules.

[lmoore]

In addition, to see the source address that has been used by your NAT rule, you could run 'pftop'. The address will be listed under the 'GW' column.

Code Select Expand

pftop -s1 -v state -f 'icmp and host 212.XXX.XXX.XXX'
The output should look something like this;

Code Select Expand

pfTop: Up State 1-2/2 (101), View: state, Order: none, Cache: 10000                                                 22:28:53

PR    DIR SRC                    DEST                  STATE   AGE   EXP  PKTS BYTES  RATE  PEAK   AVG RU GW
icmp  In  10.11.0.5:1            212.XXX.XXX.XXX:8      0:0      3     9     6   360               120  *
icmp  Out 87.XXX.XXX.XXX:27203   212.XXX.XXX.XXX:8      0:0      3     9     6   360               120  * 10.11.0.5:1