Problem with ping on one external host

[syshein]

Hi there,

i have a really strange problem. When i try to ping a specific host i don't get an answer but see the answer appear two times on the WAN interface. Here is my setup:
I have an Opnsense as a router with a business optical fibre internet connection connected to the WAN interface. The Opnsense runs as a VM and the WAN interface is accessed via PCI passthrough so only the Opnsense VM can access it. I have a bunch of LAN interfaces, e.g. one for my home net, one for my IoT devices, one for my business devices and one which is connected to another private network behind a consumer internet connection.
Everything seems to work fine, but i can't connect to or ping my cloud telephone system over my business connection.
I tried to debug it and run a ping from my PC in my home lan net (IP 10.11.0.5) to my telephone system (IP 212.XXX.XXX.XXX) while running a tcpdump on my Opnsense on the WAN interface (IP 87.XXX.XXX.XXX). My PC doesn't get an answer from my telephone system but i see the following on my WAN interface:
root@OPNsense:~ # tcpdump -i ix1 icmp -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ix1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:33:04.408713 IP 87.XXX.XXX.XXX > 212.XXX.XXX.XXX: ICMP echo request, id 19708, seq 57, length 64
09:33:04.417134 IP 212.XXX.XXX.XXX > 87.XXX.XXX.XXX: ICMP echo reply, id 19708, seq 57, length 64
09:33:04.417150 IP 212.XXX.XXX.XXX > 10.11.0.5: ICMP echo reply, id 61649, seq 57, length 64
09:33:05.432725 IP 87.XXX.XXX.XXX > 212.XXX.XXX.XXX: ICMP echo request, id 19708, seq 58, length 64
09:33:05.441115 IP 212.XXX.XXX.XXX > 87.XXX.XXX.XXX: ICMP echo reply, id 19708, seq 58, length 64
09:33:05.441137 IP 212.XXX.XXX.XXX > 10.11.0.5: ICMP echo reply, id 61649, seq 58, length 64
09:33:06.456799 IP 87.XXX.XXX.XXX > 212.XXX.XXX.XXX: ICMP echo request, id 19708, seq 59, length 64
09:33:06.465186 IP 212.XXX.XXX.XXX > 87.XXX.XXX.XXX: ICMP echo reply, id 19708, seq 59, length 64
09:33:06.465207 IP 212.XXX.XXX.XXX > 10.11.0.5: ICMP echo reply, id 61649, seq 59, length 64

For every request i see two replies, one with my public IP as DST and one with my private IP as DST. Whats even more strange, when i start the tcpdump with -Q out i see only my request and with -Q in i see the two replies. At first i thought the Opnsense gets the answer, replaces the DST with my private IP and then something went wrong while routing it but then it should be an outgoing package and not an incoming. it makes absolutely no sense that i see the second answer package as incoming because if its coming from the internet it shouldn't be routed to me and even if it is routed to me the WAN interface shouldn't accept it because its for another IP.

I think its a problem of the opnsense and not my internet provider because my private IP should not be known outside my network.

My Opnsense is also connected to a consumer conntection. When i create a route transmitting the pings via this consumer connection instead of my business connection i get an answer as expected and everything works.

Whats also strange and i don't know if its a part of the same or another problem: before i got my business internet connection two weeks ago my wan interface was connected to the private network with the consumer internet connection and i used this connection. Since i switched to my own connection and connected the wan interface to the modem (via a lancable, one direct connection, nothing else involved) two or three times after a reboot the wan interface got the IP it got when it was connected to the private network. When i go to Interfaces -> overview and click the "reload" button for the WAN interface it gets the right IP. That does not happen at every reboot but sometimes.

I can't explain how this problems can occure. it makes absolutely no sense. Maybe something is really broken in my opnsense installation, maybe i have to start from scratch but because of my many networks and firewall rules i don't wanna do it.

Thanks for your help
Adrian

[meyergru]

Do you operate the two connections from the same location? For multi-WAN setups, you must disable "Reply-To" on WAN connections to avoid assymetric routing.

Also, to avoid your RFC1918 IPs to the main gateway(s), you should set null routes to them: https://forum.opnsense.org/index.php?msg=259031