Routing Issue After 23.1 update

[nam061]

Hi Guys

I updated to 23.1 a few days ago, and now I am experiencing an issue with one of my machines. I have one machine with OPnsense installed that acts as a basic NAT router for my private subnet 192.168.50.0/24. This machine/router uses the private IP 192.168.50.1, which is the gateway for my 192.168.50.0/24 network.

I have a physical machine with 1x Public NIC and 1x Private NIC. On this machine, I have a VM running also equipped with 1x Public IP and then 1x Private IP falling in the 192.168.50.0/24 subnet. I am trying to mount an NFS share on this VM. I have two NFS shares and only ONE NFS share mounts successfully while the second refuses.

This is what the NFS mounts look like:

Successful Mount

root@s-145-VM:~# mount -t nfs -vvv 102.165.XXX.YYY:/data/secondary /mnt/SecStorage/test
mount.nfs: timeout set for Thu Feb 16 14:07:58 2023
mount.nfs: trying text-based options 'vers=4.2,addr=102.165.XXX.YYY,clientaddr=197.189.XXX.YYY'
root@s-145-VM:~#

Failed Mount:

root@s-145-VM:~# mount -t nfs 102.165.XXX.ZZZ:/data/secondary /mnt/SecStorage/test
mount.nfs: access denied by server while mounting 102.165.XXX.ZZZ:/data/secondary
root@s-145-VM:~# mount -t nfs -vvv 102.165.XXX.ZZZ:/data/secondary /mnt/SecStorage/test
mount.nfs: timeout set for Thu Feb 16 14:07:11 2023
mount.nfs: trying text-based options 'vers=4.2,addr=102.165.XXX.ZZZ,clientaddr=192.168.50.53'
mount.nfs: mount(2): Operation not permitted
mount.nfs: trying text-based options 'addr=102.165.XXX.ZZZ'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 102.165.XXX.ZZZ prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 102.165.XXX.ZZZ prog 100005 vers 3 prot UDP port 892
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting 102.165.XXX.ZZZ:/data/secondary

As you can see, in the failed mount attempt, the VM passes its private IP address when it tries to mount the second NFS. And not its public IP 197.189.XXX.YYY. And the reason this is happening is that the VM automatically after each reboot adds the following line to its IP routing table:

- 102.165.XXX.ZZZ via 192.168.50.1 dev eth1

When I delete this entry from the routing table, the NFS mounts successfully.  Something in the "Firewall" OPnsense router is causing this behaviour and making it create this entry in the IP route table. And I have no idea what it could be. What puzzles me, it only creates this entry for this IP and not for the other NFS, which is successful and they are basically on the same network space `102.165.XXX`

I have double-checked the Opnsense server and rebooted it many times and I cannot pinpoint what is causing this behaviour. All I know is, that it happened right after updating to 23.1 and rebooting this VM is now causing this behaviour.

Any suggestions for this will be extremely appreciated, please.