NFS Server WAN - mount within opnsense LAN VM

[leonidas-o]

Hello guys, completely new opnsense user here.

I'm running a proxmox cluster with two nodes. On each node, I'm trying to setup a VM with opnsense v22.7.8. I read through all the nfs threads here, in pfsense and everywhere else. I simply can't get a VM within opnsense LAN to mount a nfs share. Before I moved this VM into the opnsense lan network, it was running under a normal linux bridge and the nfs mount was working. As soon as I replace the old vmbr0 linux bridge with the new vmbr11 (opnsense VM NET), I can't mount the nfs share anymore. Internet is working, pinging is working. The linux "ls -la" command for the mounted nfs share just shows:

Code Select Expand

d?????????? ? ?         ?          ?            ?

The bridge on the proxmox node is configured like this:

Code Select Expand

auto vmbr10
iface vmbr10 inet static
        address 10.10.10.0/31
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.1/31' -o enp0s31f6 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.1/31' -o enp0s31f6 -j MASQUERADE

The NFS server, is a separate dedicated server with its own public IP. It is behind a firewall, too. I allowed all ports, all protocols, simply everything for the two Proxmox nodes.

I'm working here on the second Proxmox node, it has its own public IP. On the node, I'm redirecting all traffic to opnsense except a few tcp and udp ports needed for proxmox and ssh.

Code Select Expand

  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p tcp -m multiport ! --dport 22,8006,179 -j DNAT --to 10.10.10.1
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p udp -m multiport ! --dport 5405:5412,4789 -j DNAT --to 10.10.10.1

I would say, all is working so far, but can't get this nfs mount to work.

In opnsense I unchecked "Block private networks" and "Block bogon networks" for WAN and LAN.

I can ping the nfs server from the node itself, from the opnsense diagnostic ping tool (from WAN and LAN) and from the VM itself. When I try to manually mount, I get

Code Select Expand

mount.nfs4: Operation not permitted
I've also done a packet capture, it shows two errors, for WAN and LAN the same:

Code Select Expand

ethertype IPv4 (0x0800), length 186: (tos 0x0, ttl 60, id 53543, offset 0, flags [DF], proto TCP (6), length 172)
    136.XX.XX.XX.2049 > 10.10.10.1.18318: Flags [P.], cksum 0xe850 (correct), seq 629:749, ack 1297, win 501, options [nop,nop,TS val 3529236269 ecr 2468818434], length 120: NFS reply xid 961210637 reply ok 116 getattr ERROR: Operation not permitted
For auto-mounting I'm using "autofs" and the options "-fstype=nfs4,rw,retry=0". For the manual test mount, I'm also using nfs4.

I've also experimented with "Firewall" -> "Settings" -> "Normalisation", checked and unchecked "Disable interface scrub", "IP Do-Not-Fragment" and "IP Random id".

And finally, I added opnsense firewall rules like:

- "PASS IN IPv4 * * * 136.XX.XX.XX * * *"
- "PASS IN IPv4 * 136.XX.XX.XX * * * * *"
- "PASS OUT IPv4 * 136.XX.XX.XX * * * * *"

So basically allowing everything IN with destination 136.XX.XX.XX as well as allow everything  IN and OUT with source 136.XX.XX.XX.

But somehow something in opnsense still don't let me mount the nfs share.
Am I missing something?

[leonidas-o]

found some help here: https://forum.netgate.com/topic/176298/nfs-server-wan-mount-within-opnsense-pfsense-lan

In the end it came down to the nfs server option "insecure" flag or using the firewalls "static port" option. More details in the thread mentioned above.