Messages

1

General Discussion / Re: Cloned opnsense VM, no internet access

« on: May 28, 2023, 01:33:51 am »

Okay after I tried with opnsense 23.1.8, reset to factory defaults, even with a completely new installation and all of these approaches were showing the same behaviour, I knew it must be something else.

Fortunately it is documented in the proxmox docs (I simply forgot about it). I was missing the following entries in my network/interfaces file:

Code: [Select]

  post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
  post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

On my first proxmox node it was working without these lines but the third node needed it. I think that could be because  the third proxmox node is running on a slightly newer version, so updating the first node would maybe cause the same issue. Nevertheless it is working now. I don't want to touch the VM again, so still don't know for sure if I can clone the VM, as the latest setup was a fresh installation.

2

General Discussion / Cloned opnsense VM, no internet access

« on: May 27, 2023, 12:23:12 pm »

Hello guys,

I got a very strange behavior. I'm running opnsense OPNsense 22.7.9_3 on a VM, inside a proxmox cluster. The proxmox cluster had two nodes, I added a third one. Each node has an opnsense VM, so I backup'ed the opnsense VM on node1 and restored it on the new third proxmox node (deleted in Proxmox the networks devices MAC addresses so new ones get generated). It has no HA configuration, not for Proxmox and not for opnsense. So they all work independent from each other.
I also got a separate wireguard VM on each of my proxmox nodes and the network/interfaces file is the same except the IP addresses of course.

Code: [Select]

iface enp0s31f6 inet static
  address MYPUBLICIP
  ...
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p tcp -m multiport ! --dport 22,8006,179 -j DNAT --to 10.10.10.2
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p udp -m multiport ! --dport 5405:5412,4789 -j DNAT --to 10.10.10.2

...

auto vmbr10
iface vmbr10 inet static
        address 10.10.10.1/30
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.2/32' -o enp0s31f6 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.2/32' -o enp0s31f6 -j MASQUERADE
# OPNSense WAN - Proxmox LAN

auto vmbr11
iface vmbr11 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0
# VM Net
vmbr10 is the WAN interface and vmbr11 the LAN interface for opnsense VM.

So I tried the following:
- I've started the third opnsense vm, updated the IPv6 addresses inside the UI, IPv4 matched already as well as the interface assignments.
- I've reset to factory default and copied a config.xml with the corrected IPv6 addresses from proxmox's opnsense vm node 1 over to the third one.

The result is always the same. I don't have internet access from within LAN.
- I can access the Host server where proxmox is running on via ssh and ping whatever I want, that works.
- I can connect over the internet to the wireguard VM which is running in the opnsense LAN, I can then ping all the VMs inside LAN network.
- I can enter the opnsense web UI (Interfaces, Diagnostic, Ping) and ping the proxmox HOST's public IP (from WAN or LAN interfaces)
  - However I can NOT ping e.g. 1.1.1.1 (from WAN or LAN interfaces) -> 3 packets transmitted, 0 packets received, 100.0% packet loss.

I remember I had the same issue when I tried to clone the VM from Proxmox node1 to proxmox node2. I ended up configuring opnsense from scratch after a factory reset. Recreating all rules, port-forwardings etc. by hand.
Is there some kind of cache or anything deep buried in the system, which could be responsible for causing such behavior when you clone the vm?
It doesn't make sense to me, I cloned the VM, I adapted the rules to match the public IPv4 address of node 3 and updated also the ipv6 addresses for the interface wan gateway etc. I can also connect to wireguard from outside and ping within LAN but I cannot go out. What am I missing?

3

General Discussion / Re: NFS Server WAN - mount within opnsense LAN VM

« on: December 09, 2022, 10:57:45 am »

found some help here: https://forum.netgate.com/topic/176298/nfs-server-wan-mount-within-opnsense-pfsense-lan

In the end it came down to the nfs server option "insecure" flag or using the firewalls "static port" option. More details in the thread mentioned above.

4

General Discussion / NFS Server WAN - mount within opnsense LAN VM

« on: December 04, 2022, 10:31:03 pm »

Hello guys, completely new opnsense user here.

I'm running a proxmox cluster with two nodes. On each node, I'm trying to setup a VM with opnsense v22.7.8. I read through all the nfs threads here, in pfsense and everywhere else. I simply can't get a VM within opnsense LAN to mount a nfs share. Before I moved this VM into the opnsense lan network, it was running under a normal linux bridge and the nfs mount was working. As soon as I replace the old vmbr0 linux bridge with the new vmbr11 (opnsense VM NET), I can't mount the nfs share anymore. Internet is working, pinging is working. The linux "ls -la" command for the mounted nfs share just shows:

Code: [Select]

d?????????? ? ?         ?          ?            ?
The bridge on the proxmox node is configured like this:

Code: [Select]

auto vmbr10
iface vmbr10 inet static
        address 10.10.10.0/31
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.1/31' -o enp0s31f6 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.1/31' -o enp0s31f6 -j MASQUERADE
The NFS server, is a separate dedicated server with its own public IP. It is behind a firewall, too. I allowed all ports, all protocols, simply everything for the two Proxmox nodes.

I'm working here on the second Proxmox node, it has its own public IP. On the node, I'm redirecting all traffic to opnsense except a few tcp and udp ports needed for proxmox and ssh.

Code: [Select]

  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p tcp -m multiport ! --dport 22,8006,179 -j DNAT --to 10.10.10.1
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p udp -m multiport ! --dport 5405:5412,4789 -j DNAT --to 10.10.10.1
I would say, all is working so far, but can't get this nfs mount to work.

In opnsense I unchecked "Block private networks" and "Block bogon networks" for WAN and LAN.

I can ping the nfs server from the node itself, from the opnsense diagnostic ping tool (from WAN and LAN) and from the VM itself. When I try to manually mount, I get

Code: [Select]

mount.nfs4: Operation not permittedI've also done a packet capture, it shows two errors, for WAN and LAN the same:

Code: [Select]

ethertype IPv4 (0x0800), length 186: (tos 0x0, ttl 60, id 53543, offset 0, flags [DF], proto TCP (6), length 172)
    136.XX.XX.XX.2049 > 10.10.10.1.18318: Flags [P.], cksum 0xe850 (correct), seq 629:749, ack 1297, win 501, options [nop,nop,TS val 3529236269 ecr 2468818434], length 120: NFS reply xid 961210637 reply ok 116 getattr ERROR: Operation not permittedFor auto-mounting I'm using "autofs" and the options "-fstype=nfs4,rw,retry=0". For the manual test mount, I'm also using nfs4.

I've also experimented with "Firewall" -> "Settings" -> "Normalisation", checked and unchecked "Disable interface scrub", "IP Do-Not-Fragment" and "IP Random id".

And finally, I added opnsense firewall rules like:

- "PASS IN IPv4 * * * 136.XX.XX.XX * * *"
- "PASS IN IPv4 * 136.XX.XX.XX * * * * *"
- "PASS OUT IPv4 * 136.XX.XX.XX * * * * *"

So basically allowing everything IN with destination 136.XX.XX.XX as well as allow everything  IN and OUT with source 136.XX.XX.XX.

But somehow something in opnsense still don't let me mount the nfs share.
Am I missing something?