No Internet Access - OPNsense on VMware

[Faisal_Biyari]

TL;DR
Problem: Device(s) on LAN have no access to Internet

• OPNsense 24.1.7_4 installed on VMware ESXi 7.0.3
• 2 ethernet ports confirgured, for LAN & WAN
• Topology: Router (DHCP) > OPNsense (DHCP Server, 10.29.251.0/24) > One Single User Device (Currently)
• OPNsense can ping 8.8.8.8
• OPNsense can ping user device on 10.29.251.10
• OPNsense CANNOT ping 8.8.8.8 with source set to 10.29.251.10

I tried default settings, reset to factory settings and just used those, tried playing with the gateway, interfaces (Including unblocking private networks), firewall rules, and endless online videos, reinstalled OPNsense, installed os-vmware plugin, increased virtual machine resources...
Only thing I have not done is install OPNsense on baremetal.

Virtual Machine Specs:
CPU: 4 Cores, 2.70 Ghz (XEON Processor)
RAM: 8GB RAM
Storage: 120 GB
Network Adapter: 2 Adapters, set to VMXnet3

Also tried increasing resources to 8 cores, 12 GB RAM, as well as changing VMXnet3 to e1000.

Everything tells me "It should just work", but it's not, and I can't figure out why.
I honestly don't understand the concept of gateway or firewall rules very well, and keep blaming them.
Any help would be greatly appreciated.

P.S.
My end goal is to connect 4 different WAN Routers, setup Load Balancing on OPNsense, and connect to my switch, serving my entire establishment.

[netnut]

• OPNsense can ping 8.8.8.8
• OPNsense can ping user device on 10.29.251.10
• OPNsense CANNOT ping 8.8.8.8 with source set to 10.29.251.10

So you have a device with 10.29.251.10 connected to OPNsense, what about that device itself, can it ping 8.8.8.8 through OPNsense ?

If you try to ping _from_ OPNsense _to_ 8.8.8.8 with a source address set, you need to set the address to an OPNsense assigned interface address (something like 10.29.251.1 or 10.29.251.254, depending on your config). You can't ping "on behalf of" some other device no mather if it's connected or not.

[dishtix]

10.29.251.10 isnt bound to any interface on opnsense, thus why would that work.
assuming 10.29.251.1 is bound to the LAN interface you can use that

[Faisal_Biyari]

Thank you for your time 

So you have a device with 10.29.251.10 connected to OPNsense, what about that device itself, can it ping 8.8.8.8 through OPNsense ?

No, the device on 10.29.251.10 cannot ping/reach 8.8.8.8

If you try to ping _from_ OPNsense _to_ 8.8.8.8 with a source address set, you need to set the address to an OPNsense assigned interface address (something like 10.29.251.1 or 10.29.251.254, depending on your config). You can't ping "on behalf of" some other device no mather if it's connected or not.

OPNsense LAN port is assigned 10.29.251.203
You're saying I can only ping from 10.29.251.203 as a source address, and not any IP in the range of that OPNsense DHCP server?

I appreciate the information. I have an L3 switch which I use to ping using any source that is directly connected to it. I expected OPNsense to be the same; that's on me.

10.29.251.10 isnt bound to any interface on opnsense, thus why would that work.
assuming 10.29.251.1 is bound to the LAN interface you can use that

10.29.251.10 is a device on the LAN of OPNsense (OPNsense is the DHCP server providing this IP address)
I could try to ping 8.8.8.8 with the source being the LAN bound address (In this case, it is 10.29.251.203)


Update:
I reached a level where I assumed the problem lies in VMware.
I changed the interfaces to passthrough directly to OPNsense.
I faced a another issue with VMware not allowing this to occur, and bypassed that by disabling ACS checks.
I continued to face the same issues with OPNsense, more or less, with no internet reaching LAN.

Finally, I replaced the entire hardware with an old i5 desktop, moved the 10g ethernet boards there, and installed OPNsense on baremetal, and it just worked out of the box. (I'm actually replying using this connection)
Unfortunately, I can't have 4 WANs + 1 LAN on this device, for my load balancing schemes...

I'm still not sure where the problem was, but it seems to me that OPNsense and VMware are not as compatible out of the box. (I tried pfSense as well, which gave me pretty much the same results on VMware)
If you guys, or anyone, can suggest other ideas, I would very much appreciate it. 

[netnut]

You're saying I can only ping from 10.29.251.203 as a source address, and not any IP in the range of that OPNsense DHCP server?

If you're using the INTERFACES: DIAGNOSTICS: PING menu or OPNsense CLI, yes, that's how PING works. Again, PING doesn't work "on behalf of" some other device.

I'm still not sure where the problem was, but it seems to me that OPNsense and VMware are not as compatible out of the box. (I

OPNsense works perfectly with VMware, KVM, Proxmox or whatever HyperVisor you have. But when using virtualization you need to configure the virtual networks / underlay (bridges. sdn etc.) correctly. Roughly 99% of "OPNsense Problems" with virtualization posted here are about wrongly configured virtual networks, so focus on this part, OPNsense will work out of the box...

[Faisal_Biyari]

OPNsense works perfectly with VMware, KVM, Proxmox or whatever HyperVisor you have. But when using virtualization you need to configure the virtual networks / underlay (bridges. sdn etc.) correctly. Roughly 99% of "OPNsense Problems" with virtualization posted here are about wrongly configured virtual networks, so focus on this part, OPNsense will work out of the box...

Makes sense. I'm pretty new to this.
Thank you for your support.

Can you kindly share any guides I could use for VMware setup to be ready for OPNsense?

[dishtix]

10.29.251.10 is a device on the LAN of OPNsense (OPNsense is the DHCP server providing this IP address)
I could try to ping 8.8.8.8 with the source being the LAN bound address (In this case, it is 10.29.251.203)
• OPNsense CANNOT ping 8.8.8.8 with source set to 10.29.251.10"

I repeat... 10.29.251.10 isnt bound on an OPNsense interface you cannot simply spoof a LAN ip. Why dont you ping from 10.29.251.1 (your opnsense gw??) or use that host 10.29.251.10 to ping?

I'm not familiar with ESXI but i use proxmox on many devices.
I just finished fresh VM installation of OPNsense, 24.7 didnt workout thus went back to 24.1.10 for now.

Your setup is simple, there's no reason it wouldnt work. You're missing something obvious.. 
By default any host on OPNsense's LAN that got its dhcp from OPNsense, will be able to access WAN/internet if its up.

Since your WAN gives a local IP range, disable block private networks.
With VMXnet3 you can build your VMs on top of that interface, which I assume its for lab testing if you want to do failover. You may have read, ideally should use vt-d and dedicate LAN/WAN NICs to the firewall for best the performance and least overhead.
Add 3rd NIC using VMXnet3 as your DMZ or VM network, just a bridge without anyvirtual port and doesnt bind to any physical port.

For lab testing failover etc, VMXnet3 as configured should work as well.. Is your RTR using VLAN to access its internet? I think you need to manually set NAT rules as the default on WAN wouldnt work. Instead do it on the wanVLAN.

Let OPNsense do the DHCP to avoid complications.
- On a fresh install there is a LAN to any rule
- Add NAT onto the correct WAN interface.

[zz00mm]

Search thru the forum on VMware and/or ESX and you'll find many other posts. If you are creating a none HA firewall. you need to modify your PortGroup and enable Promiscuous mode, MAC address changes and Forged transmits.

If you are attempting to create an HA pair, the only way I've been successful to get an HA pair to work is use the same options with a vDS (Distributed Switch). Even then I finally had to tag the WAN side to get it to work.

Hopefully this helps, attaching a screen shot of the "Security section" where the above options are disabled by default.