Issues with OPNsense on VM in Proxmox on Fujitsu S920

[kubatron]

Hello,

I have a Fujitsu S920 device with a built-in 1x1GB Ethernet card and an additional PCI Express card with 2x1Gb ports. I installed Proxmox on the Fujitsu S920 to manage all services, and I have set up OPNsense as the main firewall that should handle all traffic from the internet to the LAN.

I configured bridges in Proxmox associated with all ports without assigning them IP addresses. The configuration for OPNsense is as follows:

•     enp0s0 -> vmbr0 as WAN -> vtnet0 (OPNsense)

    This connection goes from my fiber modem (RJ45) to the Fujitsu S920.

•     enp1s0f1 -> vmbr1 as VLAN (LAN) -> vtnet1 (OPNsense)

    This connection goes from the Fujitsu S920 (RJ45) to the TP Link SG108PE switch (management) with DHCP enabled.

•     enp0s0f1 -> vmbr2 as MGMT -> vtnet2 (OPNsense)

    This is set up in Proxmox with IP 192.168.1.2/24 and gateway: 192.168.1.1.

---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----

• WAN (vtnet0) in OPNsense received an address from my ISP:

    IP: 10.0.xxx.90/24
    Gateway: 10.0.xxx.1

• LAN (vtnet1) in OPNsense received DHCP and has the address:

    192.168.1.1/24

• MGMT (vtnet2) in OPNsense is currently not receiving any address.

Network Topology Diagrams
---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----

Diagram 1: Proxmox + OPNsense (without LAGG)


Diagram 2: Proxmox Bridges - OPNsense (in Proxmox)

FIREWALL is not disabled



My problem is that I cannot configure Proxmox and OPNsense correctly to work like my current simple setup:
ISP Modem -> TP Link ER605v2 (OpenWRT) -> TP Link SG108PE Switch -> IoT, PC, WiFi
In this setup, everything works without any issues.

In OPNsense, I diagnosed the problem through ping and was able to access WAN at some point, but only through ping. I checked NAT, firewall rules, and gateways, but I do not know where the problem lies.

I have read many threads on the OPNsense forum, Proxmox forum, and other services, as well as watched instructional videos on configuring, but unfortunately, nothing seems to work.

When connecting without OPNsense and only the MGMT cable with the assigned address of 192.168.1.2/24 in Proxmox, an IP address is assigned by the TP Link SG108PE and the TP Link ER605v2 (OpenWRT). However, when I disconnect the TP Link ER605v2 and replace it with the Fujitsu S920, the configuration does not work. It seems like the TP Link SG108PE switch treats the Fujitsu S920 as a PC and tries to assign it an IP address, while Proxmox with OPNsense does not function as a router or DHCP server.

I apologize in advance if I have frustrated anyone or if this issue has been covered elsewhere, but I genuinely appreciate any help you can provide.
Screenshots from Proxmox


Additionally, I would like the port labeled VLAN to function as a VLAN in the future, but I won't address that yet, even though I considered enabling VLAN on the TP Link SG108PE switch.

[meyergru]

For starters, both MGMT and LAN have 192.168.1.0/24 according to your report.

[kubatron]

For starters, both MGMT and LAN have 192.168.1.0/24 according to your report.

I forgot mention that on the screeen from Proxmox -> pve -> Network , there is no setup any IP and gateway on the vmbr1 (VLANs_ETH0) - I setup IP and gateway for VLANs_ETH0 on the OPNsense and on WAN the same, so only is setup on the MAGMT_ETH1 setup 192.168.1.178/24.

If someone can explain me how should I setup that ports, when I try many options no works...

Is there any point that I am stupid and don't understand how this works ? Or I do some stupid mistakes that is so easy to fix...

I try remove all IP's from Proxmox, but then I lose access to Proxmox via eth cable...I try only with IP on the managment port for accessing Proxmox, but then I cannot property configure OPNsense and switch.

[sc00by1984]

i would try the following:
- only assign one IP to the proxmox host, actually you have 2
- work with different subnets, not 2 times 192.168.1.x. This will not work and makes everything harder to troubleshoot.

[meyergru]

I am a little hesitant to point you anywhere, because you obviously try to manage quite a zoo of complex topics there.

The specification of vtnet0 tells me that your image above is NOT the full story. Obviously, your OpnSense is a VM on Proxmox itself.

There is a guide on how to do (just) that: https://forum.opnsense.org/index.php?topic=44159.0

However, the fact that you showed the 192.168.1.0/24 subnet on different interfaces tells me that you failed to understand that you cannot do that and still expect to have working routing in the first place, as shown here: https://forum.opnsense.org/index.php?topic=42985.0, point 1. Apart from that, you should not be using 192.168.1.0/24 at all, see this for why: https://forum.opnsense.org/index.php?topic=47099.0

Then, your "modem" is probably not a bridge only, but a router, where this would become a router-behind-router scenario, like discussed here: https://forum.opnsense.org/index.php?topic=42985.0, point 4.

And if OpnSense on Proxmox was not complex enough in the first place, you combine that with VLANs, which can be handled on Proxmox or OpnSense (or both) and of course must be configured correctly on your switch, too. There are quite a lot of obstacles with that, too, even before considering OpnSense setup (like https://forum.opnsense.org/index.php?topic=42985.0, point 3).

You say you have parts of that running already, but IDK what exactly and your diagram shows only part of the story, so it is hard to help here, maybe someone can step in...

[nero355]

Is there any point that I am stupid and don't understand how this works ? Or I do some stupid mistakes that is so easy to fix...

You are not stupid, but you are simply missing knowledge/experience I think :)

For example in this case I would never use this kind of hardware for Proxmox : Just baremetal OPNsense.

IIRC most people use these Fujitsu things as following :
- Onboard NIC as Management Network.
- Additional 2-port or Quad Port NIC for actual Router/Firewall/VLAN stuff.



Should you ever buy something Intel based and more powerfull then I would recommend to VT-D the additional NIC to your Proxmox VM with OPNsense so you can avoid all the Bridging stuff :)

[Kinerg]

I forgot mention that on the screeen from Proxmox -> pve -> Network , there is no setup any IP and gateway on the vmbr1 (VLANs_ETH0) - I setup IP and gateway for VLANs_ETH0 on the OPNsense and on WAN the same, so only is setup on the MAGMT_ETH1 setup 192.168.1.178/24.

This is confusing. You should only have IP and gateway on vmbr2 MAGMT (and not the same as LAN 192.168.1.1/24, as others have said), leave the other two blank.

It is also not clear how many DHCP servers you have on the network.

Is the TP Link SG108PE port towards Fujitsu S920 set up as trunk?

How are your interfaces set up in OPNsense?