Issues with OPNsense on VM in Proxmox on Fujitsu S920

[kubatron]

Hello,

I have a Fujitsu S920 device with a built-in 1x1GB Ethernet card and an additional PCI Express card with 2x1Gb ports. I installed Proxmox on the Fujitsu S920 to manage all services, and I have set up OPNsense as the main firewall that should handle all traffic from the internet to the LAN.

I configured bridges in Proxmox associated with all ports without assigning them IP addresses. The configuration for OPNsense is as follows:

•     enp0s0 -> vmbr0 as WAN -> vtnet0 (OPNsense)

    This connection goes from my fiber modem (RJ45) to the Fujitsu S920.

•     enp1s0f1 -> vmbr1 as VLAN (LAN) -> vtnet1 (OPNsense)

    This connection goes from the Fujitsu S920 (RJ45) to the TP Link SG108PE switch (management) with DHCP enabled.

•     enp0s0f1 -> vmbr2 as MGMT -> vtnet2 (OPNsense)

    This is set up in Proxmox with IP 192.168.1.2/24 and gateway: 192.168.1.1.

---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----

• WAN (vtnet0) in OPNsense received an address from my ISP:

    IP: 10.0.xxx.90/24
    Gateway: 10.0.xxx.1

• LAN (vtnet1) in OPNsense received DHCP and has the address:

    192.168.1.1/24

• MGMT (vtnet2) in OPNsense is currently not receiving any address.

Network Topology Diagrams
---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----

Diagram 1: Proxmox + OPNsense (without LAGG)


Diagram 2: Proxmox Bridges - OPNsense (in Proxmox)

FIREWALL is not disabled



My problem is that I cannot configure Proxmox and OPNsense correctly to work like my current simple setup:
ISP Modem -> TP Link ER605v2 (OpenWRT) -> TP Link SG108PE Switch -> IoT, PC, WiFi
In this setup, everything works without any issues.

In OPNsense, I diagnosed the problem through ping and was able to access WAN at some point, but only through ping. I checked NAT, firewall rules, and gateways, but I do not know where the problem lies.

I have read many threads on the OPNsense forum, Proxmox forum, and other services, as well as watched instructional videos on configuring, but unfortunately, nothing seems to work.

When connecting without OPNsense and only the MGMT cable with the assigned address of 192.168.1.2/24 in Proxmox, an IP address is assigned by the TP Link SG108PE and the TP Link ER605v2 (OpenWRT). However, when I disconnect the TP Link ER605v2 and replace it with the Fujitsu S920, the configuration does not work. It seems like the TP Link SG108PE switch treats the Fujitsu S920 as a PC and tries to assign it an IP address, while Proxmox with OPNsense does not function as a router or DHCP server.

I apologize in advance if I have frustrated anyone or if this issue has been covered elsewhere, but I genuinely appreciate any help you can provide.
Screenshots from Proxmox


Additionally, I would like the port labeled VLAN to function as a VLAN in the future, but I won't address that yet, even though I considered enabling VLAN on the TP Link SG108PE switch.

[meyergru]

For starters, both MGMT and LAN have 192.168.1.0/24 according to your report.

[kubatron]

For starters, both MGMT and LAN have 192.168.1.0/24 according to your report.

I forgot mention that on the screeen from Proxmox -> pve -> Network , there is no setup any IP and gateway on the vmbr1 (VLANs_ETH0) - I setup IP and gateway for VLANs_ETH0 on the OPNsense and on WAN the same, so only is setup on the MAGMT_ETH1 setup 192.168.1.178/24.

If someone can explain me how should I setup that ports, when I try many options no works...

Is there any point that I am stupid and don't understand how this works ? Or I do some stupid mistakes that is so easy to fix...

I try remove all IP's from Proxmox, but then I lose access to Proxmox via eth cable...I try only with IP on the managment port for accessing Proxmox, but then I cannot property configure OPNsense and switch.

[sc00by1984]

i would try the following:
- only assign one IP to the proxmox host, actually you have 2
- work with different subnets, not 2 times 192.168.1.x. This will not work and makes everything harder to troubleshoot.

[meyergru]

I am a little hesitant to point you anywhere, because you obviously try to manage quite a zoo of complex topics there.

The specification of vtnet0 tells me that your image above is NOT the full story. Obviously, your OpnSense is a VM on Proxmox itself.

There is a guide on how to do (just) that: https://forum.opnsense.org/index.php?topic=44159.0

However, the fact that you showed the 192.168.1.0/24 subnet on different interfaces tells me that you failed to understand that you cannot do that and still expect to have working routing in the first place, as shown here: https://forum.opnsense.org/index.php?topic=42985.0, point 1. Apart from that, you should not be using 192.168.1.0/24 at all, see this for why: https://forum.opnsense.org/index.php?topic=47099.0

Then, your "modem" is probably not a bridge only, but a router, where this would become a router-behind-router scenario, like discussed here: https://forum.opnsense.org/index.php?topic=42985.0, point 4.

And if OpnSense on Proxmox was not complex enough in the first place, you combine that with VLANs, which can be handled on Proxmox or OpnSense (or both) and of course must be configured correctly on your switch, too. There are quite a lot of obstacles with that, too, even before considering OpnSense setup (like https://forum.opnsense.org/index.php?topic=42985.0, point 3).

You say you have parts of that running already, but IDK what exactly and your diagram shows only part of the story, so it is hard to help here, maybe someone can step in...

[nero355]

Is there any point that I am stupid and don't understand how this works ? Or I do some stupid mistakes that is so easy to fix...

You are not stupid, but you are simply missing knowledge/experience I think :)

For example in this case I would never use this kind of hardware for Proxmox : Just baremetal OPNsense.

IIRC most people use these Fujitsu things as following :
- Onboard NIC as Management Network.
- Additional 2-port or Quad Port NIC for actual Router/Firewall/VLAN stuff.



Should you ever buy something Intel based and more powerfull then I would recommend to VT-D the additional NIC to your Proxmox VM with OPNsense so you can avoid all the Bridging stuff :)

[Kinerg]

I forgot mention that on the screeen from Proxmox -> pve -> Network , there is no setup any IP and gateway on the vmbr1 (VLANs_ETH0) - I setup IP and gateway for VLANs_ETH0 on the OPNsense and on WAN the same, so only is setup on the MAGMT_ETH1 setup 192.168.1.178/24.

This is confusing. You should only have IP and gateway on vmbr2 MAGMT (and not the same as LAN 192.168.1.1/24, as others have said), leave the other two blank.

It is also not clear how many DHCP servers you have on the network.

Is the TP Link SG108PE port towards Fujitsu S920 set up as trunk?

How are your interfaces set up in OPNsense?

[kubatron]

I forgot mention that on the screeen from Proxmox -> pve -> Network , there is no setup any IP and gateway on the vmbr1 (VLANs_ETH0) - I setup IP and gateway for VLANs_ETH0 on the OPNsense and on WAN the same, so only is setup on the MAGMT_ETH1 setup 192.168.1.178/24.

This is confusing. You should only have IP and gateway on vmbr2 MAGMT (and not the same as LAN 192.168.1.1/24, as others have said), leave the other two blank.

It is also not clear how many DHCP servers you have on the network.

Is the TP Link SG108PE port towards Fujitsu S920 set up as trunk?

How are your interfaces set up in OPNsense?

The other two ports in Proxmox is blank, only a MGMT port is setup.

ISP Modem (WAN) is going cable to Fujitsu S920 to port vmbr0 (WAN - blank in Proxmox, setup in OPNsense)
From Fujitsu S920 via cable is going vmbr1 (LAN - blank in Proxmox, setup in OPNsense + DHCP server) to TP Link SG108PE
In TP Link SG108PE I have disabled all VLAN's option, I can enable VLAN and TAG/UNTAG specified port.

I try many things and now I am stuck...I don't know where is issue.
I disable in OPNsense (VM) firewall option, I try with NAT outbound and switch to hybrid, I try pfctl -d. I try allow any traffic rules in OPNsense firewall.

This is very weird, I watch many videos/read articles and...

Code Select Expand

auto lo
iface lo inet loopback

iface enp2s0 inet manual
#FUJITSU BUILD-IN-ETH

iface enp1s0f0 inet manual
#Intel [EXT-ETH0]

iface enp1s0f1 inet manual
#Intel [EXT-ETH1]

iface wlp3s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.200/24
        gateway 192.168.1.1
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0
#[ETH0-F] MGMT

auto vmbr1
iface vmbr1 inet manual
        bridge-ports enp1s0f0
        bridge-stp off
        bridge-fd 0
#WAN

auto vmbr2
iface vmbr2 inet manual
        bridge-ports enp1s0f1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4096
#LAN

source /etc/network/interfaces.d/*ISP Modem -> Fujitsu S920 (Proxmox + OPNsense as a DHCP on vmbr1 (LAN) and vmbr0 (WAN - static IP from ISP (10.0.x.xxx/24, 10.0.x.1) ) -> vmbr1 (LAN) -> TP Link SG108PE port, and vmbr0 (WAN) is connected to Fujitsu S920