Help with a comples scenario.

[gilberto.ferreira41]

I have a complex scenario with Proxmox and OpnSense.

I have a VM with Opnsense on Proxmox and OPNsense has 4 network cards:

- vmbr0 -> INTERNET
- vxnet1 -> Internal Private Network on VLAN 500, with IP 172.16.0.0/24 - Gateway 172.16.0.254
- vxnet1 -> Private Internet Network on VLAN 100, with IP 172.17.0.0/24 - Gateway 172.17.0.254

Behind OPNSense I have 4 VMS.

LAN network
Debian-A - VLAN 500 - IP 172.16.0.70
Debian-B - VLAN 500 - IP 172.16.0.71

Gateway in OPNSense for Debian-A and Debian-B is 172.16.0.254

VLAN100 network

Debian-C - VLAN 100 - IP 172.17.0.70
Debian-D - VLAN 100 - IP 172.17.0.71

Gateway in OPNSense for Debian-C and Debian-D is 172.17.0.254

I have already created rules to block all traffic between the LAN network and VLAN network.

It turns out that I need the LAN network 172.16.0.0/24, access port 80 and 3306 on the VLAN network.

I'm not getting it.
And when I do, it flashes.
It accesses 2 or 3 times and then stops.
Any ideas?

Thanks.

[viragomann]

You possibly have an asymmetric routing issue.
Ensure that Proxmox doesn't have an IP in both networks.

For troubleshooting enable logging in your rules an for the default block rule.
Then check the live log for related blocks.

[nero355]

I have already created rules to block all traffic between the LAN network and VLAN network.

And do they work ?!

It turns out that I need the LAN network 172.16.0.0/24, access port 80 and 3306 on the VLAN network.

If you have succesfully blocked traffic between LAN and VLAN (as you call it) then you need to add ALLOW rules ABOVE the BLOCK rules for these two ports.

I'm not getting it.
And when I do, it flashes.
It accesses 2 or 3 times and then stops.

Any ideas?

To be honest I don't understand what you are saying here... :)

You possibly have an asymmetric routing issue.
Ensure that Proxmox doesn't have an IP in both networks.

Not sure why you think that, but if he had those issue they can be solved easily by adding some Policy Based/Source Based Routing Rules to Proxmox :)

[viragomann]

but if he had those issue they can be solved easily by adding some Policy Based/Source Based Routing Rules to Proxmox :)

Best practice is to keep Proxmox out of layer 3 network, apart from its management IP.

[nero355]

Best practice is to keep Proxmox out of layer 3 network, apart from its management IP.

Agree, but just saying that it can be solved if needed for whatever reason... :)

[gilberto.ferreira41]

Ok...

Let's break it down a bit...

I have a Cluster Proxmox, with 3 servers.
On top of this, I have a VXLAN SDN setup, which has the name vxnet2.
No, there is a VM with OPNSense (first image)

As you can see, this VM has 4 NIC.
net0 - vmbr0 is to external access to internet.
net1 - vxnet2 tag 500 is a internet LAN, shared between multiple VMs, which has the IP 172.16.0.0/24
net2 - vmbr0 this another externa access, which is deactivated for now.
net3 - vxnet2 tag 100 is the other LAN, like a DMZ, right? which has the IP 172.17.0.0/24

Now, there 2 debian box behind this OPNsense.

One, let's say Debian-A, is hooked up over net1.
All VMs in this interface, net1, which has the tag 500 on Proxmox/SDN, works fine, can see each other and the gateway 172.16.0.254.
Inside the OPNSense I can ping and do ssh access from and to any 172.16.0.0 IP.
So far so good.

The other Debian machine, let's say Debian-B, is hooked up over net2.
All VMs in this interface, net2, which has the tag 100 on Proxmox/SDN, works fine, can see each other and the gateway 172.17.0.254.
Inside the OPNSense I can ping and do ssh access from and to any 172.17.0.0 IP.
So far so good.

But now, here the problem:

From OPNSense, I can do ssh to Debian-A, 172.16.0.70. GW 172.16.0.254 (OPN)
From OPNSense, I can do ssh to Debian-B, 172.17.0.70. GW 172.17.0.254 (OPN)

But, once inside Debian-A, I can reach Debian-B, and vice-versa.

I already tried to allow everything from LAN (172.16.0) to LAN100 (172.17.0) and vice-versa.
So the ssh works 2 or 3 times, and then is blocked.
Nothing is showing in the live view.

This is the VM using TAG 500 on PVE, which has the IP 172.16.0.70
From this VM I can ping and reach 172.16.0.71, for instance, because is in the same IP/LAN/TAG

This is the other VM using TAG 100 on PVE, which has the IP 172.17.0.70
From this VM I can ping and reach 172.17.0.71, for instance, because is in the same IP/LAN/TAG

I can't do ssh to 172.17.0.70, but I can ping it, both GW and the IP 172.17.0.70.

I hope this can clarify a little bit more.


Thanks