Help with a complex scenario. [ SOLVED ]

[gilberto.ferreira41]

I have a complex scenario with Proxmox and OpnSense.

I have a VM with Opnsense on Proxmox and OPNsense has 4 network cards:

- vmbr0 -> INTERNET
- vxnet1 -> Internal Private Network on VLAN 500, with IP 172.16.0.0/24 - Gateway 172.16.0.254
- vxnet1 -> Private Internet Network on VLAN 100, with IP 172.17.0.0/24 - Gateway 172.17.0.254

Behind OPNSense I have 4 VMS.

LAN network
Debian-A - VLAN 500 - IP 172.16.0.70
Debian-B - VLAN 500 - IP 172.16.0.71

Gateway in OPNSense for Debian-A and Debian-B is 172.16.0.254

VLAN100 network

Debian-C - VLAN 100 - IP 172.17.0.70
Debian-D - VLAN 100 - IP 172.17.0.71

Gateway in OPNSense for Debian-C and Debian-D is 172.17.0.254

I have already created rules to block all traffic between the LAN network and VLAN network.

It turns out that I need the LAN network 172.16.0.0/24, access port 80 and 3306 on the VLAN network.

I'm not getting it.
And when I do, it flashes.
It accesses 2 or 3 times and then stops.
Any ideas?

Thanks.

[viragomann]

You possibly have an asymmetric routing issue.
Ensure that Proxmox doesn't have an IP in both networks.

For troubleshooting enable logging in your rules an for the default block rule.
Then check the live log for related blocks.

[nero355]

I have already created rules to block all traffic between the LAN network and VLAN network.

And do they work ?!

It turns out that I need the LAN network 172.16.0.0/24, access port 80 and 3306 on the VLAN network.

If you have succesfully blocked traffic between LAN and VLAN (as you call it) then you need to add ALLOW rules ABOVE the BLOCK rules for these two ports.

I'm not getting it.
And when I do, it flashes.
It accesses 2 or 3 times and then stops.

Any ideas?

To be honest I don't understand what you are saying here... :)

You possibly have an asymmetric routing issue.
Ensure that Proxmox doesn't have an IP in both networks.

Not sure why you think that, but if he had those issue they can be solved easily by adding some Policy Based/Source Based Routing Rules to Proxmox :)

[viragomann]

but if he had those issue they can be solved easily by adding some Policy Based/Source Based Routing Rules to Proxmox :)

Best practice is to keep Proxmox out of layer 3 network, apart from its management IP.

[nero355]

Best practice is to keep Proxmox out of layer 3 network, apart from its management IP.

Agree, but just saying that it can be solved if needed for whatever reason... :)

[gilberto.ferreira41]

Ok...

Let's break it down a bit...

I have a Cluster Proxmox, with 3 servers.
On top of this, I have a VXLAN SDN setup, which has the name vxnet2.
No, there is a VM with OPNSense (first image)

As you can see, this VM has 4 NIC.
net0 - vmbr0 is to external access to internet.
net1 - vxnet2 tag 500 is a internet LAN, shared between multiple VMs, which has the IP 172.16.0.0/24
net2 - vmbr0 this another externa access, which is deactivated for now.
net3 - vxnet2 tag 100 is the other LAN, like a DMZ, right? which has the IP 172.17.0.0/24

Now, there 2 debian box behind this OPNsense.

One, let's say Debian-A, is hooked up over net1.
All VMs in this interface, net1, which has the tag 500 on Proxmox/SDN, works fine, can see each other and the gateway 172.16.0.254.
Inside the OPNSense I can ping and do ssh access from and to any 172.16.0.0 IP.
So far so good.

The other Debian machine, let's say Debian-B, is hooked up over net2.
All VMs in this interface, net2, which has the tag 100 on Proxmox/SDN, works fine, can see each other and the gateway 172.17.0.254.
Inside the OPNSense I can ping and do ssh access from and to any 172.17.0.0 IP.
So far so good.

But now, here the problem:

From OPNSense, I can do ssh to Debian-A, 172.16.0.70. GW 172.16.0.254 (OPN)
From OPNSense, I can do ssh to Debian-B, 172.17.0.70. GW 172.17.0.254 (OPN)

But, once inside Debian-A, I can reach Debian-B, and vice-versa.

I already tried to allow everything from LAN (172.16.0) to LAN100 (172.17.0) and vice-versa.
So the ssh works 2 or 3 times, and then is blocked.
Nothing is showing in the live view.

This is the VM using TAG 500 on PVE, which has the IP 172.16.0.70
From this VM I can ping and reach 172.16.0.71, for instance, because is in the same IP/LAN/TAG

This is the other VM using TAG 100 on PVE, which has the IP 172.17.0.70
From this VM I can ping and reach 172.17.0.71, for instance, because is in the same IP/LAN/TAG

I can't do ssh to 172.17.0.70, but I can ping it, both GW and the IP 172.17.0.70.

I hope this can clarify a little bit more.


Thanks

[viragomann]

Did you enable logging of the default block rule?

BTW:

net0 - vmbr0 is to external access to internet.
net2 - vmbr0 this another externa access, which is deactivated for now.

Two interfaces connected to the same network is a bad idea at all. So even if the second is not in use, you should better remove it from the VM.

[nero355]

But now, here the problem:

From OPNSense, I can do ssh to Debian-A, 172.16.0.70. GW 172.16.0.254 (OPN)
From OPNSense, I can do ssh to Debian-B, 172.17.0.70. GW 172.17.0.254 (OPN)

But, once inside Debian-A, I can reach Debian-B, and vice-versa.

I already tried to allow everything from LAN (172.16.0) to LAN100 (172.17.0) and vice-versa.
So the ssh works 2 or 3 times, and then is blocked.

I can't do ssh to 172.17.0.70, but I can ping it, both GW and the IP 172.17.0.70.

I think you have made a mistake with your Firewall Rules so it would be a good idea to let us know how they look like.

What I would like to know :
When you leave a working SSH session open/IDLE : Does it ever time-out or simply freeze ?!

[gilberto.ferreira41]

When you leave a working SSH session open/IDLE : Does it ever time-out or simply freeze ?!

Nope... But the connection has a lot o delay to be stablished.

See here I need to use control + C 5 times, until got a prompt.
After that, everything is ok.

debian-172-16-0-70:~# ssh administrador@172.17.0.70
^C
debian-172-16-0-70:~# ssh administrador@172.17.0.70
^C
debian-172-16-0-70:~# ssh administrador@172.17.0.70
^C
debian-172-16-0-70:~# ssh administrador@172.17.0.70
^C
debian-172-16-0-70:~# ssh administrador@172.17.0.70
administrador@172.17.0.70's password:
Linux debian-172-17-0-70 6.12.74+deb13+1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.74-2 (2026-03-08) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Mar 24 19:59:16 2026 from 172.16.0.70
administrador@debian-172-17-0-70:~$ su -
Senha:
debian-172-17-0-70:~# journalctl -f
mar 24 20:02:03 debian-172-17-0-70 systemd[1145]: Listening on ssh-agent.socket - OpenSSH Agent socket.
mar 24 20:02:03 debian-172-17-0-70 systemd[1145]: Reached target sockets.target - Sockets.
mar 24 20:02:03 debian-172-17-0-70 systemd[1145]: Reached target basic.target - Basic System.
mar 24 20:02:03 debian-172-17-0-70 systemd[1145]: Reached target default.target - Main User Target.
mar 24 20:02:03 debian-172-17-0-70 systemd[1145]: Startup finished in 253ms.
mar 24 20:02:03 debian-172-17-0-70 systemd[1]: Started user@1000.service - User Manager for UID 1000.
mar 24 20:02:03 debian-172-17-0-70 systemd[1]: Started session-6.scope - Session 6 of User administrador.
mar 24 20:02:07 debian-172-17-0-70 sshd[754]: Timeout before authentication for connection from 172.16.0.70 to 172.17.0.70, pid = 1106
mar 24 20:02:09 debian-172-17-0-70 su[1176]: (to root) administrador on pts/0
mar 24 20:02:09 debian-172-17-0-70 su[1176]: pam_unix(su-l:session): session opened for user root(uid=0) by administrador(uid=1000)
^C

Attache some screenshot regarding Firewall Rules.

[gilberto.ferreira41]

ok Everybody!

I found the goddamn culprit

Since I set it to Never, everything is fine...

Thanks