Messages

I migrated my main router from ER-605v1 (TP-link) to OPN virtualized a few months back.
I installed PVE+OPN on my LAN and migrated my VLANs over 1 by 1 until I made the switch.
I've kept some notes and intended to do a post about I but never did. I could still do it if there's interest.

You might want a more straightforward setup to learn.
Testing a 1 NIC setup was on my to-do list so I did that earlier today.
Also on my to-do was a network diagram so I just did that.
You cannot view this attachment.

PVE1 + OPN1 is a reasonably straightforward install.
* PVE vmbr0 for management (on INFRA VLAN via access port on the SWA switch).
* PVE vmbr1 used for WAN (DHCP)
* PVE vmbr2 used as parents for all internal VLANs (includes WAN2, more on that later)

All switches and APs are TP-link Omada devices under a HW controller. TPL VLAN used for management of these.

PVE2 is my spare and I did this earlier today (I already had an OPN setup with WAN + LAN on separate ports, WAN plugged in an access port for the WAN2 VLAN):
* Unplugged WAN & LAN
* Added a VLAN vmbr0.INFRA and moved the static IP from bridge to VLAN.
* Reconfigured the access port on SWA as a trunk => got back into PVE2
* Reset OPN2 to default settings using the console (ends in shutdown)
* Removed vmbr1/vmbr2 (were used as WAN/LAN), added vmbr0 to OPN2 VM
* Restarted the VM and pressed a key for interface assignment
* No LAGG
* Created vtnet0_vlanWAN2 (WAN2 VLAN interface in OPN1 so the WAN side of OPN2 is isolated in my main LAN)
* Created vtnet0_vlanLAN2 (this VLAN is only known to my switches - just VLAN not interface, OPN1 is unaware)
* Assigned WAN to vtnet0_vlanWAN2, LAN to vtnet0_vlanLAN2
* WAN picked up an IP via DHCP (from OPN1)
* LAN got the default static IP
* Configured a port on SWO as an access port for LAN2.
* Connected a test machine that got IP from OPN2.LAN (full Internet connectivity via one hop on OPN1, OPN2 accessible at 192.168.1.1).

Done.
1 NIC virtualized OPN with its own WAN (double NAT but otherwise OK to experiment) and LAN (more VLANs can be added).

Realtek RTL8125 chipset...

So the bridge device shown in the OP is the LAN part of an ERX (which has WAN on eth0?).
And the ERX has an interface for all the VLANs mentioned above and is the GW for all of them.

What kind of routing is expected to happen in OPN when all the VLANs are handled in the ERX?
What are you testing?

Isn't it what the above setup just achieved?
Specify the MAC address range, mark the pool as denied for unknown clients (clients in the address range are known).
Create another pool as fallback for clients that don't match the range.

I suspect you can create a chain of pools this way...

I don't know if it's just me, but this setup is making my head spin.

So you seem to have a PVE host with a single NIC, connected to eth4 of some switch.
PVE Management IP is 10.10.0.2 in VLAN 100 with GW 10.10.0.1

There does not seem to be a way out of the switch for that VLAN.
The OPNsense VM is getting the entire vmbr0 bridge to the single NIC.
There's a vlan100(opt1) interface for that VLAN 100, yet its static IP is 10.10.0.3 (GW is 10.10.0.1).

Where's 10.10.0.1??
With the default route on that interface (Did you get rid of WAN?)! Acting as LAN?

There's a VLAN 10 (you could make your/our lives easier by following some convention for VLAN to subnet mapping) that maybe makes more sense.
eth1 untagged to a management PC.
eth2 untagged to a GW for that subnet? Ubiquiti device? 172.27.201.1, right?
eth4 tagged connected to PVE/OPN
There's an Admin(opt2) interface for that VLAN in OPN. Acting as WAN?
The device name is vlan01, not vtnet0_vlan10 to be consistent with the others? Is it set up the same way?

And then you're trying to open the GUI from a machine on VLAN 10 by connecting to the IP of VLAN 100??
I give up...

I just ran a little test on my spare instance:
* Connected a new machine (PC2) that got IP in the existing pool
* Added the partial MAC address of PC1 to the existing pool
* Checked "deny unknown client" on the existing pool
* Renewed lease on PC2 -> failure (with a weird error message)
* Created a fallback pool on the LAN interface
* Renewed lease on PC2 -> success, obviously in the fallback pool

HTH

Crowdsec on the router is not going to be controversial.
IDS/IPS is more questionable, especially if you're blocking everything on the WAN side (why bother inspecting traffic that is going to be blocked).
Additionally, encrypted traffic can't be inspected anyway. What's left?

Sensei is facing the same issues. Sensei on a TFB behind OPN with Crowdsec and IDS/IPS looks overkill/redundant.
I've seen articles with proxies terminating SSL for inspection purposes. I have not bothered trying, but it could be better use of your HW IMO.

The current date/time is actually displayed in the dashboard, in system information.

The commands are used in a physical terminal on a bare metal install, but virtualized installs offer a terminal too.
In either case, there's ssh access as well.

D'apres https://forum.opnsense.org/index.php?topic=29886.15
C'est correct. Réponse #6 en particulier.
Il y a un certain nombre de chose a faire (tunables) et ne pas faire (IP sur membre de bridge) dans ce post.

Tout le monde ne croit pas vraiment a la valeur de IDS/IPS, surtout quand aucun traffic entrant n'est accepté sur le WAN (en gros, l'IDS observe un traffic qui sera bloqué de toute façon). En plus, comme la plupart du traffic est chiffré (TLS...), l'IDS ne peut rien faire sur le contenu.

Par contre, j'ai vu au moins une video (source respectable) qui utilisait une machine moins puissante sur 5Gbps (voire 10) avec IDS/IPS.
Il etait en mode bridge (transparent filtering bridge) mais je doute que ca change grand chose.
Il etait peut-etre meme virtualisé sous Proxmox (en fait, ca peut aider si les pilotes sont meilleurs sous Debian).

Le ralentissement doit bien se voir sur une ressource qui n'est plus disponible (CPU).
'top' a la console (ou avec SSH) est un bon depart.

Ou alors c'est le test qui ne fonctionne pas comme prevu.
Est-ce tu peux télécharger un contenu important?

Multi WAN setup step 5

Il ne faut pas utiliser la passerelle "internet" pour le traffic local...

Je ne comprends pas la 2eme regle pour chacune des interfaces.
Pour LAN, la 2eme regle a source LAN2_net. Comment un traffic entrant sur LAN pourrait avoir une source dans le subnet LAN2?
Pour LAN2, la premiere regle est plus generale et precede la 2eme. La 2eme n'est sans doute jamais utilisee.

I don't see 8GB of RAM in that output. 2GB is not enough. You end up consuming some swap!

The next step to identify the devices generating interrupts is:
'vmstat -i' (aggregated) and 'systat -vmstat' (live)

I'm still a bit fuzzy with your setup.
Single NIC OPN connected to a switch port (trunk). Another port of the switch is going towards an internet gateway, the others are LAN ports?

A backup server will be connected to one of these LAN ports.
What are you going to backup? The only traffic hitting OPN is LAN <-> WAN.
The rest (apart from super low bandwidth DHCP/DNS/...) is entirely handled by the switch.
Other VMs might use that NIC, but that's also not a concern for OPN.

Oh, I forgot to mention it, your last test shows this:
[  4] local 10.255.1.3 port 59300 connected to 10.255.1.254 port 5201
where other tests had 10.255.0.254 for destination.

If that last test was within VLAN, and the others were not (i.e. inter-VLAN), that could explain the difference.
The VLAN traffic may not touch the router at all.

OPN is running off a single NIC with one VLAN for LAN and another for WAN?
I've never used that config. It's on my to-do...

I run OPNsense (4 cores, 8GB RAM, 64GB disk) virtualized on Proxmox on a N305 fanless NUC.

Looking at top at the console, the system is largely idle (sometimes completely) under background use:

Code Select Expand

last pid: 30707;  load averages:  0.22,  0.23,  0.21                                            up 9+23:37:29  15:00:58
90 processes:  1 running, 89 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

Under 300Mbps load (my ISP max), interrupts go up to ~5%.
Proxmox is pretty lightweight and has reasonable overhead (the above seems to translate to a mix of user/system ~8%).
I have no interest in pushing higher load (would force me to run some inter VLAN arbitrary test).

I'd check where the CPU is going. top is a start.
Do you actually need that kind of bandwidth (LAN/WAN or Inter-VLAN) or is it merely a test?

The homenetworkguy how-to is for a single peer (an iOS phone) to connect via Wireguard on OPN.
This is equivalent to the first link you mentioned.

Because he used a phone, a QR code scan was used to import the conf file.
Instead, copy/paste the config to the left of the QR code, save it to a file. Use that file on your PC.