10Gbe - Core Router with IPS

[PromasterAS203157e]

Hello,

I want to implement OPNSense as Core Router with IPS on VMWare ESXi 6.5 an 10Gbe Hardware.

Target 1: Routing from around 10-15 VLANs
Target 2: Inline IPS (Between Server-VLAN, Client-VLAN, WLAN-VLAN Telephone-VLAN, ...)

Currently I have no hardware but I would like to find out whether it would work like that. The Hardware is not only for the OPNSense, the Hardware is for the Whole Virtual Machines. Now we have around 25 VMs

> Lenovo 3650 M5 512 GB-RAM with 10Gb Eth. Adapters
> VmWare ESXi 6.5
> OPN Sense should mange VLANs with VMXnet3 (Rounting)
> 4 Processors
> 8-16GB RAM
> 60 GB Storage (External on Lenovo V3700 FC with 48x 900Gb)


Questions:
>Works OPNSense fine with VMXnet3?
(I dont want to use the E1000 driver, because i have 10Gbe)
(I dont want to MAP the NIC from the ESXi Host directly to the OPNSense)
> Is it possible to Manage 10-15 VLANs?
> Is for Storage Performance the external Storage enough? - Or is it better to put a little SSD (only for OPNSense) in the VMWare Host?
> Could it be that the OPNSense, would be the bottleneck?

Thanks for Help.

[Ciprian]

Hello!

I have 10 internal VLANS, 2 external (WAN) VLANS with BGP, OPNsense doing all of these in virtual port groups in a virtual switch shared with other VMs. Also, tried OPNsense with E1000 and VMXnet3. Physical clustered host machines HP DL 380 G6.

No problem above!

Still, pay attention to IPS (I also have a few replies on posts and an own post here on the forum about the following): when you activate ruleset "ET Emerging-dos" you will definitely encounter problems regarding RDP (mstsc) connections - very slow connection if any (sometimes it works, sometimes id doesn't), "initiating remote connection" <-> "configuring remote connection" loops etc etc etc. I don't know which, or if more than one of the rules in this ruleset, is causing the problem (didn't have the time to dig), but a simple test I've made was to enable rulesets one by one until problems arose, then disabled all but the last enabled (ET Emerging-dos), problems persisted, then enabled all rulsets but ET Emerging-dos, problems gone.

Also problems with Veeam BKP transfers, but with no conclusion (it's not Emerging-dos here).

RDP or Veeam transfers problems doesn't trigger/fire any rule in the log, so if you intend to use these services (and maybe others - I only encountered problems with only these 2 services), you either use IDS (without IPS), or dig for the culprits on a trial and error one-by-one approach.

Otherwise, Suricata works with little toll on CPU (I recommend Hyperscan engine). Toll on the RAM is in between 500 MB - 1 GB.

[PromasterAS203157e]

Hello hutiucip,

>Do you work with 10GBit Ethernet? Or have you experience OPNSense and 10Gbe?
I want to be sure the 10Gbe is no Problem for the OPNSense with VMXnet3. - (otherwise perhaps a Bottelneck)

>Could it be the problem with the "slow" Remote Desktop Session is a reason from to "slow" Processors or Harddrives?

[Ciprian]

Hello again, Promaster!

>Do you work with 10GBit Ethernet? Or have you experience OPNSense and 10Gbe?
I want to be sure the 10Gbe is no Problem for the OPNSense with VMXnet3. - (otherwise perhaps a Bottelneck)

VMXnet3!

As I have said, HP DL 380 G6, 1 Gb ports with teaming of at least 2 ports for a VLAN, but set on VMXnet3, with no problems, no bottleneck. It will depend on the specific hardware NIC & driver you have regarding IPS (ordering packets in the NIC's queues etc etc etc, which is vital for IPS), and I think the best way for you is to test it only for this (it shoudn't take more than an half an hour to set an OPNsense VM with default settings/ wizard, add an OPT interface, set VLANs on LAN interfaces, activate IPS, and to test the behavior in-between 2 internal LANs using the exact hardware/ drivers you have.

>Could it be the problem with the "slow" Remote Desktop Session is a reason from to "slow" Processors or Harddrives?

Unfortunately, NO! It would have been too easy!

Keeping it short & simple:

1. Enabled All the rulesets in IPS mode -> problems.
2. Disabled All the rulesets -> no problems.
3. Enabled All the rulesets in IDS mode (No IPS) -> no problems?!?!?! (No alerts fired/ triggered in either IDS or IPS mode!!!)
4. Using IPS, enabled rulesets on a one-by-one approach, testing after each enabled ruleset -> problems after enabling "ET Emerging-dos".
5. Disabled ALL other rulesets, but "ET... dos" (maybe too heavy list of all enabled rulesets for the HW I have and VM resources I allocated, I was thinking?!?!? - even if no performance counter exceeded 50%) -> Problems persisted, so no CPU, RAM or HDD issues - rules are loaded and kept in RAM when ID(P)S is activated.
6. Disabled "ET... dos", enabled ALL and EVERY other rulesets -> problem gone, so I concluded again that the list of all rulesets being too heavy was not the case -> particular rule(s) in the "ET... dos" cause(s) this issue.

(Regarding RDP, as I mentioned, I still didn't figure it out for Veeam BKP transfers).

[PromasterAS203157e]

Hello hutiucip,

OK Thanks, for your answer!