Only 23 ethernet interfaces managed ?

[Frankisk]

I'm posting my problem here because I haven't found a case anywhere similar to mine that could have helped me.

To explain this, I created an OPNSense VM in a Hyper-V lab server, as a central firewall/router, between VM servers and clients, to control who can access what.

But with more and more server and client additions, the number of ethernet interfaces has also increased, and I found that only the first 23 interfaces are manageable by the OPNSense web interface.

On the way to the console, I ran the startup messages diagnostic with the "dmesg" command, and got the following results.

For ports 0 to 22, I get:

Code Select Expand


hnXX: <Hyper-V Network Interface> on vmbus0
hnXX: Ethernet address: 00:15:5D:67:86:YY
hnXX: link state changed to UP


For port 23 and the following, I get:

Code Select Expand


hn23: <Hyper-V Network Interface> on vmbus0
hn23: gpadl_conn(chanZZ) failed: 3221225540
hn23: rxbuf gpadl conn failed: 5
device_attach: hn23 attach returned 5


I tried a reinstallation with the ISO file, but the live/install kernel already has the same flaw, and a new installation done with it also.

On the other hand, with a pfSense installation, or even with the FreeBSD distribution, the phenomenon does not manifest itself there, and we can manage more than 23 interfaces without problems.

Is this a limitation provided by OPNSense, which does not exist elsewhere, or a real problem that should not happen?

Thanks for your help.

[Vesalius]

Curious if this limitation is also present for you on 21.7.8 or newly introduced in 22.1?

[Frankisk]

I only recently exceeded the number of 23 interfaces, so I only saw it on the current version.

So, maybe this problem was already present in some previous versions, but I hadn't noticed it before.

[Vesalius]

You have the setup, VM plus configs with # of interfaces to pretty easily test and see if an always been present or 22.1 regression. Might be helpful to the devs, when they get to this during the work week.

[Frankisk]

Following your last remark, I tested a new installation with the ISO file version 21.7.1, which I had in reserve.

I got a difference, which is this time, that the limitation is 32 interfaces.

[franco]

I am suspecting the limit is a system resource, maybe even on the host depending on the HW version that Hyper-V is communicating with FreeBSD:

Code Select Expand

hn23: gpadl_conn(chanZZ) failed: 3221225540
hn23: rxbuf gpadl conn failed: 5
device_attach: hn23 attach returned 5

This is not a soft-coded limit and FreeBSD, pfSense and OPNsense should behave the same under the same conditions: FreeBSD version, host used, number of interfaces assigned from host to any number of VMs left constant.


Cheers,
Franco

[Frankisk]

Having taken note of Franco's remarks, I researched the limitations of Hyper-V, which gave me the following information:

Windows Server 2019 and later supports 68 total:
64 Hyper-V specific network adapters
4 legacy network adapters;

https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/plan/plan-hyper-v-scalability-in-windows-server

So logically the limitation of virtualized hardware is 64 interfaces, and indeed, by pushing to the maximum, I can add on the configurations of generation 2 VMs, up to 64 interfaces.

So I then tested this maximum of 64 interfaces with FreeBSD, pfSense and OPNSense 21.7.1 (as a reminder with 22.1 I only have 23), where I always stay at 32 interfaces.

But then doing the same with a linux distribution, where I used a Debian 11.2 here, I get 49 usable interfaces, then again, with a Windows 10, I then arrive at the 64 usable interfaces of the Hyper-V specification.

So, a purely virtualization host limitation seems strange to me, because it should not logically, in my opinion, give different results between all these OS, but also and especially between versions of OPNSense 21.7.1 and 22.1.

[Frankisk]

I have now updated to 22.1.1_3 and the number of interfaces has increased to 32, as for pfSense and FreeBSD.

This is a first victory, but it remains that it is not possible to exploit, the maximum of 64 interfaces provided by hyper-v, as with a Windows OS, and this remains below the 49 interfaces, as we happen to have with a Linux OS.