Ten Gig Interface problems

[deasmi]

I've recently been upgrading to 10G at home in advance of getting my line upgraded to 3gbps symmetric, London Community Fibre.

This has been a pain.

I run Opnsense on a Lenovo ThinkCentre M93p, which has an Intel i5-4570 CPU @ 3.20GHz

To this I added an Intel X520-DA2.

One port is connected to my Netgear switch using a Twinax cable, the other to my ONT using a 10Gtek 10GBASE-T SFP+.

This was unreliable, the Twinax connection was rock solid, the 10GBASE-T module flapping every 30s or so.

So, I got a Broadcom 57810S dual Copper card.

This was unreliable, the Twinax connection was rock solid, the 10GBASE-T port flapping every 5s or so.

So, I got a Intel X540-T2 dual Copper card.

This was unreliable, the Twinax connection was rock solid, the 10GBASE-T port flapping every 5-10m or so. Better, but still not usable as DHCP/DHCPv6 would sometimes drop out.

Currently I have the ONT connected to the Netgear switch using the same 10Gtek SFP+ from above. This is 100% solid, not a single flap in three week which tells me the ONT, cabling and the SFP+ are fine.

I then  run the internet connection into into Opnsense on the twinax as a VLAN.

This works, but I really don't like having the internet on the switch, even as a VLAN, and of course it's less than ideal using up bandwidth on the Twinax link that would be used for inter vlan routing on the home side.

I am at a bit of a loss here, given the SFP+ and ONT clearly like each other, so it's not cabling, why can't I get a stable copper connection on opnsense?

All of the above have been tested with hardware offload on/off.

Any ideas gratefully received.

Thanks

[Greg_E]

I have one of those 10Gtek modules for connection between Mikrotik switch and my NAS, no problems with it in this use case, so I doubt the module is the issue.

But is it possible that there is a power or heat problem with that module in the Lenovo Tiny, the 10gbase-T stuff gets HOT, hot mean power draw.

I haven't had a lot of luck with Broadcom cards except for the HP modules that went in some of my servers.

[pfry]

I'd expect overheating to be more likely in the switch, unless it's a big Netgear with aggressive fans. The native twisted-pair cards should be fine, too, unless everything's cooking... which, given the power draw of all of those cards, is a slight possibility (the transceiver silicon can take some heat). Some of them may export thermal sensor data (via ACPI), but the system BIOS may not have support and/or it might be tough to read in FreeBSD.

I'd look at the OPNsense logs, as I'd tend to suspect a higher-level/layer issue.

[deasmi]

I'd expect overheating to be more likely in the switch, unless it's a big Netgear with aggressive fans. The native twisted-pair cards should be fine, too, unless everything's cooking... which, given the power draw of all of those cards, is a slight possibility (the transceiver silicon can take some heat). Some of them may export thermal sensor data (via ACPI), but the system BIOS may not have support and/or it might be tough to read in FreeBSD.

I'd look at the OPNsense logs, as I'd tend to suspect a higher-level/layer issue.

The logs show kernel level 'interface down' messages, and then all the expected things due to that. Very little help there, but that leads me to suspect it's not a higher level issue.

I guess I might need to try some other hardware.

[Greg_E]

Do you have a wifi a+e card slot in that computer? You might try one of the Intel i226-v a+e cards in there to see if you can at least get a stable connection. Only 2.5gbps but you might at least get somewhere. I have a bunch of these cards installed, but only using them at gigabit right now https://www.ebay.com/itm/365139931898 you should be able to find them from other sellers closer to your location or direct from China.

Is it possible the ONT just doesn't work with direct connections like this, or that the speed is not auto-negotiating properly with the cards?

What speed shows on the switch when you connect it that way?

Hacky solution... Get a small 10gbe switch to "translate" between ONT and your firewall?