Support for Intel X553 NIC and Marvell 88E6190 switch

[itbarr]

I have recently acquired a WatchGuard Firebox M270 and would like to run opnsense on it.

The motherboard is a Lanner NCB-WG2511A v0.5.
The processor is Intel(R) Atom(TM) CPU C3558
This includes the Intel X553 integrated dual-port 10 GbE LAN controller (see here: https://www.intel.com/content/www/us/en/products/docs/processors/atom/c-series/atom-c3000-family-programmer-reference-manual.html)
It also appears to have a Marvell 88E6190 switch (see here: https://studylib.net/doc/25668514/88e6390)
This has the following interfaces:
x8: 10/100/1000BASE-T
x2: SERDES (2.5G/1G)
x1: MII/RMII/RGMII

opnsense is detecting the X553

ix0: <Intel(R) X553 (KR Backplane)> mem 0xdfa00000-0xdfbfffff,0xdfc04000-0xdfc07fff irq 16 at device 0.0 on pci6
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 4 RX queues 4 TX queues
ix0: Using MSI-X interrupts with 5 vectors
ix0: allocated for 4 queues
ix0: allocated for 4 rx queues
ix0: Ethernet address: 00:a0:c9:00:00:02
ix0: eTrack 0x80000863
ix0: netmap queues/slots: TX 4/2048, RX 4/2048
ix1: <Intel(R) X553 (KR Backplane)> mem 0xdf800000-0xdf9fffff,0xdfc00000-0xdfc03fff irq 17 at device 0.1 on pci6
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 4 RX queues 4 TX queues
ix1: Using MSI-X interrupts with 5 vectors
ix1: allocated for 4 queues
ix1: allocated for 4 rx queues
ix1: Ethernet address: 34:12:78:56:01:03
ix1: eTrack 0x80000863
ix1: netmap queues/slots: TX 4/2048, RX 4/2048

It is not however detecting the switch. In an ideal world I'd expect to see /dev/etherswitch0 appearing. I don't see any MDIO messages suggesting that this isn't being set up correctly.

The WatchGuard runs software based on Linux (I can share a boot log if it helps to see the devices it detects).
More interestingly it appears that pfSense can be got to run on the device. See here: https://forum.netgate.com/topic/154974/pfsense-on-watchguard-m270

This seems to suggest they have made some changes to the drivers to handle this hardware. i don't however believe those changes are publicly available anywhere, which is a shame. I don't see anything in FreeBSD for example reflecting the hints mentioned in the pfSense post.

Looking at the code I see the following issues:

1. The e6000sw device uses FDT which is not available for amd64. I think I can work around this with some "hacks" using hints instead (similar hints as those mentioned in the pfSense post).
2. The ixgbe device doesn't seem to be setting up the MDIO connection to the switch.

I don't really know what way this should work but as an experiment I added the following code in  /usr/src/sys/dev/ixgbe/if_ix.c at the end of the ixgbe_if_attach_post function. My idea was to use a similar hint as mentioned in the pfSense post hint.mdio.0.at="ix1" to tell the ixgbe driver that it needs to setup the mdio connection to allow the switch to be found.


        if ( 0 == resource_string_value( "mdio", 0, "at", &mdio_dev ) ) {
                sprintf(dev_name, "%s%d", device_get_name(dev), device_get_unit(dev));
                if( 0 == strcmp( mdio_dev, dev_name ) ) {
                        device_printf(dev, "hint found for mdio on %s - adding child device\n", dev_name);
                        child = device_add_child( dev, "mdio", 0 );
                        if (child == NULL) {
                                device_printf(dev, "creating mdio child device failed\n");
                        } else {
                                retval = bus_generic_attach(dev);
                                device_printf(dev, "bus_generic_attach(dev) returned %d\n", retval);
                                retval = bus_generic_attach(child);
                                device_printf(dev, "bus_generic_attach(child) returned %d\n", retval);
                        }

                }

This appears to execute successfully. devinfo -rv is showing mdio0 but it is not attached. It is still not detecting the switch. I put some debug in /usr/src/sys/dev/mdio/mdio.c and it doesn't appear that any of the functions are running e.g. probe, attach, identify.

I'd appreciate any ideas on what needs to be done to get this working!

[itbarr]

Just as an FYI I set up pfSense 23.01 on my appliance. I've captured the relevant part of the boot log to show how it detects the switch. I setup the hints as described in the link my previous post.

ix0: <Intel(R) X553 (KR Backplane)> mem 0xdfa00000-0xdfbfffff,0xdfc04000-0xdfc07fff irq 16 at device 0.0 on pci6
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 4 RX queues 4 TX queues
ix0: Using MSI-X interrupts with 5 vectors
ix0: allocated for 4 queues
ix0: allocated for 4 rx queues
ix0: Ethernet address: 00:a0:c9:00:00:02
ix0: eTrack 0x80000863
ix0: netmap queues/slots: TX 4/2048, RX 4/2048
ix1: <Intel(R) X553 (KR Backplane)> mem 0xdf800000-0xdf9fffff,0xdfc00000-0xdfc03fff irq 17 at device 0.1 on pci6
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 4 RX queues 4 TX queues
ix1: Using MSI-X interrupts with 5 vectors
ix1: allocated for 4 queues
ix1: allocated for 4 rx queues
ix1: Ethernet address: 34:12:78:56:01:03
ix1: eTrack 0x80000863
mdio0: <MDIO> on ix1
e6000sw0: <Marvell 88E6190> on mdio0
e6000sw0: single-chip addressing mode
miibus0: <MII bus> on e6000sw0
e1000phy0: <Marvell 88E1000 Gigabit PHY> PHY 1 on miibus0
e1000phy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
miibus1: <MII bus> on e6000sw0
e1000phy1: <Marvell 88E1000 Gigabit PHY> PHY 2 on miibus1
e1000phy1:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
miibus2: <MII bus> on e6000sw0
e1000phy2: <Marvell 88E1000 Gigabit PHY> PHY 3 on miibus2
e1000phy2:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
miibus3: <MII bus> on e6000sw0
e1000phy3: <Marvell 88E1000 Gigabit PHY> PHY 4 on miibus3
e1000phy3:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
miibus4: <MII bus> on e6000sw0
e1000phy4: <Marvell 88E1000 Gigabit PHY> PHY 5 on miibus4
e1000phy4:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
miibus5: <MII bus> on e6000sw0
e1000phy5: <Marvell 88E1000 Gigabit PHY> PHY 6 on miibus5
e1000phy5:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
miibus6: <MII bus> on e6000sw0
e1000phy6: <Marvell 88E1000 Gigabit PHY> PHY 7 on miibus6
e1000phy6:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
miibus7: <MII bus> on e6000sw0
e1000phy7: <Marvell 88E1000 Gigabit PHY> PHY 8 on miibus7
e1000phy7:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
e6000sw0: switch is ready.
etherswitch0: <Switch controller> on e6000sw0
ix1: netmap queues/slots: TX 4/2048, RX 4/2048

[Patrick M. Hausen]

With a bit of luck you "only" need to compile a custom kernel. Clone the OPNsense source tree from github in a FreeBSD 13.1 based environment (a VM will be enough) or use my Vagrant project for that:

https://github.com/punktDe/vagrant-opnsense

Then modify the kernel config file to contain those lines:

Code: [Select]

           device etherswitch
           device miiproxy
           device iicbus
and build a new kernel.

If successful I doubt @franco & Co. would refuse a feature or even better complete pull request.

I am only wondering why that is not available as a kernel module by default. Going to read some source code, now 

HTH,
Patrick

[itbarr]

Hey Patrick,

I've tried compiling my own kernel with various options enabled. Sadly I'm pretty sure it's not going to work "out of the box" without further changes.

As I mentioned in the original post looking at the code I see the following issues:

1. The e6000sw device uses FDT (Flattened Device Tree) which, unless I'm mistaken, is not available for amd64. I think I can work around this with some "hacks" using hints instead (similar hints as those mentioned in the pfSense post).
2. The ixgbe device doesn't seem to be setting up the MDIO connection to the switch.

Number 2 is the first thing I need to solve.

I found this interesting page: https://wiki.freebsd.org/StefanBethke/EtherSwitch

I've been looking at the code of sys/mips/atheros/if_arge.c and can see lots of MDIO stuff which doesn't seem to have an equivalent in the ixgbe driver.

Regards,
Ivan

[Patrick M. Hausen]

Sorry ... I overlooked that. I was wondering why the module was not built and installed by default.

So there is a driver that is unsupported on the top tier 1 platform but supported ... where? This is weird. Until today I was convinced amd64 > any other platform in terms of features.

Contact the FreeBSD mailing lists. -stable and -net would be the ones I'd try. They are a friendly bunch of people. Don't go for the forum. Managers and regulars consider this mainly as an end user support platform and plain refuse to address "derivatives" like OPNsense or TrueNAS. The mailing lists are where the developers talk and from your post I get that you know what you are doing so you probably won't run into any people related problems.

HTH,
Patrick

[bridledsword]

I'm in the same boat having just acquired a M270. Did you make any progress with this or was it abandoned?
Thank you.

[Patrick M. Hausen]

I did not pursue this any further. I was just trying to assist.

Anyone interested in development in this area should take the discussion to the freebsd-net mailing list, IMHO.

[sl0wfi]

I tried your patch above but it does not seem to work with the current version of the driver.

Code: [Select]

/usr/src/sys/dev/ixgbe/if_ix.c:1139:53: error: use of undeclared identifier 'mdio_dev'

And then falls apart from there. Is there a different place this should be applied?