1
Development and Code Review / Support for Intel X553 NIC and Marvell 88E6190 switch
« on: March 13, 2023, 12:57:32 pm »
I have recently acquired a WatchGuard Firebox M270 and would like to run opnsense on it.
The motherboard is a Lanner NCB-WG2511A v0.5.
The processor is Intel(R) Atom(TM) CPU C3558
This includes the Intel X553 integrated dual-port 10 GbE LAN controller (see here: https://www.intel.com/content/www/us/en/products/docs/processors/atom/c-series/atom-c3000-family-programmer-reference-manual.html)
It also appears to have a Marvell 88E6190 switch (see here: https://studylib.net/doc/25668514/88e6390)
This has the following interfaces:
x8: 10/100/1000BASE-T
x2: SERDES (2.5G/1G)
x1: MII/RMII/RGMII
opnsense is detecting the X553
ix0: <Intel(R) X553 (KR Backplane)> mem 0xdfa00000-0xdfbfffff,0xdfc04000-0xdfc07fff irq 16 at device 0.0 on pci6
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 4 RX queues 4 TX queues
ix0: Using MSI-X interrupts with 5 vectors
ix0: allocated for 4 queues
ix0: allocated for 4 rx queues
ix0: Ethernet address: 00:a0:c9:00:00:02
ix0: eTrack 0x80000863
ix0: netmap queues/slots: TX 4/2048, RX 4/2048
ix1: <Intel(R) X553 (KR Backplane)> mem 0xdf800000-0xdf9fffff,0xdfc00000-0xdfc03fff irq 17 at device 0.1 on pci6
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 4 RX queues 4 TX queues
ix1: Using MSI-X interrupts with 5 vectors
ix1: allocated for 4 queues
ix1: allocated for 4 rx queues
ix1: Ethernet address: 34:12:78:56:01:03
ix1: eTrack 0x80000863
ix1: netmap queues/slots: TX 4/2048, RX 4/2048
It is not however detecting the switch. In an ideal world I'd expect to see /dev/etherswitch0 appearing. I don't see any MDIO messages suggesting that this isn't being set up correctly.
The WatchGuard runs software based on Linux (I can share a boot log if it helps to see the devices it detects).
More interestingly it appears that pfSense can be got to run on the device. See here: https://forum.netgate.com/topic/154974/pfsense-on-watchguard-m270
This seems to suggest they have made some changes to the drivers to handle this hardware. i don't however believe those changes are publicly available anywhere, which is a shame. I don't see anything in FreeBSD for example reflecting the hints mentioned in the pfSense post.
Looking at the code I see the following issues:
1. The e6000sw device uses FDT which is not available for amd64. I think I can work around this with some "hacks" using hints instead (similar hints as those mentioned in the pfSense post).
2. The ixgbe device doesn't seem to be setting up the MDIO connection to the switch.
I don't really know what way this should work but as an experiment I added the following code in /usr/src/sys/dev/ixgbe/if_ix.c at the end of the ixgbe_if_attach_post function. My idea was to use a similar hint as mentioned in the pfSense post hint.mdio.0.at="ix1" to tell the ixgbe driver that it needs to setup the mdio connection to allow the switch to be found.
if ( 0 == resource_string_value( "mdio", 0, "at", &mdio_dev ) ) {
sprintf(dev_name, "%s%d", device_get_name(dev), device_get_unit(dev));
if( 0 == strcmp( mdio_dev, dev_name ) ) {
device_printf(dev, "hint found for mdio on %s - adding child device\n", dev_name);
child = device_add_child( dev, "mdio", 0 );
if (child == NULL) {
device_printf(dev, "creating mdio child device failed\n");
} else {
retval = bus_generic_attach(dev);
device_printf(dev, "bus_generic_attach(dev) returned %d\n", retval);
retval = bus_generic_attach(child);
device_printf(dev, "bus_generic_attach(child) returned %d\n", retval);
}
}
This appears to execute successfully. devinfo -rv is showing mdio0 but it is not attached. It is still not detecting the switch. I put some debug in /usr/src/sys/dev/mdio/mdio.c and it doesn't appear that any of the functions are running e.g. probe, attach, identify.
I'd appreciate any ideas on what needs to be done to get this working!
The motherboard is a Lanner NCB-WG2511A v0.5.
The processor is Intel(R) Atom(TM) CPU C3558
This includes the Intel X553 integrated dual-port 10 GbE LAN controller (see here: https://www.intel.com/content/www/us/en/products/docs/processors/atom/c-series/atom-c3000-family-programmer-reference-manual.html)
It also appears to have a Marvell 88E6190 switch (see here: https://studylib.net/doc/25668514/88e6390)
This has the following interfaces:
x8: 10/100/1000BASE-T
x2: SERDES (2.5G/1G)
x1: MII/RMII/RGMII
opnsense is detecting the X553
ix0: <Intel(R) X553 (KR Backplane)> mem 0xdfa00000-0xdfbfffff,0xdfc04000-0xdfc07fff irq 16 at device 0.0 on pci6
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 4 RX queues 4 TX queues
ix0: Using MSI-X interrupts with 5 vectors
ix0: allocated for 4 queues
ix0: allocated for 4 rx queues
ix0: Ethernet address: 00:a0:c9:00:00:02
ix0: eTrack 0x80000863
ix0: netmap queues/slots: TX 4/2048, RX 4/2048
ix1: <Intel(R) X553 (KR Backplane)> mem 0xdf800000-0xdf9fffff,0xdfc00000-0xdfc03fff irq 17 at device 0.1 on pci6
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 4 RX queues 4 TX queues
ix1: Using MSI-X interrupts with 5 vectors
ix1: allocated for 4 queues
ix1: allocated for 4 rx queues
ix1: Ethernet address: 34:12:78:56:01:03
ix1: eTrack 0x80000863
ix1: netmap queues/slots: TX 4/2048, RX 4/2048
It is not however detecting the switch. In an ideal world I'd expect to see /dev/etherswitch0 appearing. I don't see any MDIO messages suggesting that this isn't being set up correctly.
The WatchGuard runs software based on Linux (I can share a boot log if it helps to see the devices it detects).
More interestingly it appears that pfSense can be got to run on the device. See here: https://forum.netgate.com/topic/154974/pfsense-on-watchguard-m270
This seems to suggest they have made some changes to the drivers to handle this hardware. i don't however believe those changes are publicly available anywhere, which is a shame. I don't see anything in FreeBSD for example reflecting the hints mentioned in the pfSense post.
Looking at the code I see the following issues:
1. The e6000sw device uses FDT which is not available for amd64. I think I can work around this with some "hacks" using hints instead (similar hints as those mentioned in the pfSense post).
2. The ixgbe device doesn't seem to be setting up the MDIO connection to the switch.
I don't really know what way this should work but as an experiment I added the following code in /usr/src/sys/dev/ixgbe/if_ix.c at the end of the ixgbe_if_attach_post function. My idea was to use a similar hint as mentioned in the pfSense post hint.mdio.0.at="ix1" to tell the ixgbe driver that it needs to setup the mdio connection to allow the switch to be found.
if ( 0 == resource_string_value( "mdio", 0, "at", &mdio_dev ) ) {
sprintf(dev_name, "%s%d", device_get_name(dev), device_get_unit(dev));
if( 0 == strcmp( mdio_dev, dev_name ) ) {
device_printf(dev, "hint found for mdio on %s - adding child device\n", dev_name);
child = device_add_child( dev, "mdio", 0 );
if (child == NULL) {
device_printf(dev, "creating mdio child device failed\n");
} else {
retval = bus_generic_attach(dev);
device_printf(dev, "bus_generic_attach(dev) returned %d\n", retval);
retval = bus_generic_attach(child);
device_printf(dev, "bus_generic_attach(child) returned %d\n", retval);
}
}
This appears to execute successfully. devinfo -rv is showing mdio0 but it is not attached. It is still not detecting the switch. I put some debug in /usr/src/sys/dev/mdio/mdio.c and it doesn't appear that any of the functions are running e.g. probe, attach, identify.
I'd appreciate any ideas on what needs to be done to get this working!