[Resolved] Exar XR17V352 puc/uart serial port output intermittant and stalls

[pxpunx]

I have a PCIe card with an Exar XR17V352 2-port UART onboard. It's 16550 compatible. More information here:

https://www.maxlinear.com/product/interface/uarts/pcie-uarts/xr17v352

The card works fine on Debian Linux 12.5, OpenBSD 7.4, and FreeBSD 13.2. I'm able to see a raw, continuous stream of serial data when I cat the serial device. This is the expected behavior.

With both FreeBSD and OPNsense, I had to set hw.puc.msi_disable="1" to see any output from the card. This tunable was added in FreeBSD 10.3 to disable the puc driver's new MSI interrupt feature.

https://www.freebsd.org/releases/10.3R/relnotes/

With OPNsense, a cat of the serial device shows a few lines and then stops. I see a similar response from cu, but I have to hit Return to see any output.

Any idea why the response from this card would be so unreliable in OPNsense 24.x on FreeBSD 13.2?

Some data below.

FreeBSD

Code: [Select]

# uname -a
FreeBSD freebsd 13.2-RELEASE FreeBSD 13.2-RELEASE releng/13.2-n254617-525ecfdad597 GENERIC amd64


Code: [Select]

# dmesg|grep puc
puc0: <Exar XR17V352> mem 0xfc900000-0xfc903fff at device 0.0 on pci8
uart2: <16x50 with 256 byte FIFO> at port 1 on puc0
uart3: <16x50 with 256 byte FIFO> at port 2 on puc0


Code: [Select]

# pciconf -lBbcevV pci0:8:0:0
puc0@pci0:8:0:0: class=0x070002 rev=0x03 hdr=0x00 vendor=0x13a8 device=0x0352 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Exar Corp.'
    device     = 'XR17V3521 Dual PCIe UART'
    class      = simple comms
    subclass   = UART
    bar   [10] = type Memory, range 32, base 0xfc900000, size 16384, enabled
    cap 05[50] = MSI supports 1 message, 64 bit
    cap 01[78] = powerspec 3  supports D0 D3  current D0
    cap 10[80] = PCI-Express 2 endpoint MSI 1 max data 128(256) RO NS
                 max read 512
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
    ecap 0002[100] = VC 1 max VC0

OPNsense

Code: [Select]

# uname -a
FreeBSD OPNsense.localdomain 13.2-RELEASE-p10 FreeBSD 13.2-RELEASE-p10 stable/24.1-n254984-f7b006edfa8 SMP amd64


Code: [Select]

# dmesg|grep puc
puc0: <Exar XR17V352> mem 0xaa400000-0xaa403fff irq 16 at device 0.0 numa-domain 0 on pci1
uart2: <16x50 with 256 byte FIFO> at port 1 on puc0
uart3: <16x50 with 256 byte FIFO> at port 2 on puc0


Code: [Select]

# pciconf -lBbcevV pci0:1:0:0
puc0@pci0:1:0:0: class=0x070002 rev=0x03 hdr=0x00 vendor=0x13a8 device=0x0352 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Exar Corp.'
    device     = 'XR17V3521 Dual PCIe UART'
    class      = simple comms
    subclass   = UART
    bar   [10] = type Memory, range 32, base 0xaa400000, size 16384, enabled
    cap 05[50] = MSI supports 1 message, 64 bit
    cap 01[78] = powerspec 3  supports D0 D3  current D0
    cap 10[80] = PCI-Express 2 endpoint MSI 1 max data 128(256) RO NS
                 max read 512
                 link x1(x1) speed 2.5(2.5) ASPM L1(L0s/L1)
    ecap 0002[100] = VC 1 max VC0


[pxpunx]

Some additional info from the OPNsense install:

Code: [Select]

# stty -f /dev/cuau3
speed 9600 baud;
lflags: -icanon -isig -iexten -echo echoe echoke echoctl
iflags: -icrnl ixoff
oflags: -opost tab0
cflags: cs8 -parenb clocal

I set the serial port to 4800 baud and it returned jibberish, so 9600 is the correct baud rate for this particular serial port.

pucdata.c is the same for both FreeBSD and OPNsense:

https://github.com/opnsense/src/blob/58b0ed1f53c2a1e226bda9128504086924333dbb/sys/dev/puc/pucdata.c#L693

https://github.com/freebsd/freebsd-src/blob/8c3a85eaeb0552569e81be43091a18b9e702cc41/sys/dev/puc/pucdata.c#L691

As is the puc_config_exar_pcie function:

https://github.com/opnsense/src/blob/58b0ed1f53c2a1e226bda9128504086924333dbb/sys/dev/puc/pucdata.c#L1418

https://github.com/freebsd/freebsd-src/blob/8c3a85eaeb0552569e81be43091a18b9e702cc41/sys/dev/puc/pucdata.c#L1423

I'm not well-versed in BSD drivers so I'm not sure what else to check that may provide hints as to what the difference between FreeBSD 13.2 and OPNsense 24.1 on FreeBSD 13.2 is as far as this port is concerned.

Outside of actual kernel/driver issues, my only other assumption is interference from another process. However, there are no PIDs w/ this file open:

Code: [Select]

# fuser /dev/cuau3
/dev/cuau3:

Here's an example of cat:

Code: [Select]

# cat /dev/cuau3
<GPS_DATA>
$GPGSV,<GPS_DATA>
$GPGSV,<GPS_DATA>
$GPGSV,<GPS_DATA>
$GPGS,<DIED_HERE>

This should be a continuous stream of data, but it died (in this case) after 5 lines.

I removed the GPS data.

[pxpunx]

Moved to another PCIe slot. The previous slot was on the PCH and the new one is on the CPU. I was curious if there would be a difference.

Code: [Select]

# dmesg|grep puc
puc0: <Exar XR17V352> mem 0xfbe00000-0xfbe03fff irq 48 at device 0.0 numa-domain 0 on pci12
uart2: <16x50 with 256 byte FIFO> at port 1 on puc0
uart3: <16x50 with 256 byte FIFO> at port 2 on puc0

It didn't work at first, but I found that I had removed hw.puc.msi_disable at some point. After I put it back under System > Settings > Tunables, the serial port works.

Something didn't appreciate the card being on a PCH-driven PCIe slot.

I'll sort out the offset later.

Code: [Select]

# ntpq -pn
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
x127.127.22.0    .PPS.            0 l   15   16  377    0.000  +47.743   1.248
x127.127.20.0    .GPS.            0 l   11   16  377    0.000  -73.404   1.346
 0.opnsense.pool .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 1.opnsense.pool .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 2.opnsense.pool .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 3.opnsense.pool .POOL.          16 p    -   64    0    0.000   +0.000   0.000