[SOLVED] High-Availability + CARP IP + No Traffic

[franco]

It looks like the interface is completely ignored. Could you provide the full ifconfig output please?

[romain]

Code: [Select]

oce0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1500
        options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
        ether 00:90:fa:9d:29:d8
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
oce1: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1500
        options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
        ether 00:90:fa:9d:29:d8
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 0c:c4:7a:32:63:f4
        inet IP_PUBLIC netmask 0xfffffff0 broadcast IP_PUBLIC_BROCAST
        inet6 fe80::ec4:7aff:fe32:63f4%igb0 prefixlen 64 scopeid 0x3
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 0c:c4:7a:32:63:f5
        inet IP_PRIVATE netmask 0xfffffff8 broadcast IP_PRIVATE_BROADCAST
        inet6 fe80::ec4:7aff:fe32:63f5%igb1 prefixlen 64 scopeid 0x4
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb2: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
        ether 0c:c4:7a:32:63:f6
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: no carrier
igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 0c:c4:7a:32:63:f7
        inet6 fe80::ec4:7aff:fe32:63f7%igb3 prefixlen 64 scopeid 0x6
        inet IP_PRIVATE netmask 0xffffff80 broadcast IP_PRIVATE_BROADCAST
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
pflog0: flags=100<PROMISC> metric 0 mtu 33160
pfsync0: flags=0<> metric 0 mtu 1500
        syncpeer: 0.0.0.0 maxupd: 128 defer: off
enc0: flags=0<> metric 0 mtu 1536
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0 prefixlen 64 scopeid 0xb
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        laggproto failover lagghash l2,l3,l4
        laggport: oce1 flags=0<>
        laggport: oce0 flags=5<MASTER,ACTIVE>
lagg0_vlan8: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0_vlan8 prefixlen 64 scopeid 0xc
        inet IP_PRIVATE netmask 0xffffff80 broadcast IP_PRIVATE_BROADCAST
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 8 parent interface: lagg0
lagg0_vlan11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0_vlan11 prefixlen 64 scopeid 0xd
        inet IP_PRIVATE netmask 0xffffff80 broadcast IP_PRIVATE_BROADCAST
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 11 parent interface: lagg0
lagg0_vlan16: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0_vlan16 prefixlen 64 scopeid 0xe
        inet IP_PRIVATE netmask 0xffffff80 broadcast IP_PRIVATE_BROADCAST
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 16 parent interface: lagg0
lagg0_vlan24: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0_vlan24 prefixlen 64 scopeid 0xf
        inet IP_PRIVATE netmask 0xffffff80 broadcast IP_PRIVATE_BROADCAST
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 24 parent interface: lagg0
igb3_vlan247: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 0c:c4:7a:32:63:f7
        inet6 fe80::ec4:7aff:fe32:63f7%igb3_vlan247 prefixlen 64 scopeid 0x10
        inet IP_PRIVATE netmask 0xffffff80 broadcast IP_PRIVATE_BROADCAST
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 247 parent interface: igb3
lagg0_vlan248: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0_vlan248 prefixlen 64 scopeid 0x11
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 248 parent interface: lagg0
lagg0_vlan2000: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0_vlan2000 prefixlen 64 scopeid 0x12
        inet IP_PRIVATE netmask 0xffffff80 broadcast IP_PRIVATE_BROADCAST
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 2000 parent interface: lagg0
lagg0_vlan2010: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0_vlan2010 prefixlen 64 scopeid 0x13
        inet IP_PRIVATE netmask 0xffffff80 broadcast IP_PRIVATE_BROADCAST
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 2010 parent interface: lagg0
ovpns1: flags=8010<POINTOPOINT,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ovpns2: flags=8010<POINTOPOINT,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


[franco]

Are oce0 and oce1 assigned in the GUI?

[romain]

Not at all

[franco]

That explains why the settings are not being scrubbed. You'll have to assign and use them in order to see if the fix helps.

[romain]

So even if I don't want to use them, I need to set an interface with any configuration ?

[franco]

Well, no, but the parent interface isn't touched when it is not selected so the GUI won't reset its settings. This is because then you are free to configure the interface on your own in FreeBSD style if you so choose.

The *_vlan devices are the interesting ones in this case. Although I don't know interface flags are being propagated in such cases like yours or if we do have to handle a unassigned parent interface. It might not even be there on boot, so this is rather tricky. I'll take a fresh look tomorrow.

[romain]

I can't assign them into the gui as they are part of the LAGG interface.

However I can assign the Lagg interface if needed.

[franco]

I'm trying to deactivate RXCSUM,TXCSUM and TSO too but can't find a way to do it properly

Another patch will go into 15.7.6 to make this run smoothly for VLAN acceleration flags: https://github.com/opnsense/core/commit/280a00d800281f7dfff7c67e1fb7b769ab59eb8b

I've tested this quite a bit this morning and I found that:

o rxcsum, txcsum and tso can only be disabled on the parent interface if it is assigned
o all vlan acceleration flags will be added to/stripped off the parent interface correctly now

If the behaviour for rxcsum and others is undesired, we can engineer a fix too. Please try that with 15.7.6 tomorrow and let me know.


Cheers,
Franco

[romain]

Thank you Franco, So I will try tomorrow. Do I need to assign the Lagg0 interface to a network ? My two OCE ports are grouped in my lagg0.

[franco]

I'm beginning to think there may be an issue with parent interface being LAGG as opposed to the real hardware device. Sorry for all the trouble; it seems this is not a often-deployed use case. In any case, looking forward to the 15.7.6 feedback.

[romain]

Yes. No problem but do you want me to assign the lagg interface because today I only use Vlan interface on that lagg

[romain]

Thank you Franco for the good job but nothing works. I sill have all options activate even if everything is checked under System > Network > Disable **

Here the output under opnsense 15.7.6

Code: [Select]

oce0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1500
        options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>

oce1: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1500
        options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>

lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
lagg0_vlan2000: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>

As mentionned, the two OCE network card are part of the lagg which is not assigned in the guy. Only the vlan interface.

Please let me know if I can help you somehow.

Have a nice day
Romain

[franco]

It's not so much true that nothing works: we added a VLAN flags disable and added more flags to the code path. I for one am sad that the code we own was in a really bad shape to begin with. The only thing we can do here is pick it up and fix it, piece by piece.

My next debug session will be to find out why LAGG interfaces are handled differently in the first place.

[romain]

Sorry Franco that was not what I tried to mean. I know it's not nothing and you're producing great work. But it's not working in my case.

Please let me know if you want me to test some new code and so far. I have a spare firewall that I can play with.