[SOLVED] High-Availability + CARP IP + No Traffic

[franco]

It's alright. I think I finally traced it back to its origin. I still have to implement pushing the vlan flags to the lagg member interfaces, but consider this fixed in time for 15.7.7.

FYI: https://github.com/opnsense/core/commit/c0bc0c2b71cddf248cd9709cf5f7d1eb357657c0

[romain]

Hello Franco,

Good job. The flag/options on the network card are all good now. I didn't have any bad cksum on my packet.

I tried to reactivate the CARP protocol but I can't make it works. I removed my IP alias, create a new CARP IP.  The two firewall share correctly the VIP but any traffic go through. However it's working fine with IP Alias.

1.1.1.2                        1.1.1.3
-------------                 ------------
-    FW1    ---------------    FW 2  -
-------------                 -------------
                        | VIP : 1.1.1.1
                        |
                        |
                        |
               ---------------
               -      VM1     -
               ---------------
              1.1.2.4

If I look on my network card, everything seems to be okay :

Code: [Select]

        options=400a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 00:90:fa:9d:29:d8
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
oce1: flags=8143<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> metric 0 mtu 1500
        options=400a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 00:90:fa:9d:29:d8
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=400a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0 prefixlen 64 scopeid 0xb
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        laggproto failover lagghash l2,l3,l4
        laggport: oce1 flags=0<>
        laggport: oce0 flags=5<MASTER,ACTIVE>
lagg0_vlan100: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0_vlan2000 prefixlen 64 scopeid 0x12
        inet 1.1.1.2 netmask 0xfffffff0 broadcast 1.1.1.15
        inet 1.1.1.1 netmask 0xfffffff0 broadcast 1.1.1.15 vhid 1
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 100 parent interface: lagg0
        carp: MASTER vhid 1 advbase 1 advskew 100

But If I try to ping the IP 1.1.1.4 from the VLAN interface 1.1.1.2 it's working well. If I try to do the same from the VIP, it's not working at all.

Here some tcpdump

Ping from 1.1.1.2

Code: [Select]

root@opnsense:~ # ping -S 1.1.1.2 1.1.1.4
PING 1.1.1.4 (1.1.1.4) from 1.1.1.2: 56 data bytes
64 bytes from 1.1.1.4: icmp_seq=0 ttl=64 time=0.231 ms
64 bytes from 1.1.1.4: icmp_seq=1 ttl=64 time=0.222 ms
64 bytes from 1.1.1.4: icmp_seq=2 ttl=64 time=0.275 ms
64 bytes from 1.1.1.4: icmp_seq=3 ttl=64 time=0.265 ms

tcpdump: listening on lagg0_vlan100, link-type EN10MB (Ethernet), capture size
13:46:54.005648 IP (tos 0x0, ttl 64, id 22366, offset 0, flags [none], proto ICM
    1.1.1.2 > 1.1.1.4 : ICMP echo request, id 51502, seq 0, length 64
13:46:54.005799 IP (tos 0x0, ttl 64, id 13437, offset 0, flags [none], proto ICM
    1.1.1.4 > 1.1.1.2 : ICMP echo reply, id 51502, seq 0, length 64
13:46:55.013300 IP (tos 0x0, ttl 64, id 930, offset 0, flags [none], proto ICMP
    1.1.1.2  > 10.20.200.3: ICMP echo request, id 51502, seq 1, length 64
13:46:55.013456 IP (tos 0x0, ttl 64, id 13460, offset 0, flags [none], proto ICM
    1.1.1.4 > 1.1.1.2 : ICMP echo reply, id 51502, seq 1, length 64
13:46:56.028299 IP (tos 0x0, ttl 64, id 3204, offset 0, flags [none], proto ICMP

Ping from 1.1.1.1

Code: [Select]

root@opnsense:~ # ping -S 1.1.1.1 1.1.1.4
PING 1.1.1.4 (1.1.1.4) from 1.1.1.1: 56 data bytes


tcpdump: listening on lagg0_vlan100, link-type EN10MB (Ethernet), capture size 65535 bytes
13:46:15.760291 IP (tos 0x0, ttl 64, id 14690, offset 0, flags [none], proto ICMP (1), length 84)
    1.1.1.1 > 1.1.1.4: ICMP echo request, id 18870, seq 39, length 64
13:46:16.762298 IP (tos 0x0, ttl 64, id 32272, offset 0, flags [none], proto ICMP (1), length 84)
    1.1.1.1 > 1.1.1.4: ICMP echo request, id 18870, seq 40, length 64
13:46:17.763862 IP (tos 0x0, ttl 64, id 32835, offset 0, flags [none], proto ICMP (1), length 84)
    1.1.1.1 > 1.1.1.4: ICMP echo request, id 18870, seq 41, length 64
13:46:18.765300 IP (tos 0x0, ttl 64, id 23995, offset 0, flags [none], proto ICMP (1), length 84)
    1.1.1.1 > 1.1.1.4: ICMP echo request, id 18870, seq 42, length 64
13:46:19.766300 IP (tos 0x0, ttl 64, id 59428, offset 0, flags [none], proto ICMP (1), length 84)

It seems that the packet is sent but there is no answer. How it can be possible ? Do I need to allow specific protocol on the firewall ?

On the client side in the ARP table, the machine can't find the mac address asssociated to the VIP.

However, on the firewall side, I can find the associated mac of the machine even with a ping from 1.1.1.1

Code: [Select]

root@opnsense:~ # arp -a
root@opnsense:~ # ping -S 1.1.1.1 1.1.1.4
PING 1.1.1.4 (1.1.1.4) from 1.1.1.1: 56 data bytes
^C
root@opnsense:~ # arp -a
? (1.1.1.4) at 00:50:56:94:39:35 on lagg0_vlan100 expires in 1196 seconds [vlan]

I quite lost...

[AdSchellevis]

hi romain,

I did create a small document a couple of weeks a go for CARP, this may be of use if your issue is CARP related.
https://wiki.opnsense.org/index.php/Configure_CARP


There are some ports that need to be opened for CARP to work, if this doesn't fix your issue maybe you could share your configs (without passwords and privatekeys) with me so I can review your CARP settings.

Cheers,

Ad ( email ad <at> project domain)