OPNsense Forum

Archive => 15.7 Legacy Series => Topic started by: romain on July 07, 2015, 06:11:27 pm

Title: [SOLVED] High-Availability + CARP IP + No Traffic
Post by: romain on July 07, 2015, 06:11:27 pm
Hello,

i continue to test deeply OpnSense but I encounter a trouble.

I have two identical boxes with 4x1Gbe Intel ports and 2x 10Gbe Emulex ports. I have a lagg configured as failover for the two 10Gbe ports. I have 5 tagged vlans going though this lagg. Everything working fine.

i tried to configured HA between the two boxes. I added my Carp VIPs on every vlans but I have strange behaviour with it.

The gateway of my every vlan subnets are my Carp VIPs. Everything seems to be okay on the OPNSense side. The master manages and deals the CARP IP and the backup is waiting for a failure of the master (when I reboot the master, the backup takes correctly the VIPs) However, if I try to ping or go through the CARP IP nothing works unless I use a machine on FreeBSD too. In that case it's works. If I take a windows machine plugged on the same switch with the same tag configuration, it's not working at all.

If I look deeper, I can see that every two firewall can ping and reach the windows machine through their own IP. if I do a ping -S VIP_ADDRESS IP_WINDOWS it's not working.

On the other side, if I try to ping the VIP of the subnet, I have a timeout. But if I look the arp table I can see the right mac address defined by the carp prototol (00:00:...:01).

I tried to deactivate the firewall to see if my issue was related to some missing rules but not.. it's not working better.

I'm pretty sure my CARP are okay because the WAN Side works well with a OpenVPN server. 

Does someone have idea of what going on and what I'm doing wrong ?

Please let me know if you need any more information ?

Romain
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on July 07, 2015, 09:02:40 pm
I continue to debug. I found two things very strange :

The CARP Announcement packet have public IP inside.. I should not only have same subnet IP (my two firewall are in 172.28.11.101 and 172.28.11.102) ?

172.28.11.101 > vrrp.mcast.net: vrrp 172.28.11.101 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 0, authtype none, intvl 1s, length 36, addrs(7): p4FE15735.dip0.t-ipconnect.de,251.222.243.34,66.146.73.124.broad.dynamic.hf.ah.cndata.com,127.76.101.79,251.40.1.5,36.138.207.21,sto95-4-88-178-136-1.fbx.proxad.net

I also note many many bad cksum 0 on different type of packet (CARP Announcement or ICMP) :

21:00:08.219352 IP (tos 0x10, ttl 255, id 46264, offset 0, flags [DF], proto VRRP (112), length 56, bad cksum 0 (->2ef9)!)

Code: [Select]
root@OPNSENSE:~ # tcpdump -i lagg0_vlan8 -vvv "carp"
tcpdump: listening on lagg0_vlan8, link-type EN10MB (Ethernet), capture size 65535 bytes
21:00:08.219352 IP (tos 0x10, ttl 255, id 46264, offset 0, flags [DF], proto VRRP (112), length 56, [b]bad cksum 0 (->2ef9)!)[/b]
    172.28.11.101 > vrrp.mcast.net: vrrp 172.28.11.101 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 0, authtype none, intvl 1s, length 36, addrs(7): p4FE15735.dip0.t-ipconnect.de,251.222.243.28,15.sub-97-205-206.myvzw.com,adsl-75-24-212-125.dsl.pltn13.sbcglobal.net,dynamic.sdtv.net.tw,219.164.243.201,softbank126252245163.bbtec.net
21:00:09.220242 IP (tos 0x10, ttl 255, id 30484, offset 0, flags [DF], proto VRRP (112), length 56, bad cksum 0 (->6c9d)!)
    172.28.11.101 > vrrp.mcast.net: vrrp 172.28.11.101 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 0, authtype none, intvl 1s, length 36, addrs(7): p4FE15735.dip0.t-ipconnect.de,251.222.243.29,143.126.25.101,149.104.16.164,c-68-36-70-172.hsd1.mi.comcast.net,slip139-92-30-202.fra.de.prserv.net,142.41.200.122
21:00:10.221352 IP (tos 0x10, ttl 255, id 15770, offset 0, flags [DF], proto VRRP (112), length 56, bad cksum 0 (->a617)!)
    172.28.11.101 > vrrp.mcast.net: vrrp 172.28.11.101 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 0, authtype none, intvl 1s, length 36, addrs(7): p4FE15735.dip0.t-ipconnect.de,251.222.243.30,147.16.210.213,58.204.203.159,44.59.163.33,42.213.235.216,168.192.80.249
21:00:11.222240 IP (tos 0x10, ttl 255, id 14066, offset 0, flags [DF], proto VRRP (112), length 56, bad cksum 0 (->acbf)!)
    172.28.11.101 > vrrp.mcast.net: vrrp 172.28.11.101 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 0, authtype none, intvl 1s, length 36, addrs(7): p4FE15735.dip0.t-ipconnect.de,251.222.243.31,ip-109-90-25-189.hsi11.unitymediagroup.de,51.192.163.97,55.187.192.51,118.201.211.21,64.16.244.104
21:00:12.223336 IP (tos 0x10, ttl 255, id 8975, offset 0, flags [DF], proto VRRP (112), length 56, bad cksum 0 (->c0a2)!)
    172.28.11.101 > vrrp.mcast.net: vrrp 172.28.11.101 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 0, authtype none, intvl 1s, length 36, addrs(7): p4FE15735.dip0.t-ipconnect.de,251.222.243.32,30.28.37.249,202.5.199.134,169.62.123.150,236.221.81.16,133.206.52.220
21:00:13.224235 IP (tos 0x10, ttl 255, id 4142, offset 0, flags [DF], proto VRRP (112), length 56, bad cksum 0 (->d383)!)
    172.28.11.101 > vrrp.mcast.net: vrrp 172.28.11.101 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 0, authtype none, intvl 1s, length 36, addrs(7): p4FE15735.dip0.t-ipconnect.de,251.222.243.33,163.0.108.130,softbank219053055176.bbtec.net,199.188.240.51,c-67-182-72-116.hsd1.ca.comcast.net,233.200.43.96
21:00:14.225353 IP (tos 0x10, ttl 255, id 32762, offset 0, flags [DF], proto VRRP (112), length 56, bad cksum 0 (->63b7)!)
^C
    172.28.11.101 > vrrp.mcast.net: vrrp 172.28.11.101 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 0, authtype none, intvl 1s, length 36, addrs(7): p4FE15735.dip0.t-ipconnect.de,251.222.243.34,66.146.73.124.broad.dynamic.hf.ah.cndata.com,127.76.101.79,251.40.1.5,36.138.207.21,sto95-4-88-178-136-1.fbx.proxad.net

Any idea ?
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on July 08, 2015, 04:45:35 pm
I read that it can be from the TSO and LRO which are active on my network card. I can disable it by using ifconfig command. However, TSO6 always stay and when I rebook the firewall, options are coming back.

As I'm using lagg the options are set at the startup of the firewall. How can I be sure that these options are disabled permanently ?

I use oce.ko driver delivered by Emulex directly for FreeBSD 10.1
Title: Re: High-Availability + CARP IP + No Traffic
Post by: jschellevis on July 08, 2015, 04:56:55 pm
You can disable LRO and/or TSO in the GUI  System-> Settings-> Networking

LRO is known to cause issues with a lot of hardware, so you better disable it.
TSO usally works well, but if not disable it as well.


Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on July 08, 2015, 06:19:13 pm
Thank. I already did that but I m not sure it s okay. HowHcan can I be sure?

I still have the option activated on my network card.

I also change the value in the sysctl
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on July 10, 2015, 08:03:00 am
Hello,

I would like to setup the options of my network card permanently. How can I do that ? If I do a ifconfig oce0 -vlhwfilter it's works now but if I reboot these change are gone. As I'm using lagg on this interface I need to force the option before the boot.

Any idea to do it cleanly ?

Thank you !
Title: Re: High-Availability + CARP IP + No Traffic
Post by: franco on July 10, 2015, 08:14:12 am
We can add a knob for vlanhwfilter in the GUI. For now, you'll have to put the custom command it into e.g. /usr/local/etc/rc before the rc.bootup invoke. Please not this will get wiped on firmware updates as well. Ticket here:

https://github.com/opnsense/core/issues/252
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on July 10, 2015, 09:06:38 am
Basically it's not only for vlanhwfilter it's for every option. I should be great if we have a place where we can give you the option to activate or not. In my case something like :

ifconfig oce0 -lro -tso -tso4 -tso6 -rxcsum -txcsum
ifconfig oce1 -lro -tso -tso4 -tso6 -rxcsum -txcsum

To configure the interface as I wanted
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on July 10, 2015, 09:38:13 am
Thank you Franco for the tips. but I can't make it work. Here what I did :

/usr/local/etc/rc
Code: [Select]
#MODIF ROMAIN
echo -n "Modification ifconfig oce0..."
ifconfig oce0 -lro -tso -tso4 -tso6 -rxcsum -txcsum > /root/oce0.txt 2>&1
echo -n "Modification ifconfig oce1..."
ifconfig oce1 -lro -tso -tso4 -tso6 -rxcsum -txcsum > /root/oce1.txt 2>&1

# let the PHP-based configuration subsystem set up the system now
echo -n "Launching the init system..."
rm -f /root/lighttpd*
touch /var/run/booting
/usr/local/etc/rc.bootup
rm /var/run/booting

The file oc1.txt and oce0.txt are created. But if I do a ifconfig right after the boot, the removed options are still there :

Code: [Select]
root@TEST:~ # ifconfig oce1
oce1: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1500
        options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
        ether 00:90:fa:9d:29:d8
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
root@TEST:~ # ifconfig oce0
oce0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1500
        options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
        ether 00:90:fa:9d:29:d8
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
Title: Re: High-Availability + CARP IP + No Traffic
Post by: franco on July 10, 2015, 09:52:03 am
Meh, ok. I'll take a closer look soon, thanks for testing.
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on July 10, 2015, 10:49:10 am
Thank you. Please let me know it quit blocking today for my configuration.

Let me know if you want me to test anything
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on July 15, 2015, 10:59:34 am
Sorry it was here.

Do you need me to test some fix ? I would like to be able to manage the options loaded on my network card before the lagg creation.
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on July 24, 2015, 08:49:07 am
Hello Franco,

Did you have time for my bug ?

Thank you for your work anyway.
Title: Re: High-Availability + CARP IP + No Traffic
Post by: franco on July 25, 2015, 09:53:17 pm
15.7.4 has a new option under "System: Settings: Networking", see attached screenshot. Could you try this and see if it helps your case?
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on July 27, 2015, 08:00:26 am
Same trouble. I have checked and rebooted.

After the reboot, the option is still there :

Code: [Select]
ifconfig oce0
oce0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1500
        options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
        ether 00:90:fa:9d:29:d8
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active

I'm trying to deactivate RXCSUM,TXCSUM and TSO too but can't find a way to do it properly
Title: Re: High-Availability + CARP IP + No Traffic
Post by: franco on July 27, 2015, 01:36:26 pm
It looks like the interface is completely ignored. Could you provide the full ifconfig output please?
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on July 27, 2015, 01:43:57 pm
Code: [Select]
oce0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1500
        options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
        ether 00:90:fa:9d:29:d8
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
oce1: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1500
        options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
        ether 00:90:fa:9d:29:d8
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 0c:c4:7a:32:63:f4
        inet IP_PUBLIC netmask 0xfffffff0 broadcast IP_PUBLIC_BROCAST
        inet6 fe80::ec4:7aff:fe32:63f4%igb0 prefixlen 64 scopeid 0x3
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 0c:c4:7a:32:63:f5
        inet IP_PRIVATE netmask 0xfffffff8 broadcast IP_PRIVATE_BROADCAST
        inet6 fe80::ec4:7aff:fe32:63f5%igb1 prefixlen 64 scopeid 0x4
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb2: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
        ether 0c:c4:7a:32:63:f6
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: no carrier
igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 0c:c4:7a:32:63:f7
        inet6 fe80::ec4:7aff:fe32:63f7%igb3 prefixlen 64 scopeid 0x6
        inet IP_PRIVATE netmask 0xffffff80 broadcast IP_PRIVATE_BROADCAST
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
pflog0: flags=100<PROMISC> metric 0 mtu 33160
pfsync0: flags=0<> metric 0 mtu 1500
        syncpeer: 0.0.0.0 maxupd: 128 defer: off
enc0: flags=0<> metric 0 mtu 1536
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0 prefixlen 64 scopeid 0xb
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        laggproto failover lagghash l2,l3,l4
        laggport: oce1 flags=0<>
        laggport: oce0 flags=5<MASTER,ACTIVE>
lagg0_vlan8: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0_vlan8 prefixlen 64 scopeid 0xc
        inet IP_PRIVATE netmask 0xffffff80 broadcast IP_PRIVATE_BROADCAST
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 8 parent interface: lagg0
lagg0_vlan11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0_vlan11 prefixlen 64 scopeid 0xd
        inet IP_PRIVATE netmask 0xffffff80 broadcast IP_PRIVATE_BROADCAST
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 11 parent interface: lagg0
lagg0_vlan16: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0_vlan16 prefixlen 64 scopeid 0xe
        inet IP_PRIVATE netmask 0xffffff80 broadcast IP_PRIVATE_BROADCAST
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 16 parent interface: lagg0
lagg0_vlan24: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0_vlan24 prefixlen 64 scopeid 0xf
        inet IP_PRIVATE netmask 0xffffff80 broadcast IP_PRIVATE_BROADCAST
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 24 parent interface: lagg0
igb3_vlan247: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 0c:c4:7a:32:63:f7
        inet6 fe80::ec4:7aff:fe32:63f7%igb3_vlan247 prefixlen 64 scopeid 0x10
        inet IP_PRIVATE netmask 0xffffff80 broadcast IP_PRIVATE_BROADCAST
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 247 parent interface: igb3
lagg0_vlan248: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0_vlan248 prefixlen 64 scopeid 0x11
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 248 parent interface: lagg0
lagg0_vlan2000: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0_vlan2000 prefixlen 64 scopeid 0x12
        inet IP_PRIVATE netmask 0xffffff80 broadcast IP_PRIVATE_BROADCAST
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 2000 parent interface: lagg0
lagg0_vlan2010: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0_vlan2010 prefixlen 64 scopeid 0x13
        inet IP_PRIVATE netmask 0xffffff80 broadcast IP_PRIVATE_BROADCAST
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 2010 parent interface: lagg0
ovpns1: flags=8010<POINTOPOINT,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ovpns2: flags=8010<POINTOPOINT,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Title: Re: High-Availability + CARP IP + No Traffic
Post by: franco on July 27, 2015, 02:04:52 pm
Are oce0 and oce1 assigned in the GUI?
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on July 27, 2015, 02:05:34 pm
Not at all
Title: Re: High-Availability + CARP IP + No Traffic
Post by: franco on July 27, 2015, 05:13:27 pm
That explains why the settings are not being scrubbed. You'll have to assign and use them in order to see if the fix helps.
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on July 27, 2015, 07:24:06 pm
So even if I don't want to use them, I need to set an interface with any configuration ?
Title: Re: High-Availability + CARP IP + No Traffic
Post by: franco on July 27, 2015, 09:29:45 pm
Well, no, but the parent interface isn't touched when it is not selected so the GUI won't reset its settings. This is because then you are free to configure the interface on your own in FreeBSD style if you so choose.

The *_vlan devices are the interesting ones in this case. Although I don't know interface flags are being propagated in such cases like yours or if we do have to handle a unassigned parent interface. It might not even be there on boot, so this is rather tricky. I'll take a fresh look tomorrow.
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on July 28, 2015, 02:22:48 pm
I can't assign them into the gui as they are part of the LAGG interface.

However I can assign the Lagg interface if needed.

Title: Re: High-Availability + CARP IP + No Traffic
Post by: franco on July 30, 2015, 11:24:01 am
I'm trying to deactivate RXCSUM,TXCSUM and TSO too but can't find a way to do it properly

Another patch will go into 15.7.6 to make this run smoothly for VLAN acceleration flags: https://github.com/opnsense/core/commit/280a00d800281f7dfff7c67e1fb7b769ab59eb8b

I've tested this quite a bit this morning and I found that:

o rxcsum, txcsum and tso can only be disabled on the parent interface if it is assigned
o all vlan acceleration flags will be added to/stripped off the parent interface correctly now

If the behaviour for rxcsum and others is undesired, we can engineer a fix too. Please try that with 15.7.6 tomorrow and let me know.


Cheers,
Franco
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on July 30, 2015, 02:43:31 pm
Thank you Franco, So I will try tomorrow. Do I need to assign the Lagg0 interface to a network ? My two OCE ports are grouped in my lagg0.
Title: Re: High-Availability + CARP IP + No Traffic
Post by: franco on July 30, 2015, 04:14:28 pm
I'm beginning to think there may be an issue with parent interface being LAGG as opposed to the real hardware device. Sorry for all the trouble; it seems this is not a often-deployed use case. In any case, looking forward to the 15.7.6 feedback.
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on July 30, 2015, 04:28:37 pm
Yes. No problem but do you want me to assign the lagg interface because today I only use Vlan interface on that lagg
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on August 01, 2015, 04:06:56 pm
Thank you Franco for the good job but nothing works. I sill have all options activate even if everything is checked under System > Network > Disable **

Here the output under opnsense 15.7.6

Code: [Select]
oce0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1500
        options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>

oce1: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1500
        options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>

lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
lagg0_vlan2000: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>

As mentionned, the two OCE network card are part of the lagg which is not assigned in the guy. Only the vlan interface.

Please let me know if I can help you somehow.

Have a nice day
Romain
Title: Re: High-Availability + CARP IP + No Traffic
Post by: franco on August 02, 2015, 07:13:37 pm
It's not so much true that nothing works: we added a VLAN flags disable and added more flags to the code path. I for one am sad that the code we own was in a really bad shape to begin with. The only thing we can do here is pick it up and fix it, piece by piece.

My next debug session will be to find out why LAGG interfaces are handled differently in the first place.
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on August 02, 2015, 08:52:47 pm
Sorry Franco that was not what I tried to mean. I know it's not nothing and you're producing great work. But it's not working in my case.

Please let me know if you want me to test some new code and so far. I have a spare firewall that I can play with.

Title: Re: High-Availability + CARP IP + No Traffic
Post by: franco on August 03, 2015, 08:34:50 am
It's alright. :) I think I finally traced it back to its origin. I still have to implement pushing the vlan flags to the lagg member interfaces, but consider this fixed in time for 15.7.7.

FYI: https://github.com/opnsense/core/commit/c0bc0c2b71cddf248cd9709cf5f7d1eb357657c0
Title: Re: High-Availability + CARP IP + No Traffic
Post by: romain on August 05, 2015, 02:22:38 pm
Hello Franco,

Good job. The flag/options on the network card are all good now. I didn't have any bad cksum on my packet.

I tried to reactivate the CARP protocol but I can't make it works. I removed my IP alias, create a new CARP IP.  The two firewall share correctly the VIP but any traffic go through. However it's working fine with IP Alias.

1.1.1.2                        1.1.1.3
-------------                 ------------
-    FW1    ---------------    FW 2  -
-------------                 -------------
                        | VIP : 1.1.1.1
                        |
                        |
                        |
               ---------------
               -      VM1     -
               ---------------
              1.1.2.4

If I look on my network card, everything seems to be okay :

Code: [Select]
        options=400a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 00:90:fa:9d:29:d8
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
oce1: flags=8143<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> metric 0 mtu 1500
        options=400a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 00:90:fa:9d:29:d8
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=400a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0 prefixlen 64 scopeid 0xb
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        laggproto failover lagghash l2,l3,l4
        laggport: oce1 flags=0<>
        laggport: oce0 flags=5<MASTER,ACTIVE>
lagg0_vlan100: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 00:90:fa:9d:29:d8
        inet6 fe80::290:faff:fe9d:29d8%lagg0_vlan2000 prefixlen 64 scopeid 0x12
        inet 1.1.1.2 netmask 0xfffffff0 broadcast 1.1.1.15
        inet 1.1.1.1 netmask 0xfffffff0 broadcast 1.1.1.15 vhid 1
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 100 parent interface: lagg0
        carp: MASTER vhid 1 advbase 1 advskew 100

But If I try to ping the IP 1.1.1.4 from the VLAN interface 1.1.1.2 it's working well. If I try to do the same from the VIP, it's not working at all.

Here some tcpdump

Ping from 1.1.1.2
Code: [Select]
root@opnsense:~ # ping -S 1.1.1.2 1.1.1.4
PING 1.1.1.4 (1.1.1.4) from 1.1.1.2: 56 data bytes
64 bytes from 1.1.1.4: icmp_seq=0 ttl=64 time=0.231 ms
64 bytes from 1.1.1.4: icmp_seq=1 ttl=64 time=0.222 ms
64 bytes from 1.1.1.4: icmp_seq=2 ttl=64 time=0.275 ms
64 bytes from 1.1.1.4: icmp_seq=3 ttl=64 time=0.265 ms

tcpdump: listening on lagg0_vlan100, link-type EN10MB (Ethernet), capture size
13:46:54.005648 IP (tos 0x0, ttl 64, id 22366, offset 0, flags [none], proto ICM
    1.1.1.2 > 1.1.1.4 : ICMP echo request, id 51502, seq 0, length 64
13:46:54.005799 IP (tos 0x0, ttl 64, id 13437, offset 0, flags [none], proto ICM
    1.1.1.4 > 1.1.1.2 : ICMP echo reply, id 51502, seq 0, length 64
13:46:55.013300 IP (tos 0x0, ttl 64, id 930, offset 0, flags [none], proto ICMP
    1.1.1.2  > 10.20.200.3: ICMP echo request, id 51502, seq 1, length 64
13:46:55.013456 IP (tos 0x0, ttl 64, id 13460, offset 0, flags [none], proto ICM
    1.1.1.4 > 1.1.1.2 : ICMP echo reply, id 51502, seq 1, length 64
13:46:56.028299 IP (tos 0x0, ttl 64, id 3204, offset 0, flags [none], proto ICMP

Ping from 1.1.1.1
Code: [Select]
root@opnsense:~ # ping -S 1.1.1.1 1.1.1.4
PING 1.1.1.4 (1.1.1.4) from 1.1.1.1: 56 data bytes


tcpdump: listening on lagg0_vlan100, link-type EN10MB (Ethernet), capture size 65535 bytes
13:46:15.760291 IP (tos 0x0, ttl 64, id 14690, offset 0, flags [none], proto ICMP (1), length 84)
    1.1.1.1 > 1.1.1.4: ICMP echo request, id 18870, seq 39, length 64
13:46:16.762298 IP (tos 0x0, ttl 64, id 32272, offset 0, flags [none], proto ICMP (1), length 84)
    1.1.1.1 > 1.1.1.4: ICMP echo request, id 18870, seq 40, length 64
13:46:17.763862 IP (tos 0x0, ttl 64, id 32835, offset 0, flags [none], proto ICMP (1), length 84)
    1.1.1.1 > 1.1.1.4: ICMP echo request, id 18870, seq 41, length 64
13:46:18.765300 IP (tos 0x0, ttl 64, id 23995, offset 0, flags [none], proto ICMP (1), length 84)
    1.1.1.1 > 1.1.1.4: ICMP echo request, id 18870, seq 42, length 64
13:46:19.766300 IP (tos 0x0, ttl 64, id 59428, offset 0, flags [none], proto ICMP (1), length 84)

It seems that the packet is sent but there is no answer. How it can be possible ? Do I need to allow specific protocol on the firewall ?

On the client side in the ARP table, the machine can't find the mac address asssociated to the VIP.

However, on the firewall side, I can find the associated mac of the machine even with a ping from 1.1.1.1

Code: [Select]
root@opnsense:~ # arp -a
root@opnsense:~ # ping -S 1.1.1.1 1.1.1.4
PING 1.1.1.4 (1.1.1.4) from 1.1.1.1: 56 data bytes
^C
root@opnsense:~ # arp -a
? (1.1.1.4) at 00:50:56:94:39:35 on lagg0_vlan100 expires in 1196 seconds [vlan]

I quite lost...
Title: Re: High-Availability + CARP IP + No Traffic
Post by: AdSchellevis on August 05, 2015, 03:11:51 pm
hi romain,

I did create a small document a couple of weeks a go for CARP, this may be of use if your issue is CARP related.
https://wiki.opnsense.org/index.php/Configure_CARP
 (https://wiki.opnsense.org/index.php/Configure_CARP)

There are some ports that need to be opened for CARP to work, if this doesn't fix your issue maybe you could share your configs (without passwords and privatekeys) with me so I can review your CARP settings.

Cheers,

Ad ( email ad <at> project domain)