[CALL FOR TESTING] Destination Host unreachable (PPPoE)

[dragon2611]

I've had a it a few times where I lose PPPoE for whatever reason, traffic switches over to the other wan, PPPoE recovers and when traffic switches back I cannot access the internet.

Any ping/tracert.etc just return destination host unreachable coming from OpnSense's lan interface Ip. 

Gateway status shows the gateway is up (I have it ping openDNS at the moment to check that)

Actually this is pretty much what happens all the time if IDS is ON (which is why I can't use the inline IDS)

This is running in Proxmox/KVM with VirtIO Nics, 4x multiqueue (Although that doesn't make much difference as far as I can tell)


2 of my virtual NIC's do connect to the same vswitch and physical NIC atm as I'm waiting on addational network cards (one is DHCP the other is PPPoE) so I'm not sure if that's related although it shouldn't be as only one of the ISP's is PPPoE and the NIC's have their own MAC.

Proxmox config for the Opnsense vm is below, the vmbr's are using openvswitch NOT linux bridging.

Code: [Select]

                 

bootdisk: virtio0
cores: 4
cpu: host
cpuunits: 99999
ide2: none,media=cdrom
memory: 3096
name: OpnSense
net0: virtio=32:66:62:33:36:30,bridge=vmbr0,queues=4
net1: virtio=32:37:38:33:33:33,bridge=vmbr1,queues=4
net2: virtio=32:36:31:39:34:35,bridge=vmbr1,queues=4
net3: virtio=32:39:37:36:31:65,bridge=vmbr4,queues=4
numa: 0
onboot: 1
ostype: other
smbios1: uuid=84331f80-4308-4407-ae79-045178613e26
sockets: 1
startup: order=1
virtio0: local-lvm:vm-104-disk-1,size=20G

Also the ports get mirrored to the secuirty onion vm.

Rebooting OpnSense fixes the problem, as long as I don't try and enable IDS

[dragon2611]

Nics arrived this morning, also found a filter rule for IPv6 pointing at the 2nd ISP that was failing (I think because that interface didn't pickup a V6 address at the time so it was an invalid gateway.

Code: [Select]

opnsense: /usr/local/etc/rc.newwanip: New alert found: There were error(s) loading the rules: /tmp/rules.debug:189: no routing address with matching address family found. - The line in question reads [189]: pass in quick on $LAN $GWSKY_DHCP6 inet6 from 2001:xxx:xx:fe77::/64 to any keep state label "USER_RULE: Default allow LAN IPv6 to any rule"
Also that prefix wouldn't work with that gateway anyway, I was seeing if NPT could be used between 2 public subnets but I don't think it can.

[Andreas]

Perhaps you should take a look to
https://github.com/opnsense/core/issues/850

[franco]

Andreas, I do think that could be a different issue if failing firewall rules are involved.

Is this only for IPv6 or all connectivity?

Can you provide the output of:

# ls /var/etc/mpd_*.conf


Cheers,
Franco

[dragon2611]

Ipv4 I haven't found a way to configure V6 to failover yet as I'm having a lot of trouble with the pinger on v6 atm,

Now I have my NIC's and the line fault is fixed (hopefully) i'll prob swap out my ISP supplied router on the 2nd line for a modem and hand the public IPv4 and IPv6 PD to opnsense.

Edit:

As requested

 ls /var/etc/mpd_*.conf
/var/etc/mpd_opt1.conf

Also removed multiqueues on the hypervisor as I think it was causing some very werid response issues that were hard to pin down (stuff felt laggy even though latency.etc seemed ok, e.g VNC sessions)

Whilst I like proxmox/KVM I do wonder if this perticullar box might work better as an ESXI host if It will run on there and detect the drives (A friend says he uses opnsense in ESXi and his works well including IDS)

[franco]

Ok, you seem to be running into the same issue after all. My bad. It's that PPPoE on OPT interfaces had a very old restriction in them. You can apply the patch manually and restart the box:

# opnsense-patch b349470

Andreas reported this fixed the problem for him.

We're overly cautious about this fix, not wanting to bring it into the images for 16.7, so this will be in 16.7.1. Make sure you reapply the patch once you upgrade.

[dragon2611]

Thanks I'll apply the patch.

Also I've just found out the NIC i'd put in for the PPPoE connection is infact faulty, was getting terrible throughput and the odd bit of packetloss 

Swapped the VNIC > Physical NIC mappings around in the hypervisor and the problem moved onto the other connection

[dragon2611]

Hmm I don't think the patch has done it 

For whatever reason this morning I got failed over to backup for a couple mins and on the primary coming back I ended up back with the destination host unreachable problem.

[franco]

Are you using the PPPOE as your primary WAN? I'm asking because the code seemed to assume strongly that the main connection should be WAN itself and we saw that OPT1 is used here. Not saying the setup is wrong, only looking for further clues in getting this solved.

I'm hoping there are no failed scripts in the log or crash reports that would indicate a malfunction of the required interface reload on a linkup event.


Cheers,
Franco

[dragon2611]

Yes, But I think it just happened to be the order I created the interfaces in meant I ended up setting up the other connection first.

Btw most LAN > WAN traffic is set through a gateway group with the PPPoE as Tier 1 and the other connection "Wan" as Tier2

[franco]

The change will be in 16.7.1, now we'll only have to figure out what else goes wrong. When this happens again, can you provide System log and PPP log? We'll also need to know whether an interface save on your PPPoE + Apply will bring the connectivity back.


Thanks,
Franco