[SOLVED] 1000Mbps fibre connection only achieves 100Mbps on OPNsense

[chris.dempsey]

UPDATE - schoolboy error - the LAN connection from OPNsense to the switch had negotiated at 100Mbps.

You'd think I would have noticed the LED liot up orange (for 100) instead of green (for 1000).

Original post follows.

Can anyone help diagnose the issue described below please?

My gigabit FTTP connection was working at full speed, 900Mbps down/100Mbps up for 6 months without issue.  The line was regraded from consumer to business so that 8x public IP addresses could be added, at this point line speed became wildly intermittent and as low as 0.8Mbps down.

The ISP advised the NOC replaced some kit at the exchange and they are no longer seeing the original issue (Ethernet Handover Failure).

OPNsense now achieves a consistent 90Mbps down/100Mbps up

However with both a second OPNsense box and the ISP supplied Fritzbox line speeds are the full 900Mbps down/100Mbps expected.

I think a setting may have been changed accidentally when messing with the configuration, however can't rule out a coincidental hardware failure.  In relation to that the latest update to OPNsense 23.1.11-amd64.

Plugins

• Zenarmour (basic)
  Crowdsec

Hardware

• Dell R210ii
  Xeon E3 1270 3.4Ghz Quad Core
  32GB DDR 1333
  Dual Port Broadcom NetXtreme II BCM5716 Gigabit Ethernet
  Netgear GS324 and GS308 gigabit switches (both new)

I have an Intel E1G44ET Gigabit ET Quad Port Adapter but this is not fitted nor am I sure if it would help (although from what I've read Intel NICs are billed as the preferred choice).

Things I have tried

• read dozens of posts
  have power cycled
  replaced the cables from ONT to WAN port on R210ii and GS324 to GS308 with the same cables that work on the second OPNsense box

`$ ifconfig -a` returns

```
media: Ethernet autoselect (1000baseT <full-duplex>)
```

I am not convinced this is the WAN port OPNsense sees though because from the GUI two interfaces appear to be related to bce0, even though this is the port connected to te ONT.

Under Interfaces > Assignments the WAN (opt2) entry shows `pppoe (bce0) and the New interface dropdown shows `bce0` as available.  I'm unsure why `bce0` appears twice.

I tried deleting the WAN interface assignment and adding it again but the behavior remains the same.


`$ ifconfig | grep media` returns

```
media: Ethernet autoselect (1000baseT <full-duplex>)
media: Ethernet autoselect (100baseTX <full-duplex>)
```

I guess the two entries relate to `bce0` WAN and `bce1` LAN?

From other posts users looking at the same symptoms refer to the `Speed and duplex`. My GUI shows

• Interfaces > LAN has dropdown for `Speed and duplex` under the MSS setting
  Interfaces > WAN does not have the `Speed and duplex` option

I'm unsure why the dropdown is missing from the WAN interface or if this is relevant (may be because this is PPPoE?). The solution on the post I saw this under was to remove and recreate the interfaces which I've already done.

Any ideas welcomed.

Thanks,

Chris

[Taunt9930]

I'm a bit confused what the issue/exam question is? You say -

 "OPNsense now achieves a consistent 900Mbps down/100Mbps up

However with both a second OPNsense box and the ISP supplied Fritzbox line speeds are the full 900Mbps down/100Mbps expected."

So everything is working?

On one question, for a PPPoE connection, the parent interface isn't assigned by default (not an issue usually), so you will see it again as unassigned in the interfaces menu, as well as shown as the PPPoE.



Sent from my SM-S918B using Tapatalk

[Taunt9930]

Assume that is a Typo, and you have performance issues, only getting 100Mbp/s down as per the title, with multiple different routers?

I'm guessing the issues started only when you moved to the new business package?

I'm guessing your ISP is Zen?

I'm assuming when you moved to the business package, you were subject to a GEA migration - whereby you are moved from the BT Wholsesale GEA in your exchange to Zens own GEA - you might be able to see this if you log into the Zen portal, go to the 'old' portal, and look at order history.

Assuming all of the above is correct, it is a Zen issue. They have some kind of issue with their GEA links on connections >500Mb/s that 1. They can't work out, and 2. Largely refuse to accept, claiming it is an historic issue already fixed.

Go onto Think Broadband Forums, and Kitz.co.uk forums and do a search for 'Zen GEA' - you'll find reports of the same issue...On Kitz, look in the FTTP Issues section...there is quite a recent one with symptoms as bad as yours....

....good luck.

Sent from my SM-S918B using Tapatalk

[chris.dempsey]

> Assume that is a Typo, and you have performance issues, only getting 100Mbp/s down as per the title, with multiple different routers?
Yes, my original post had a typo in the body, downstream maxes out at 90Mbps and has done for weeks although it was worse (wildly intermittent from 0.8 to about 500Mbps) before stabilising at this level.

> I'm guessing your ISP is Zen?
At first I thought you knew Zen are handing out Fritzboxes now but from what you say it seems their issues are all over the tech forums now.

The reason I raised this on the OPNsense forum now was Zen advised the issue is resolved as they replaced kit at the Exchange.  So to verify I hooked the ONT up to their Fritzbox and connected a laptop by LAN.  It ran 800/100 so it looked like they did indeed resolve the issue.  However on connecting the ONT back to the OPNsense appliance and wider office network the 90/100 issue presented.

So suspicion fell on the OPNsense appliance as potentially I knocked a setting over the past 10 weeks attempting to figure this out.  I've now spent about 6 hours there which now seems like yet more time burned for nothing.

The original symptoms were always wildly fluctuating speeds.  Potentially I just got lucky with 800/100 when the Fritzbox was on.  I'll hook it up again at the weekend to see if runs that speed again or if it's still just fluctuating.

> I'm guessing the issues started only when you moved to the new business package?
The trigger was definitely when the 8x IPs were assigned, guy on the phone said "I'm doing it now, allow 15-20min for propagation, officially it's 24hrs but usually 20min.

The saga started 3 weeks earlier.  Zen regraded the line, adjusted billing and issued paperwork but the IP block never appeared. Called, was told 1 week, no IPs, called again, told 1 week, no ips, called again and this time your man could make the switch immediately.

Line speed was 900/100 throughout the saga, soon as he hit the button the problems begain.

Its been mental, 5 engineers on site in a two week period back at the start. Every time they called in advance to advise the remote test shows the fault as Ethernet Handover Failure which relates to Zen's rack at the Exchange which Openreach can't touch.

> I'm assuming when you moved to the business package, you were subject to a GEA migration - whereby you are moved from
> the BT Wholsesale GEA in your exchange to Zens own GEA - you might be able to see this if you log into the Zen portal,
> go to the 'old' portal, and look at order history.
Spot on, the customer portal shows an order for "Zen FTTP GEA Migration".

> Assuming all of the above is correct, it is a Zen issue. They have some kind of issue with their GEA links on
> connections >500Mb/s that 1. They can't work out, and 2. Largely refuse to accept, claiming
> it is an historic issue already fixed.
Appreaciate you pointing this out, I hadn't looked though the forums assuming it was largely isolated to myself and a few othe Zen customers.

One of the Openreach Engineers had searched the fault code on a closed group forum the Engineers use, at the time there were 6 mentions of the code and 5 related to Zen.  He said he'd never seen that particular fault code before.

> Go onto Think Broadband Forums, and Kitz.co.uk forums and do a search for 'Zen GEA' - you'll find reports of the
> same issue...On Kitz, look in the FTTP Issues section...there is quite a recent one with symptoms as bad as yours....
> ....good luck.
I'll take a look now.

Appreciate your insights.

[chris.dempsey]

Issue is resolved - embarrassingly it turns out I can't tell the difference between orange and green - see updated first post if you fancy a laugh at my expense.

[Taunt9930]

Excellent. At least you haven't been caught up with the whole GEA issue, as that would be much more difficult to navigate out of!

A lesson for us all, check and check again - I've done it - sometimes we see (or don't see) what we want to when we get all tied up in an issue.

Sent from my SM-S918B using Tapatalk

[chris.dempsey]

At least you haven't been caught up with the whole GEA issue

I'm not sure we'll ever find out what happened.  I spent a while reading the thread you referred me to last night - specifically the epic 16 page thread.

Something definitely went wrong during the IP address provisioning process.  The timing was immediate on their man pressing the button and the consumer to business regrade took place 3 weeks prior.

5 Openreach engineers all insisted the issue was with Zen's rack at the Exchange, plus a sixth higher level Openreach engineer who had clearance for the Exchange (N27 level was it?) agreed.  Nothing was touched at my office, the ONT was never replaced and 3 different routers with new cables performed the same.  Even the workstation wired directly to the ONT with a dial up PPPoE connection was intermittent.

Interesting to learn about the wider issue on Zen's side though. 

Plus I learned a couple of things myself.  Next time I'll start with is it switched on? And if it's not that we'll know it's DNS, it's always DNS. Except when it's cache invalidation.

Thanks again.