Hey guys,
I implemented the proposed solution as per this thread https://forum.opnsense.org/index.php?topic=28725.0 (https://forum.opnsense.org/index.php?topic=28725.0) by disabling Spanning Tree (not sure is the safest thing to do in any case) on my Meraki Switches (2x MS125-24p and 1x MS120-8 running on FW: 14.33.1), however, interfaces are still showing errors on ax0 for both LAN (physical interface) and its VLANS (LAN being the parent interface).
Any words of wisdom anyone? My DEC3840 has given me headaches with 10G since day1 and I'm running out of options here...
This is annoying and only happens on ax0/1 interfaces, igb0/1/2/3/4 are looking good (with or without STP enabled)
Cheers,
NW4FUN
Did you disable hardware CRC offloading?
What real world impact do these error counters have? It seems that e.g. Netmap (Zenarmor or Intrusion Detection IPS mode) with VLANs generates spurious errors...
Cheers,
Franco
Are you using SFP+ 10Gb modules?
Cisco Meraki doesn't have 10Gb sfp+ (it has 1Gb SFP mini-GBIC), so you might get some errors due to that
Oh my bad, you were talking about Cisco Meraki Cloud Managed series switches (just noticed there are 2 different Meraki series switches), yea those do have 10Gb SFP+ ports
Quote from: franco on July 06, 2022, 08:35:56 AM
What real world impact do these error counters have? It seems that e.g. Netmap (Zenarmor or Intrusion Detection IPS mode) with VLANs generates spurious errors...
Cheers,
Franco
Hi Franco,
I do not know what's the "real world impact" for this, however, errors are showing not only on the VLANs, also on physical interfaces (LAN). Bizarrely enough I'm experiencing those errors exclusively on ax0/1 ports, igb0/1/2/3 are totally fine error counting wise.
NW4FUN
Error counting keeps growing at a massive pace...
Any idea anyone?
Quote from: NW4FUN on July 09, 2022, 08:50:30 AM
Error counting keeps growing at a massive pace...
Any idea anyone?
Ok I searched Cisco specification site, since they do sell models with different port amount and types of SFP ports as well (meaning model with 48 ports has SFP+ where exactly same model with 8 or 24 ports only has 1Gb SFP)
https://documentation.meraki.com/MS/MS_Overview_and_Specifications/MS125_Overview_and_Specifications <--- according those specifications, Meraki MS120 series switches don't have 10Gb SFP+. so it might be your 8 port meraki causing those (last numbers in each model name defines the amount of ports the switch has).
Nothing to worry about, should be fixed when you switch the 10Gb SFP+ cable connected to MS120-8 to 1Gb SFP cable. Basically could indicate that errors arre related to how many packets are dropped due to port not being able to receive them at faster rate, so if all works just fine, then you can just ignore it
And here are specifications for MS120 switches
https://documentation.meraki.com/MS/MS_Overview_and_Specifications/MS120_Overview_and_Specifications
It is easy to confuse with SFP unless you have been working with those. SFP is 1Gb and SFP+ 10Gb and it is backwards compatible with SFP (to my knowledge, but there are SFP to ethernet modules as well https://www.cisco.com/c/en/us/products/collateral/interfaces-modules/small-business-network-accessories/datasheet-c78-741408.html).
But I suspect that reason for errors is basically because you have connected 10Gb SFP+ to SFP port and therefore, everything isn't going through and your network gear has to wait for those packets or something (Collisions are what you really want to avoid).
You can test if it is something you should fix by storing 500GB or 1TB file to your computer connected to MS125-24 switch and send it to computer connected to MS120-8 switch. If file doesn't get corrupted and can be read, repeat the test with multiple files which together are huge and if you can, add few more clients to the test.
Basically only thing that should be effected, is that it takes bit more longer than what average 10Gb/s shared between amount of clients with 1 Gb/s connections would be. To get 10Gb/s speeds, you need to have computers with 10Gb network interfaces connected to 10Gb ports.
Also you need to consider Switching Capacity of each switch, obviously there will be some delay when faster switch is pushing things to slower one, this is why it is recommended to combine 2 of same models, even if you don't need all of their ports.
It is worth to check, since 1Gb/s ethernet can reach higher speeds than actually 1Gb/s so despite some errors, connections could hold quite well untill total bandwidth that switch has to handle exceeds it's switching capacity by a long shot.
If you do test sending files, remember that network speed is counted in bits per second (so 1GB file is actually 10Gb not 1Gb way you calculate it is bits per second / 8 ), so there it's not question of if issues could occur, but more likely how many files or clients can the switch handle without issues other than speeds slowing down.
Quote from: Vilhonator on July 09, 2022, 09:26:20 AM
Quote from: NW4FUN on July 09, 2022, 08:50:30 AM
Error counting keeps growing at a massive pace...
Any idea anyone?
Ok I searched Cisco specification site, since they do sell models with different port amount and types of SFP ports as well (meaning model with 48 ports has SFP+ where exactly same model with 8 or 24 ports only has 1Gb SFP)
https://documentation.meraki.com/MS/MS_Overview_and_Specifications/MS125_Overview_and_Specifications <--- according those specifications, Meraki MS120 series switches don't have 10Gb SFP+. so it might be your 8 port meraki causing those (last numbers in each model name defines the amount of ports the switch has).
Nothing to worry about, should be fixed when you switch the 10Gb SFP+ cable connected to MS120-8 to 1Gb SFP cable. Basically could indicate that errors arre related to how many packets are dropped due to port not being able to receive them at faster rate, so if all works just fine, then you can just ignore it
Hi Vilhonator,
Thanks for taking the time to look into this, however you must have confused my topology...
All it matters is the core layer which is a MS125-24p where the DEC3840 is hooked into.
The access layer is yet another MS125-24p which is bound to the core via 20G LACP link.
The small MS120-8 you're referring to, is a small service switch indeed connected at 1G into the access layer exclusively used for serving 2 devices.
For clarity I'm adding a screenshot of my topology.
Also, I'm adding a screenshot of the interface errors on OPNsense, hopefully someone (Franco??) may help in shedding some light here.
Cheers,
NW4FUN
Interface errors screenshot
Hmm shouldn't really be much of an issue.
If ports which indicate errors are connected to switches and all devices connected to them get internet connection through the firewall, then errors could happen becase maybe traffic going thru the firewall being to huge and Opnsense showing errors in too high detail to be bit confusing. (https://shop.opnsense.com/product/dec3840-opnsense-rack-security-appliance/ <----- check the System Performance statistics of your model)
Whenever you do face errors of this kind, you should check if it is something that needs to be taken care of, since it indicates that firewall or some other device connected to it just can't keep up.
If that is the case, then simple traffic shaping (https://docs.opnsense.org/manual/how-tos/shaper_share_evenly.html) should fix it.
Also it could be just some bug in Opnsense and worth reporting (which I think you allready have when you created the post)
Shaping a 10G/10G link on a FW that is supposed to be able to route 17Gbps NGFW to me seems at least odd...
@Franco - what's your take on these errors?
Quote from: NW4FUN on July 11, 2022, 06:53:29 PM
Shaping a 10G/10G link on a FW that is supposed to be able to route 17Gbps NGFW to me seems at least odd...
@Franco - what's your take on these errors?
That's not odd, it's simplicity in its finest, reason why your firewall might get overwhelmed, is because you have only one firewall and multiple switches forwarding internet connections in and out thru it. If too many computers are sending and recieveing packages from and to the internet at the same time, your firewall can get overwhelmed and errors appear (that's how simplest form of DoS attacks work, send and request too many packages till firewall just dies out and shuts all connections)
It's same as with connecting too many electronics to same power distributor, when power consumption exceeds (usually 10A), fuse will kick in and shuts down power, firewalls and switches aren't any different except they don't shut connections untill maximum threashold is reached or they get massively overwhelmed.
Also you don't shape it to 10Gb/s, that way connections will suffer, instead you shape it around 17Gb/s. You can exceed maximum throughput to certain extent, but not massively much.
Also way you count what kind of bandwidth your network has is by multiplying the amount of ports in use with maximum speed + some overhead.
for example if all 4 1Gb ports are connected, then theoratical maximum bandwidth is somewhere between 4 to 10Gb (1Gb x4 = 4 and 2.5Gb * 4 = 10Gb)
Yes, 10Gb and 1Gb ports can reach higher bandwiths (best I have reached with 1Gb port is 2.5Gb/s)
It could also be a bug, since I doubt that your network could reach much over 17Gb/s bandwidth since first of all, your ISP is most major limiter of that (unless your Internet speed is around 1Gb or close to that)
But dropped packets due to congestion will not lead to interface errors. On a modern system Ethernet is full-duplex with flow control. There should be no errors at all on the link. If the kernel drops a packet because of a queue overflow, that's not an interface error - or is it? I might be mistaken here. But up to now I thought interface errors are strictly transmission errors on the wire. Which should not happen.
Quote from: pmhausen on July 11, 2022, 07:49:13 PM
But dropped packets due to congestion will not lead to interface errors. On a modern system Ethernet is full-duplex with flow control. There should be no errors at all on the link. If the kernel drops a packet because of a queue overflow, that's not an interface error - or is it? I might be mistaken here. But up to now I thought interface errors are strictly transmission errors on the wire. Which should not happen.
Good point. Forgot that completely. I would read what kinds of things could be causing these errors. Quite possibly all is fine and either internet or your internal network side, some servers and computers encounter time out errors, that is if they could be the culprit.
By what I mean, would be for example trying to connect a website or server which is down kind of situations.
For example computer downloading OS updates and client shutting down due to power outage or forced shutdown or user closing browser while it tries to connect or download something without canceling the download or manually stop connection first could be one such thing
Hi Vilhonator,
I'm genuinely puzzled from what I'm reading as this is extremely misleading and I'm not exactly sure where this information (2.5G from a 1G ETH???) is coming from!!
As far as the link goes, as I stated quite clearly in a previous post, my ISP brings in a 10G Download/10G Upload symmetric DIA, which makes the whole idea of shaping simply pointless.
As per your supposed congestion point, I'm not quite sure what you are referring to...my topology is based on the three-layer hierarchical model (with the only difference that core and distribution are aggregated in one due to the simple design). Between the layers there's a 20G LACP link and switches are capable of forwarding 176 Gbps.
I really appreciate your efforts, however, if you keep misreading (or misunderstanding) my previous posts where I'm describing the problem, this is not helping as it just adds more confusion.
Let's make this very simple.
If traffic to all your VLANs go through single ethernet cable (you haven't set OpnSense SFP+ ports to VLANs and setup all SFP ports on your Switches to VLAN Trunk mode), then maximum speed to all VLANs is 10Gbps (ethernet cable is able to transfer maximum of 10Gbps in or out and it is shared between VLANs, you can't have over 10Gb going through multiple VLANs through single ethernet cable) that also applies to ethernet ports.
Spanning tree is also good thing to use, but most important is to make sure, that traffic between all vlans doesn't go trhough single cable if cable isn't able to handle the traffic alone.
Don't know the english term of the factor, but due to it, ethernet and SFP is able to have slightly faster or slower speeds than what it is marketed for (which is why network card drivers can have difference), I mean your switches are able to forward traffic up to 176Gbps, that alone should make you doubt it, if ethernet and SFP would be limited to whatever speeds they are marketed for and your switches don't have 2 100Gb QSFP ports.
How am I able to get 2.5Gbs while transferring stuff from 1 computer to another computer locally? Simple, 1Gb/s out from 1 VLAN to 1Gb/s in to another VLAN = 2Gbps + factor I am referring to.
Quote from: Vilhonator on July 11, 2022, 07:41:34 PM
Yes, 10Gb and 1Gb ports can reach higher bandwiths (best I have reached with 1Gb port is 2.5Gb/s)
What I am refferring there, is to
BANDWIDTH, it's calculated in bits per second (as all network stuff is) but refers to total traffic going in and out your network, including internet.
For example you don't need 10Gb internet because your clients have to be able to download and upload stuff to the internet at 10Gbps speed, you need 10Gb internet to provide enough bandwidth for all your clients being able to watch netflix etc. online without latency issues. Sounds silly, but 126 clients even with just 1Gb connection isn't exactly small bandwidth eater. Watching netflix @ 4K/Ultra HD requires 15Mbps (which with 126 clients would be around 1.9-2Gb/s)
That's where QoS and Traffic shaping comes to play, you can prioritize different services and set bandwidth limitations, to make sure that your firewall doesn't just let everything that is allowed through at max speed it can handle.
Traffic shaping is bit less of an hassle to setup, but it is a must, when you need to make sure there's enough bandwidth to go around.
I wasn't expecting support from a CCIE on here, however it'd have good to hear from somebody who knows what's he/she is talking about...
@franco - any words of wisdom from your end?