OPNsense Forum
English Forums => Hardware and Performance => Topic started by: packetmaster on October 06, 2020, 05:44:08 pm
-
Hi all,
Has anyone experienced any of the issues with Intel NICs in the latest OPNSense (20.x) outlined in this thread?
https://forum.opnsense.org/index.php?topic=5511.0
I am looking to implement a new firewall set up with latest version of OPNSense. I am looking at hardware from guys like Jetway, Qotom, Protectli with newer Kaby Lake processors. Jetway and Qotom use the i210 / i211 and i350 NICs while the Protectli uses older 82583V.
I would think the newer NICs would be better but that thread has me worried. That said, seems like Amazon reviews have a lot of people using PFSense or OPNSense on the Qotom which lead me to believe the i210 / i350 issues have been sorted out, or Wener (poster in the thread above) was running into issues with the particular Jetway hardware.
-
FWIW...I'm using 82575 based cards and no issues so far....that said, just built my box and just recently started using OPN so who knows at this point.....wanted better/newer cards, but couldn't force myself to spend the extra money on the newer ones.....maybe that'll come back to bite me :P
-
i211 doesn't work really stable .. rest is ok
-
Using a QOTOM with a i211. Experiencing daily drops in the interface assigned to LAN. Logs are unclear. I'm not the only one. There appear to be a couple of threads about it. The i211 appears to be the common denominator.
I think its Opnsense related, since the same hardware worked perfectly on PFsense.
-
> I think its Opnsense related, since the same hardware worked perfectly on PFsense.
Modify message
Qualified statements please: Which OPNsense and pfSense versions are we comparing? You are aware the two have different OS versions in production releases? ;)
Cheers,
Franco
-
> I think its Opnsense related, since the same hardware worked perfectly on PFsense.
Modify message
Qualified statements please: Which OPNsense and pfSense versions are we comparing? You are aware the two have different OS versions in production releases? ;)
Cheers,
Franco
Fair comment. The pfSense version was the latest stable. (2.4.5) pfSense doesn't post updates very often. Their stable branch is still on BSD 11 I think. They were on the verge of 2.5, but i'm not sure if that has been released yet. I lost interest in pfSense after finding out more about the organizations history.
Every Opnsense version I've tried has been BSD12.x. I ran 20.7 for a bit over a week then upgraded to 21.1.
The problem may lie with BSD12.
-
I think so too. I have some more things to merge from Intel from later stable patches after 12.1 was finalised anyway and look for more, but I need a bit of time...
https://github.com/opnsense/src/issues/100
Cheers,
Franco
-
Nice to read, franco, that you are working on the intel issues!
Can we hopefully forget the issues with the common intel i211 NICs in the next months?
Would be great!
-
I know there are a lot of priorities, but this seems like it should be a higher one.
This "glitch" effectively means the newer OPNSense versions cannot work reliability on many of the QOTOM/Protectli mini-computers that you can get off amazon/aliexpress that are very popular to run with PFSense/OPNSense. A majority of them use the i211 chipset.
I'm actually a little surprised more people aren't remarking on this. It makes me feel as though there is something unusual about my configuration. I suppose its possible most home/SOHO users may not notice dropouts several times a day.
-
Some of the PC Engines APU2/3/4 also have the i211 onboard. (Some do have the i210, which seems fine).
Since these APU boards are widely used: Has no owner noticed yet? (I have just APUs with i210.)
-
I am running an apu4d4 with the 211 chipset and don't experience any issues.
2 of the 4 interfaces are combined in a lagg, plugged into a Cisco switch with LACP, on top of that I run 2 VLANs for LAN and OPT1. The system does not have a WAN interface.
Sorry, sort of - just humming along.
-
To be honest, I probably wouldn't have noticed if we didn't have people working from home due to Covid. Lots of video conference calls getting interrupted. According to the logs, the dropouts happen 3 or 4 times a day, seemingly at random (no idea what triggers it). It flipflops for about 10-15 minutes then comes back up.
A lot of the time, this happens overnight, or during work hours. It is very possible someone could be having this issue and simply not noticing it.
If you look through your system logs, you will see something like:
igb0: link state changed to DOWN
several times per day.
Then up and down a few more times. If this happened at a time when you weren't actively using your internet connection, you may not have noticed, or assumed it was a one-off.
-
Nope ;)
root@opnsense:/var/log # uptime
5:17PM up 5 days, 1:37, 1 user, load averages: 0.50, 0.43, 0.38
root@opnsense:/var/log # dmesg | grep 'link state'
root@opnsense:/var/log # grep 'link state' *
I used to have minor problems with 20.7. Initially I bridged all 4 interfaces of my apu4 to use the device as a small home switch - which JeGr despises but I find perfectly reasonable. Sometimes after config changes I would lose connectivity to my desk where my Mac is connected via a Thunderbolt docking station with Ethernet and only a reboot of OPNsense would bring it back.
Since I bought a Cisco 2960-L 16-port switch and connected all servers (2x TrueNAS, 1x OPNsense) to that via LACP and then access ports to other systems in the apartment I have not had any problems.
So I *guess* the device on the other end can play a role, too.
-
That's kind of what I was afraid of.
I've replaced the cable, switch, and isolated the physical network down to a few devices and it has not resolved the dropouts. I've even changed the port (the device has 4).
When I started seeing threads about the i211 pop up, I assumed that was it. Especially since it wasn't happening on older BSD versions.
I'm starting to wonder if I have a bad QOTOM unit. Given the sketchy nature of the manufacturing on these things, I suppose it's possible.
-
1) I simply dont understand why pcengines switched away from i210 and use i211 in their higher numbered APUs (APU4, APU5, APU6, these are not even listed on pcengines.ch, the secrecy inner workings of this swiss company is confusing as hell to me). As you all should be aware i211 is inferior to i210 (in contrast what the model number may suggest, bigger number != better product in this case). i210 supports 4 TX&RX queues per port, i211 has only 2 TX&RX queues per port. Is significant difference, because Receive Side Scaling can distribute incoming (TX) packets between the CPU cores, if there are as many queues as CPU cores. Otherwise only 1 or max. 2 cores will get all the processing work of the incoming packets, while the rest of the CPU cores will sit idle nothing to do. That is a real life issue above 100Mbit speed, as the APU 1Ghz AMD embedded CPU has very weak single core performance, and all 4 cores would be needed to process packets at 1Gbit, which is more or less impossible with the i211 nic in many of the APUs.
Pcengines does not seem to work on a future improvement on this dead horse APU design to put higher clocked AMD SoC in to the PCB, and replace i2xx with i3xx that already has 8 TX & RX queues per port.
-
1) I simply dont understand why pcengines switched away from i210 and use i211 in their higher numbered APUs (APU4, APU5, APU6, these are not even listed on pcengines.ch, the secrecy inner workings of this swiss company is confusing as hell to me). As you all should be aware i211 is inferior to i210...
I don't understand a lot of your post, but I do understand human nature. Usually when a company/manufacturer makes a decision to downgrade a product, but give it a higher model number, the reason is usually always "money".
I would wager the i211 is cheaper to build.
-
It's perfectly normal that there are hardware components with different performance. It does not always have to be the fastest one. Rather, everything has to be coordinated and fulfill the purpose. And price and power consumption also play a role for certain applications.
And for me, the APUs are very sensible low-cost devices. Whether with i210 or i211. In practice, this does not necessarily make a big difference.
And if it has to be more performance, there are many other solutions. But then they also cost more. And burn more energy.
But we are a bit off topic. That the i211 makes problems here is not caused by PC Engines but a fundamentally different problem.
-
Hi all!
For months now I'm running the actual opnsense versions on my hardware Qotom Q370G4 ram8G ssd256G
with no issues regarding link state up/down changes!
The ports are connected to
Cisco cable modem EPC 3212
2x TP-Link switches TL-SG108E
LogiLink switch NS0106
I'm happy ;)
-
Has this been fixed? I am running qotom-like system, with 4 i211 NICs and the interfaces get randomly disconnected many times per day (~once per hour). All interfaces are connected to Unifi switches.
Running a fresh install of OPNsense 21.1.7
-
An update on this topic...
I just updated from 20.1.x to 22.1 (beta, latest as of ~Dec 28). I really could not stay on 20.1 any longer as security updates are critical (for any network!).
Problem still exists. It has happened twice in about 40 hours.
I'm running a Qotom with four i211 Ethernet ports.
More details in the thread I started here: https://forum.opnsense.org/index.php?topic=20456.0
Example log message:
2022-01-01T00:33:28-05:00 Error opnsense-devel /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for static lan(igb1)
It seemed to be related to EEE in 20.1 and disabling EEE kept the network very stable (like 100s of days with interface flapping).
I'm investigating the EEE settings in 22.1. What else can be done to diagnose this?
-
Have you tried to install a switch between your OPNsense and ISP;s equipment ?
Namely some switch with controllable EEE , so you could additionally control EEE of the counterpart to your i211.
Seems to me that disabling EEE on your i211 NICs is necessary regardless how much the FreeBSD driver is being updated & bugfixed. So I would ruled out the driver from the list of possible causes.
-
PS C:\Program Files\Hrping> .\hrping.exe 192.168.2.1 -t
This is hrPING v5.07.1149 by cFos Software GmbH -- http://www.cfos.de
Source address is 192.168.2.224; using ICMP echo-request, ID=246f
Pinging 192.168.2.1 [192.168.2.1]
with 32 bytes data (60 bytes IP):
setsockopt IP_HDRINCL failed: Error 10013: An attempt was made to access a socket in a way forbidden by its access permissions.
From 192.168.2.1: bytes=60 seq=0001 TTL=64 ID=64c9 time=0.454ms
From 192.168.2.1: bytes=60 seq=0002 TTL=64 ID=0c66 time=0.392ms
From 192.168.2.1: bytes=60 seq=0003 TTL=64 ID=13b9 time=0.425ms
From 192.168.2.1: bytes=60 seq=0004 TTL=64 ID=077d time=0.402ms
From 192.168.2.1: bytes=60 seq=0005 TTL=64 ID=8630 time=0.408ms
From 192.168.2.1: bytes=60 seq=0006 TTL=64 ID=60d1 time=0.346ms
From 192.168.2.1: bytes=60 seq=0007 TTL=64 ID=409c time=0.437ms
From 192.168.2.1: bytes=60 seq=0008 TTL=64 ID=cf86 time=0.425ms
From 192.168.2.1: bytes=60 seq=0009 TTL=64 ID=4313 time=0.449ms
From 192.168.2.1: bytes=60 seq=000a TTL=64 ID=e4f0 time=0.422ms
From 192.168.2.1: bytes=60 seq=000b TTL=64 ID=8098 time=0.384ms
From 192.168.2.1: bytes=60 seq=000c TTL=64 ID=f150 time=0.409ms
From 192.168.2.1: bytes=60 seq=000d TTL=64 ID=c33f time=0.337ms
From 192.168.2.1: bytes=60 seq=000e TTL=64 ID=0857 time=0.428ms
From 192.168.2.1: bytes=60 seq=000f TTL=64 ID=3721 time=0.434ms
From 192.168.2.1: bytes=60 seq=0010 TTL=64 ID=eac0 time=0.401ms
From 192.168.2.1: bytes=60 seq=0011 TTL=64 ID=2a48 time=0.486ms
From 192.168.2.1: bytes=60 seq=0012 TTL=64 ID=c82b time=0.413ms
From 192.168.2.1: bytes=60 seq=0013 TTL=64 ID=c2b5 time=0.369ms
From 192.168.2.1: bytes=60 seq=0014 TTL=64 ID=4600 time=0.409ms
From 192.168.2.1: bytes=60 seq=0015 TTL=64 ID=d4bb time=0.403ms
From 192.168.2.1: bytes=60 seq=0016 TTL=64 ID=18d3 time=0.400ms
From 192.168.2.1: bytes=60 seq=0017 TTL=64 ID=43d3 time=0.402ms
From 192.168.2.1: bytes=60 seq=0018 TTL=64 ID=8b08 time=0.426ms
From 192.168.2.1: bytes=60 seq=0019 TTL=64 ID=d940 time=0.299ms
From 192.168.2.1: bytes=60 seq=001a TTL=64 ID=72ac time=0.366ms
From 192.168.2.1: bytes=60 seq=001b TTL=64 ID=7d4c time=0.405ms
From 192.168.2.1: bytes=60 seq=001c TTL=64 ID=8ac4 time=0.407ms
[Aborting...]
Packets: sent=28, rcvd=28, error=0, lost=0 (0.0% loss) in 13.501999 sec
RTTs in ms: min/avg/max/dev: 0.299 / 0.404 / 0.486 / 0.036
Bandwidth in kbytes/sec: sent=0.124, rcvd=0.124
PS C:\Program Files\Hrping> .\hrping.exe 192.168.2.1 -t
This is hrPING v5.07.1149 by cFos Software GmbH -- http://www.cfos.de
Source address is 192.168.2.224; using ICMP echo-request, ID=884c
Pinging 192.168.2.1 [192.168.2.1]
with 32 bytes data (60 bytes IP):
setsockopt IP_HDRINCL failed: Error 10013: An attempt was made to access a socket in a way forbidden by its access permissions.
From 192.168.2.1: bytes=60 seq=0001 TTL=64 ID=8790 time=0.275ms
From 192.168.2.1: bytes=60 seq=0002 TTL=64 ID=073c time=0.263ms
From 192.168.2.1: bytes=60 seq=0003 TTL=64 ID=f4b9 time=0.297ms
From 192.168.2.1: bytes=60 seq=0004 TTL=64 ID=eeaa time=0.274ms
From 192.168.2.1: bytes=60 seq=0005 TTL=64 ID=4512 time=0.203ms
From 192.168.2.1: bytes=60 seq=0006 TTL=64 ID=a8e4 time=0.273ms
From 192.168.2.1: bytes=60 seq=0007 TTL=64 ID=ae4a time=0.284ms
From 192.168.2.1: bytes=60 seq=0008 TTL=64 ID=ab11 time=0.255ms
From 192.168.2.1: bytes=60 seq=0009 TTL=64 ID=c840 time=0.291ms
From 192.168.2.1: bytes=60 seq=000a TTL=64 ID=0ef1 time=0.261ms
From 192.168.2.1: bytes=60 seq=000b TTL=64 ID=8e7a time=0.266ms
From 192.168.2.1: bytes=60 seq=000c TTL=64 ID=a4ff time=0.246ms
From 192.168.2.1: bytes=60 seq=000d TTL=64 ID=6a43 time=0.247ms
From 192.168.2.1: bytes=60 seq=000e TTL=64 ID=631b time=0.256ms
From 192.168.2.1: bytes=60 seq=000f TTL=64 ID=9ca2 time=0.284ms
From 192.168.2.1: bytes=60 seq=0010 TTL=64 ID=f874 time=0.242ms
From 192.168.2.1: bytes=60 seq=0011 TTL=64 ID=3b3b time=0.234ms
From 192.168.2.1: bytes=60 seq=0012 TTL=64 ID=9937 time=0.279ms
From 192.168.2.1: bytes=60 seq=0013 TTL=64 ID=4fc9 time=0.240ms
From 192.168.2.1: bytes=60 seq=0014 TTL=64 ID=4153 time=0.287ms
From 192.168.2.1: bytes=60 seq=0015 TTL=64 ID=9ceb time=0.270ms
From 192.168.2.1: bytes=60 seq=0016 TTL=64 ID=400e time=0.301ms
From 192.168.2.1: bytes=60 seq=0017 TTL=64 ID=8c82 time=0.273ms
From 192.168.2.1: bytes=60 seq=0018 TTL=64 ID=2109 time=0.257ms
From 192.168.2.1: bytes=60 seq=0019 TTL=64 ID=36ff time=0.229ms
From 192.168.2.1: bytes=60 seq=001a TTL=64 ID=afb6 time=0.281ms
From 192.168.2.1: bytes=60 seq=001b TTL=64 ID=1633 time=0.263ms
From 192.168.2.1: bytes=60 seq=001c TTL=64 ID=5aec time=0.222ms
From 192.168.2.1: bytes=60 seq=001d TTL=64 ID=c85b time=0.279ms
From 192.168.2.1: bytes=60 seq=001e TTL=64 ID=2d6d time=0.311ms
[Aborting...]
Packets: sent=30, rcvd=30, error=0, lost=0 (0.0% loss) in 14.514310 sec
RTTs in ms: min/avg/max/dev: 0.203 / 0.264 / 0.311 / 0.024
Bandwidth in kbytes/sec: sent=0.124, rcvd=0.124
PS C:\Program Files\Hrping>
Wanted to share my findings. changed from a qotom 6 port i211 to a supermicro x11 with 4 i210 ports. Red is the i211.
green is i210
pretty much proves the i210 is better. this test was done with the exact same opnsense cfg and done within 1 hour of each other. i210 more consistent and lower latency.
the x11 does have a faster processor, but shouldnt have anything to do with this as i could hit 1gbps speeds without hitting 50% cpu on the qotom.