WAN link gone sometimes (igb driver, I211 nics), ifconfig d/u fixes it

[franco]

Fingers crossed for 11.1.

[grolon]

Hi all, i have the same issue

My test server is a Lenovo 3000 J Series
Pentium 4 HT 3.2 GHz, 1GB RAM, IDE HDD
OPNsense 17.7.3 updated (2017-SEP-25)

Configured my WAN and LAN, and after a couple of minutes WAN is DOWN.
WAN and LAN are both Realtek 8169 PCI GBE Family Controllers cards

Installed Ubuntu Server 16.04 to test my hardware and everything is OK.

Just OPNsense is having this issue,


thanks

[grolon]

I did this

root@opnsense:~ # cat /boot/loader.conf.local
dev.re.0.eee_disabled=1
dev.re.1.eee_disabled=1
hw.pci.do_power_suspend=0
dev.re.0.fe=0
dev.re.1.fe=0

root@opnsense:~ # cat /etc/sysctl.conf
# $FreeBSD$
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
dev.re.0.eee_disabled=1
dev.re.1.eee_disabled=1
hw.pci.do_power_suspend=0
dev.re.0.fe=0
dev.re.1.fe=0
root@opnsense:~ #

WAN is down after 20 minutes aprox.

[franco]

Hi grolon,

This topic is about the Intel (igb) driver, not Realtek (re). You won't be able to do a lot with the workarounds described here with a different driver. People sporadicly post here to say they have issues with Realtek chipsets and, unfortunately, the state in BSD is not as good as it could be with ends up in solving the issue by migrating to better network cards.


Cheers,
Franco

[grolon]

Hi and thanks,

Too bad,

I was planning to move from Zentyal 4/5 (ubuntu server 14) to pFsense or OPNsense, my hardware is 100% OK, and Realtek NICs are very popular over here.

Thanks anyway folks,

[franco]

OT: We do have the official Realtek driver since 17.1 in contrast to FreeBSD and pfSense. Overall, it doesn't get much better than this, but that is still sub par compared to Linux.


Cheers,
Franco

[dcol]

I have done a lot of testing with different configurations and it seems that IPS needs a higher end processor processor to function without errors. The lower end Atom processors do not cut it and show high RTT on a gateway when IPS is using the WAN/Gateway interface. Just my observation. There could be other factors involved.

I would like to hear from other users as to what hardware is working with IPS enabled on the WAN Interface. I really want to pin down if this is a performance issue or not.

[Werner Fischer]

Regarding the igb issues: I have no solution yet, but got some hints from different people - I just summarize them here (I have not tested them yet):

• Reports and some resolution hints for problems with igb on pfSense 2.4 (which is based on FreeBSD 11.1):  https://www.administrator.de/wissen/pfsense-2-4-1-port-flapping-packet-loss-354821.html - mentioning "hw.igb.num_queues=1"
• Problems with igb on FreeBSD 11.0: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212413 (The crash seem to be related to a certain type and amount of packets. Removing ALTQ support from kernel fix the issue.) - Some more info on ALTQ can be found here: https://www.freebsdnews.com/2016/04/01/enable-altq-igb-driver-freebsd/
• https://redmine.pfsense.org/issues/7149 (also mentioning "hw.igb.num_queues=1")
• Commit on github/pfSense https://github.com/pfsense/FreeBSD-src/commit/215ddb035593bc4cee275b9dbbf8fc3a7579aee1 - but as I now also got reports from another user that he has seen the issues with PFsense 2.4 (FreeBSD11) and PFsense 2.4.2 (FreeBSD 11.1p4) I don't think that this change in the source of the driver made the situation better for the described problem

I will do further testing within the next days and keep you updated.

[Werner Fischer]

After some further research, I'll try these settings for the upcoming days:

Code: [Select]

root@OPNsense:~ # sysctl hw.igb.num_queues
hw.igb.num_queues: 0
root@OPNsense:~ # sysctl hw.pci.enable_msix
hw.pci.enable_msix: 1
root@OPNsense:~ # sysctl hw.igb.enable_msix
hw.igb.enable_msix: 1
root@OPNsense:~ # echo "Settings for next boots, try to fix nic issues:"
Settings for next boots, try to fix nic issues:
root@OPNsense:~ # cat /boot/loader.conf.local
hw.igb.num_queues=1
hw.pci.enable_msix=0
hw.igb.enable_msix=0

As the issues arise only from time to time, it could again take some days until I can say whether or not these settings could help. I'll keep you updated.

[dcol]

I am also doing research based on different hardware configurations. My tests all have to do with how OPNsense handles IPS and netmap. I used Lan Speed Test Client and Server as my endpoints. With and without internet connectivity.

I discovered one key fact. Noise is a major contributor in my tests. I will post details later on, but the short of it is, if I place a cheap router between the ISP and the OPNsense box running in IPS mode, everything seems to operate flawlessly because the background chatter is gone and netmap is not stressed. I can see this in the traffic graphs. The cheap router is set to only pass traffic for the WAN IP. It is amazing how much other traffic is on the ISP modem port. This is why some people don't see any issues. It all depends on the traffic coming from the ISP and the type of interface. I use a /28 block of static IP's

Here is the results, so far, of the WAN traffic usage.
On my production PFsense box, WAN traffic averages 2-4 Mb/s. Only about 100k/sec is legitimate traffic. On the OPNsense box, behind the small router, the usage averages under 500 b/s. Huge difference

I setup this small router to route and not bridge. This allows only legitimate traffic to pass on to the OPNsense box. Then OPNsense can do it's thing to properly firewall only legitimate traffic. But I am not so sure that high bandwidth operations would work well with a cheap front end router.

wefinet, try to setup a similar configuration just using a small throw away router on the front end and see if this ends the problems. Did for me.

Here is how I setup the small router.
Turned off any firewall settings.
Put the static IP info into the WAN settings and setup the LAN for 192.168.0.1/24
Then used the routers IP (192.168.0.1) as the gateway in the OPNsense box and 192.168.0.100 as the WAN address.Then setup the LAN on OPNsense to any other subnet. Simple setup.
I suppose if the router supports DMZ, that would be another way to do it.
The other advantage is you can tap into the small router to get unfiltered LAN traffic for streaming or large downloads to reduce traffic to the OPNsense box. This would be nice for media streaming devices that do not need to be behind a firewall.

Lets hear some thoughts on this approach.

[dcol]

Ran tests on different systems. All had an SSD drive, 16GB memory, and used a Quad NIC Intel i340-T4.
Fresh install of OPNsense using the same configuration on all systems.
Promiscuous mode made no difference. All readings in Mbps. Isolated the internet with an external router. Used one Workstation outside the router and one inside. Used Lan Speed Test and LST Server on both workstations.
Each system was tested with IPS on and off.

Conclusion is IPS did slow down the firewall on the slower devices. But the i7-7700 had unexpected results being mush slower than expected. The weakest processor, C2758, had the poorest IPS results and the best IPS off results even though it has 8 cores, but the slowest bus speeds @2.4Ghz.

# of cores improved non IPS bandwidth and higher bus speeds improved IPS performance. So I would conclude to choose at least 4 cores and the highest bus speed processor available.

[Werner Fischer]

Thank you dcol for your hints. According to your reports you see different speeds. I think this is another separate topic, as for my situation the interface is somehow completely down.

I my current tests I'm running OPNsense 17.7.8-amd64 using FreeBSD 11.0-RELEASE-p15. Unfortunately limiting hw.igb.num_queues to 1 did not help. About 5 minutes after boot, I got the issues again. Only an ifconfig down/up did help:

Code: [Select]

root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
^C
--- 10.1.102.1 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # arp -a && date
? (10.1.102.1) at (incomplete) on igb9 expired [ethernet]
? (10.1.102.55) at 00:30:18:cd:ec:63 on igb9 permanent [ethernet]
? (192.168.1.1) at 00:30:18:cd:e8:54 on igb0 permanent [ethernet]
? (192.168.1.100) at f0:de:f1:f3:17:88 on igb0 expires in 1151 seconds [ethernet]
Tue Dec  5 09:25:35 UTC 2017
root@OPNsense:~ # ifconfig igb9 down && date
Tue Dec  5 09:26:08 UTC 2017
root@OPNsense:~ # ifconfig igb9 up && date
Tue Dec  5 09:26:18 UTC 2017
root@OPNsense:~ # arp -a && date
OPNsense.test.thomas-krenn.com (192.168.1.1) at 00:30:18:cd:e8:54 on igb0 permanent [ethernet]
? (192.168.1.100) at f0:de:f1:f3:17:88 on igb0 expires in 1168 seconds [ethernet]
Tue Dec  5 09:26:22 UTC 2017
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
64 bytes from 10.1.102.1: icmp_seq=0 ttl=64 time=0.467 ms
64 bytes from 10.1.102.1: icmp_seq=1 ttl=64 time=0.358 ms
64 bytes from 10.1.102.1: icmp_seq=2 ttl=64 time=0.276 ms
64 bytes from 10.1.102.1: icmp_seq=3 ttl=64 time=0.384 ms
^C
--- 10.1.102.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.276/0.371/0.467/0.068 ms
root@OPNsense:~ # arp -a && date
? (10.1.102.1) at 4c:5e:0c:4b:23:30 on igb9 expires in 1198 seconds [ethernet]
OPNsense.test.thomas-krenn.com (10.1.102.55) at 00:30:18:cd:ec:63 on igb9 permanent [ethernet]
OPNsense.test.thomas-krenn.com (192.168.1.1) at 00:30:18:cd:e8:54 on igb0 permanent [ethernet]
? (192.168.1.100) at f0:de:f1:f3:17:88 on igb0 expires in 1193 seconds [ethernet]
Tue Dec  5 09:26:34 UTC 2017
root@OPNsense:~ # freebsd-version -ku
11.0-RELEASE-p15
11.0-RELEASE-p15
root@OPNsense:~ # sysctl hw.igb.num_queues
hw.igb.num_queues: 1
root@OPNsense:~ # sysctl hw.pci.enable_msix
hw.pci.enable_msix: 0
root@OPNsense:~ # sysctl hw.igb.enable_msix
hw.igb.enable_msix: 0
root@OPNsense:~ #

I'll continue to switch to OPNsense 18.1 Beta as described by Franco here: https://forum.opnsense.org/index.php?topic=6257.0

I'll keep you updated.

[Werner Fischer]

I have kept limiting hw.igb.num_queues to 1 and having both hw.pci.enable_msix and hw.igb.enable_msix set 0 and have updated to OPNsense 18.1 Beta (using FreeBSD 11.1).

I did not help. As sure as I have started testing again, the problem occured. Starting a speed test on fast.com on a client led immediately to the problem. Only running "ifconfig igb9 down" and "ifconfig igb9 up" again helped:

Code: [Select]

root@OPNsense:~ # opnsense-update -bkgr 18.1.b -n "snapshots\/beta"
Fetching base-18.1.b-amd64.obsolete: ... done
Fetching base-18.1.b-amd64.txz: .........................................^C
root@OPNsense:~ # ifconfig igb9 down
root@OPNsense:~ # ifconfig igb9 up
root@OPNsense:~ # opnsense-update -bkgr 18.1.b -n "snapshots\/beta"
Fetching base-18.1.b-amd64.obsolete: ... done
Fetching base-18.1.b-amd64.txz: .......................... done
Fetching kernel-dbg-18.1.b-amd64.txz: ................................ done
!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!
! A critical upgrade is in progress. !
! Please do not turn off the system. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Installing kernel-dbg-18.1.b-amd64.txz... done
Installing base-18.1.b-amd64.txz... done
Installing base-18.1.b-amd64.obsolete... done
Please reboot.
root@OPNsense:~ #
root@OPNsense:~ # /usr/local/etc/rc.reboot
>>> Invoking stop script 'beep'
>>> Invoking stop script 'freebsd'
>>> Invoking stop script 'backup'
Cannot 'stop' flowd_aggregate. Set flowd_aggregate_enable to YES in /etc/rc.conf or use 'onestop' instead of 'stop'.
Shutdown NOW!
shutdown: [pid 63573]
root@OPNsense:~ #                                                                               
*** FINAL System shutdown message from root@OPNsense.test.thomas-krenn.com ***
                                                                             

System going down IMMEDIATELY                                                 

                                                                               

System shutdown time has arrived
Connection to 192.168.1.1 closed by remote host.
Connection to 192.168.1.1 closed.
wfischer@tpw:~$ ssh root@192.168.1.1
Password for root@OPNsense.test.thomas-krenn.com:
Last login: Tue Dec  5 09:24:52 2017 from 192.168.1.100
----------------------------------------------
|      Hello, this is OPNsense 17.7          |         @@@@@@@@@@@@@@@
|                                            |        @@@@         @@@@
| Website: https://opnsense.org/        |         @@@\\\   ///@@@
| Handbook: https://docs.opnsense.org/   |       ))))))))   ((((((((
| Forums: https://forum.opnsense.org/  |         @@@///   \\\@@@
| Lists: https://lists.opnsense.org/  |        @@@@         @@@@
| Code: https://github.com/opnsense  |         @@@@@@@@@@@@@@@
----------------------------------------------

  0) Logout                              7) Ping host
  1) Assign interfaces                   8) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Upgrade from console
  6) Reboot system                      13) Restore a backup

Enter an option: 8

root@OPNsense:~ # freebsd-version -ku
11.1-RELEASE-p2
11.1-RELEASE-p2
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
^C
--- 10.1.102.1 ping statistics ---
8 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # arp -a && date
? (10.1.102.1) at 4c:5e:0c:4b:23:30 on igb9 expires in 1168 seconds [ethernet]
OPNsense.test.thomas-krenn.com (10.1.102.55) at 00:30:18:cd:ec:63 on igb9 permanent [ethernet]
OPNsense.test.thomas-krenn.com (192.168.1.1) at 00:30:18:cd:e8:54 on igb0 permanent [ethernet]
? (192.168.1.100) at f0:de:f1:f3:17:88 on igb0 expires in 1152 seconds [ethernet]
Tue Dec  5 10:23:35 UTC 2017
root@OPNsense:~ #
root@OPNsense:~ #
root@OPNsense:~ # clog /var/log/system.log
[...]
Dec  5 10:22:13 OPNsense kernel: aesni0: No AESNI support.
Dec  5 10:22:13 OPNsense kernel:
Dec  5 10:22:13 OPNsense kernel: igb9: link state changed to DOWN
Dec  5 10:22:13 OPNsense sshlockout[15498]: sshlockout/webConfigurator v3.0 starting up
Dec  5 10:22:13 OPNsense configd.py: [499c4346-ad71-4e2b-9e64-ffce20ce3d3c] Linkup stopping igb9
Dec  5 10:22:18 OPNsense kernel: igb9: link state changed to UP
Dec  5 10:22:18 OPNsense configd.py: [6df83b34-21a5-4d42-a53b-f492e8b7193b] Linkup starting igb9
Dec  5 10:22:18 OPNsense opnsense: /usr/local/etc/rc.bootup: Accept router advertisements on interface igb9
Dec  5 10:22:18 OPNsense kernel: igb0: link state changed to DOWN
Dec  5 10:22:18 OPNsense configd.py: [c7a56dbc-d156-4b0a-9022-97f35d436b47] Linkup stopping igb0
Dec  5 10:22:19 OPNsense kernel: pflog0: promiscuous mode enabled
Dec  5 10:22:19 OPNsense kernel: .done.
Dec  5 10:22:19 OPNsense sshd[40530]: Server listening on :: port 22.
Dec  5 10:22:19 OPNsense sshd[40530]: Server listening on 0.0.0.0 port 22.
Dec  5 10:22:19 OPNsense configd.py: [2bb1592c-4e7e-4285-a882-2a110317d983] generate template OPNsense/WebGui
Dec  5 10:22:19 OPNsense kernel: done.
Dec  5 10:22:19 OPNsense configd.py: generate template container OPNsense/WebGui
Dec  5 10:22:19 OPNsense lighttpd[41414]: (log.c.217) server started
Dec  5 10:22:20 OPNsense opnsense: /usr/local/etc/rc.bootup: ROUTING: setting IPv4 default route to 10.1.102.1
Dec  5 10:22:20 OPNsense kernel: done.
Dec  5 10:22:20 OPNsense kernel: done.
Dec  5 10:22:20 OPNsense kernel: done.
Dec  5 10:22:21 OPNsense kernel: done.
Dec  5 10:22:21 OPNsense kernel: done.
Dec  5 10:22:21 OPNsense configd.py: [4ac9229b-5738-4acf-8e67-ba3af24f9232] generate template *
Dec  5 10:22:21 OPNsense kernel: ....done.
Dec  5 10:22:22 OPNsense configd.py: generate template container OPNsense/Auth
Dec  5 10:22:22 OPNsense configd.py: generate template container OPNsense/Captiveportal
Dec  5 10:22:22 OPNsense configd.py: generate template container OPNsense/Cron
Dec  5 10:22:22 OPNsense configd.py: generate template container OPNsense/IDS
Dec  5 10:22:23 OPNsense configd.py: generate template container OPNsense/IPFW
Dec  5 10:22:23 OPNsense kernel: igb0: link state changed to UP
Dec  5 10:22:23 OPNsense configd.py: [2a3a9380-edb6-4a8a-9940-b38c2068244a] Linkup starting igb0
Dec  5 10:22:23 OPNsense configd.py: generate template container OPNsense/Macros
Dec  5 10:22:23 OPNsense configd.py: generate template container OPNsense/Netflow
Dec  5 10:22:24 OPNsense configd.py: generate template container OPNsense/Proxy
Dec  5 10:22:25 OPNsense configd.py: generate template container OPNsense/Sample
Dec  5 10:22:25 OPNsense configd.py: generate template container OPNsense/Sample/sub1
Dec  5 10:22:25 OPNsense configd.py: generate template container OPNsense/Sample/sub2
Dec  5 10:22:25 OPNsense configd.py: generate template container OPNsense/Syslog
Dec  5 10:22:25 OPNsense configd.py: generate template container OPNsense/WebGui
Dec  5 10:22:27 OPNsense configd.py: [58e260da-2e89-4290-a9a1-e985c024ff15] generate template OPNsense/Syslog
Dec  5 10:22:27 OPNsense kernel: done.
Dec  5 10:22:28 OPNsense configd.py: generate template container OPNsense/Syslog
Dec  5 10:22:28 OPNsense kernel: done.
Dec  5 10:22:31 OPNsense configd.py: [831530b7-a519-4d60-b14e-2d35f351ad66] restarting cron
Dec  5 10:22:31 OPNsense sshlockout[15018]: sshlockout/webConfigurator v3.0 starting up
Dec  5 10:22:31 OPNsense kernel: OK
Dec  5 10:22:33 OPNsense kernel:
Dec  5 10:22:54 OPNsense sshd[27160]: Postponed keyboard-interactive for root from 192.168.1.100 port 52728 ssh2 [preauth]
Dec  5 10:22:57 OPNsense opnsense: user 'root' authenticated successfully
Dec  5 10:22:57 OPNsense sshd[27160]: Postponed keyboard-interactive/pam for root from 192.168.1.100 port 52728 ssh2 [preauth]
Dec  5 10:22:57 OPNsense sshd[27160]: Accepted keyboard-interactive/pam for root from 192.168.1.100 port 52728 ssh2
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
^C
--- 10.1.102.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # sysctl hw.igb.num_queues
hw.igb.num_queues: 1
root@OPNsense:~ # sysctl hw.pci.enable_msix
hw.pci.enable_msix: 0
root@OPNsense:~ # sysctl hw.igb.enable_msix
hw.igb.enable_msix: 0
root@OPNsense:~ # cat /boot/loader.conf.local
hw.igb.num_queues=1
hw.pci.enable_msix=0
hw.igb.enable_msix=0
root@OPNsense:~ # rm /boot/loader.conf.local
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
^C
--- 10.1.102.1 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # ifconfig down igb9
ifconfig: interface down does not exist
root@OPNsense:~ # ifconfig igb9 down
root@OPNsense:~ # ifconfig igb9 up
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
64 bytes from 10.1.102.1: icmp_seq=3 ttl=64 time=0.354 ms
64 bytes from 10.1.102.1: icmp_seq=4 ttl=64 time=0.279 ms
64 bytes from 10.1.102.1: icmp_seq=5 ttl=64 time=26.825 ms
64 bytes from 10.1.102.1: icmp_seq=6 ttl=64 time=16.797 ms
^C
--- 10.1.102.1 ping statistics ---
7 packets transmitted, 4 packets received, 42.9% packet loss
round-trip min/avg/max/stddev = 0.279/11.064/26.825/11.317 ms
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
^C
--- 10.1.102.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
^C
--- 10.1.102.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # ifconfig igb9 down
root@OPNsense:~ # ifconfig igb9 up
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
64 bytes from 10.1.102.1: icmp_seq=4 ttl=64 time=0.333 ms
64 bytes from 10.1.102.1: icmp_seq=5 ttl=64 time=0.263 ms
64 bytes from 10.1.102.1: icmp_seq=6 ttl=64 time=0.342 ms
64 bytes from 10.1.102.1: icmp_seq=7 ttl=64 time=0.284 ms
64 bytes from 10.1.102.1: icmp_seq=8 ttl=64 time=0.285 ms
64 bytes from 10.1.102.1: icmp_seq=9 ttl=64 time=0.299 ms
64 bytes from 10.1.102.1: icmp_seq=10 ttl=64 time=0.314 ms
64 bytes from 10.1.102.1: icmp_seq=11 ttl=64 time=0.364 ms
^C
--- 10.1.102.1 ping statistics ---
26 packets transmitted, 8 packets received, 69.2% packet loss
round-trip min/avg/max/stddev = 0.263/0.310/0.364/0.032 ms
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
^C
--- 10.1.102.1 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # ifconfig igb9 down
root@OPNsense:~ # ifconfig igb9 up
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
64 bytes from 10.1.102.1: icmp_seq=3 ttl=64 time=0.415 ms
64 bytes from 10.1.102.1: icmp_seq=4 ttl=64 time=0.258 ms
64 bytes from 10.1.102.1: icmp_seq=5 ttl=64 time=0.258 ms
^C
--- 10.1.102.1 ping statistics ---
6 packets transmitted, 3 packets received, 50.0% packet loss
round-trip min/avg/max/stddev = 0.258/0.310/0.415/0.074 ms
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
^C
--- 10.1.102.1 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # ifconfig igb9 down
root@OPNsense:~ # ifconfig igb9 up
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
64 bytes from 10.1.102.1: icmp_seq=3 ttl=64 time=0.443 ms
64 bytes from 10.1.102.1: icmp_seq=4 ttl=64 time=0.255 ms
64 bytes from 10.1.102.1: icmp_seq=5 ttl=64 time=0.341 ms
64 bytes from 10.1.102.1: icmp_seq=6 ttl=64 time=0.290 ms
64 bytes from 10.1.102.1: icmp_seq=7 ttl=64 time=0.288 ms
64 bytes from 10.1.102.1: icmp_seq=8 ttl=64 time=0.350 ms
64 bytes from 10.1.102.1: icmp_seq=9 ttl=64 time=0.318 ms
64 bytes from 10.1.102.1: icmp_seq=10 ttl=64 time=0.376 ms
64 bytes from 10.1.102.1: icmp_seq=11 ttl=64 time=0.301 ms
64 bytes from 10.1.102.1: icmp_seq=12 ttl=64 time=0.324 ms
64 bytes from 10.1.102.1: icmp_seq=13 ttl=64 time=0.287 ms
64 bytes from 10.1.102.1: icmp_seq=14 ttl=64 time=0.285 ms
64 bytes from 10.1.102.1: icmp_seq=15 ttl=64 time=0.279 ms
64 bytes from 10.1.102.1: icmp_seq=16 ttl=64 time=0.326 ms
64 bytes from 10.1.102.1: icmp_seq=17 ttl=64 time=0.267 ms
64 bytes from 10.1.102.1: icmp_seq=18 ttl=64 time=0.474 ms
64 bytes from 10.1.102.1: icmp_seq=19 ttl=64 time=0.264 ms
64 bytes from 10.1.102.1: icmp_seq=20 ttl=64 time=0.234 ms
64 bytes from 10.1.102.1: icmp_seq=21 ttl=64 time=0.339 ms
64 bytes from 10.1.102.1: icmp_seq=22 ttl=64 time=0.369 ms
64 bytes from 10.1.102.1: icmp_seq=23 ttl=64 time=0.476 ms
64 bytes from 10.1.102.1: icmp_seq=24 ttl=64 time=0.293 ms
64 bytes from 10.1.102.1: icmp_seq=25 ttl=64 time=0.413 ms
64 bytes from 10.1.102.1: icmp_seq=26 ttl=64 time=0.429 ms
64 bytes from 10.1.102.1: icmp_seq=27 ttl=64 time=0.345 ms
64 bytes from 10.1.102.1: icmp_seq=28 ttl=64 time=0.411 ms
64 bytes from 10.1.102.1: icmp_seq=29 ttl=64 time=0.292 ms
64 bytes from 10.1.102.1: icmp_seq=30 ttl=64 time=0.268 ms
64 bytes from 10.1.102.1: icmp_seq=31 ttl=64 time=0.237 ms
64 bytes from 10.1.102.1: icmp_seq=32 ttl=64 time=0.281 ms
64 bytes from 10.1.102.1: icmp_seq=33 ttl=64 time=0.385 ms
64 bytes from 10.1.102.1: icmp_seq=34 ttl=64 time=0.371 ms
64 bytes from 10.1.102.1: icmp_seq=35 ttl=64 time=0.332 ms
64 bytes from 10.1.102.1: icmp_seq=36 ttl=64 time=0.343 ms
64 bytes from 10.1.102.1: icmp_seq=37 ttl=64 time=0.314 ms
64 bytes from 10.1.102.1: icmp_seq=38 ttl=64 time=0.329 ms
64 bytes from 10.1.102.1: icmp_seq=39 ttl=64 time=0.712 ms
64 bytes from 10.1.102.1: icmp_seq=40 ttl=64 time=0.340 ms
64 bytes from 10.1.102.1: icmp_seq=41 ttl=64 time=0.328 ms
64 bytes from 10.1.102.1: icmp_seq=42 ttl=64 time=0.387 ms
64 bytes from 10.1.102.1: icmp_seq=43 ttl=64 time=0.252 ms
64 bytes from 10.1.102.1: icmp_seq=44 ttl=64 time=0.343 ms
64 bytes from 10.1.102.1: icmp_seq=45 ttl=64 time=0.368 ms
64 bytes from 10.1.102.1: icmp_seq=46 ttl=64 time=0.245 ms
64 bytes from 10.1.102.1: icmp_seq=47 ttl=64 time=0.466 ms
64 bytes from 10.1.102.1: icmp_seq=48 ttl=64 time=0.414 ms
64 bytes from 10.1.102.1: icmp_seq=49 ttl=64 time=0.302 ms
64 bytes from 10.1.102.1: icmp_seq=50 ttl=64 time=0.464 ms
64 bytes from 10.1.102.1: icmp_seq=51 ttl=64 time=0.262 ms
64 bytes from 10.1.102.1: icmp_seq=52 ttl=64 time=0.524 ms
64 bytes from 10.1.102.1: icmp_seq=53 ttl=64 time=0.421 ms
64 bytes from 10.1.102.1: icmp_seq=54 ttl=64 time=0.226 ms
^C
--- 10.1.102.1 ping statistics ---
55 packets transmitted, 52 packets received, 5.5% packet loss
round-trip min/avg/max/stddev = 0.226/0.346/0.712/0.087 ms
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
64 bytes from 10.1.102.1: icmp_seq=0 ttl=64 time=0.325 ms
64 bytes from 10.1.102.1: icmp_seq=1 ttl=64 time=0.356 ms
^C
--- 10.1.102.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.325/0.341/0.356/0.015 ms
root@OPNsense:~ #

After that, I have deleted /boot/loader.conf.local (to get the default values after the next boot). I have powered off the OPNsense system, powered it on again and now when I start a speed test on fast.com on a client, I only see on the OPNsense system that the ping times increase - when the speed test is finished the ping times go down again:

Code: [Select]

root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
64 bytes from 10.1.102.1: icmp_seq=0 ttl=64 time=0.422 ms
64 bytes from 10.1.102.1: icmp_seq=1 ttl=64 time=0.319 ms
64 bytes from 10.1.102.1: icmp_seq=2 ttl=64 time=0.471 ms
64 bytes from 10.1.102.1: icmp_seq=3 ttl=64 time=23.658 ms
64 bytes from 10.1.102.1: icmp_seq=4 ttl=64 time=32.818 ms
64 bytes from 10.1.102.1: icmp_seq=5 ttl=64 time=31.154 ms
64 bytes from 10.1.102.1: icmp_seq=6 ttl=64 time=27.961 ms
64 bytes from 10.1.102.1: icmp_seq=7 ttl=64 time=18.703 ms
64 bytes from 10.1.102.1: icmp_seq=8 ttl=64 time=31.381 ms
64 bytes from 10.1.102.1: icmp_seq=9 ttl=64 time=33.733 ms
64 bytes from 10.1.102.1: icmp_seq=10 ttl=64 time=0.243 ms
64 bytes from 10.1.102.1: icmp_seq=11 ttl=64 time=0.357 ms
64 bytes from 10.1.102.1: icmp_seq=12 ttl=64 time=0.290 ms
64 bytes from 10.1.102.1: icmp_seq=13 ttl=64 time=0.246 ms
^C
--- 10.1.102.1 ping statistics ---
14 packets transmitted, 14 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.243/14.411/33.733/14.528 ms
root@OPNsense:~ #

I'll continue with some more tests with a HBJC385F551-63U-B - see http://www.jetwaycomputer.com/JBC385F551.html - and check the 4 i350, the one i211 and the one i219. As the current system has 10x i211 I'm curious how things will run on this other system on the i211 NIC.

I'll keep you updated.

[Werner Fischer]

I have now the HBJC385F551-63U-B up and running (it comes with an Intel Core i5-6300U CPU).

I'm using the following NICs:

• igb0 as LAN (the first port of the quard-port i350 NIC chip of the system)
• igb4 as WAN (the I211 NIC of the system
• I do not use igb1/2/3 (the other three ports of the i350 NIC chip) and I do not use em0 (the i219 NIC chip of the system)

Currently I'm running the default OPNsense 17.7.8-amd64 with FreeBSD 11.0-RELEASE-p15. No issues so far. I'll keep you updated.

[Werner Fischer]

Hi Franco & Team,

as it now turned out the NIC issue was really somehow caused by the power management function of the I211.

Turning EEE off via the driver did not help, as outlined in https://www.thomas-krenn.com/de/wiki/OPNsense_igb_EEE_Funktion_deaktivieren

We now received a BIOS update for the system, where the power management of the LAN ports has been switched off via firmware. Up until now, we did not detect any problems any more.

We will do q&a testing of the new BIOS/UEFI-firmware and provide the firmware once the tests are finished in the Downloads-section of our site: https://www.thomas-krenn.com/de/download.html?product=15417

Thank you all for your help.

PS: In case that you are reading this because you are experiencing issues with FreeBSD 11.0/11.1 based systems with embedded I211 NICs, check with your hardware/firmware vendor and ask for a firmware which has the power management functions deactivated 

Best regards,
Werner