Messages

Another workaround:

I have now also come across this error. It happens immediately after choosing the "GPT/UEFI mode" as seen on https://www.thomas-krenn.com/de/wikiDE/images/6/60/OPNsense-Installation-06-Select-install-mode.png

Attached is a screen-shot of the error (you need to log in into the forum to view attachments).

As you can see, the login prompt is shown after the error message. When you then log in again using the "installer" user and the password "opnsense", I find myself again on the screen to choose the install mode. When I then choose again "GPT/UEFI mode", the installation continues and the system runs fine then.

Does anybody know what exactly causes this error and if this could be fixed somehow (so that no workaround is necessary)?

Danke nochmal für die Info Franco  :)

Ich habe es bei uns im Wiki nun auch dokumentiert:

https://www.thomas-krenn.com/de/wiki/OPNsense_von_Rechner_im_Layer_2_WAN_Netzwerk_ansprechen

Schöne Grüße,
Werner

[Das "reply-to" ist sehr sicher schuld]

Hallo Franco,

ich bin soeben auch auf das Problem gestoßen, dass ich WAN-seitig ein normalen /24 Netz habe (10.1.102.*) und ich nach Erstellung einer entsprechenden Firewall-Regel dennoch keine Antwort auf Pings von anderen Rechnern im 10.1.102.* Netz auf die OPNsense bekommen habe.

Du hattest dazu in diesem Thread geschrieben:
> Moin Uwe,
> Das "reply-to" ist sehr sicher schuld, welches manuell abgeschaltet werden kann für die Ping-erlauben Regel. Funktioniert es dann?

Deine Vermutung kann ich nun hiermit bestätigen. Anbei siehst du den Screenshot wie OPNsense das Echo Reply auf das Default Gateway zurückschickt (anstelle es an die MAC des Senders zu schicken).

Nachdem ich bei der entsprechenden Firewall-Regel nach Klick bei den "Advanced Options" auf "Show/Hide" die Checkbox bei "disable reply-to" aktiviert habe, hat der Ping funktioniert.

Schöne Grüße,
Werner

Hi all,

although I have no C3000 system yet, I'm just curious and would like to know whether OPNsense 18.1 has an updated ix/ixv driver to support the integrated X553 10GBit NICs of the C3000 Denverton CPUs? For CPU infos see https://ark.intel.com/products/codename/63508/Denverton and https://www.intel.com/content/www/us/en/design/products-and-solutions/processors-and-chipsets/denverton/ns/atom-processor-c3000-series.html

FreeNAS 11.1 has included updated drivers: https://redmine.ixsystems.com/issues/26292

FreeBSD 11.1 does not support the X553 NICs, it seems that FreeBSD 11-STABLE has support for them included:
https://lists.freebsd.org/pipermail/freebsd-announce/2017-September/001803.html (search for "553")
https://lists.freebsd.org/pipermail/freebsd-stable/2017-July/087387.html

Best regards,
Werner

Unfortunately, I got now once again the problem :(

With the new BIOS running, I changed the setting "System State after Power Failure" from "Always Off" to "Always On". I have then saved&exited (using the F4 key) and booted OPNsense. After a while, I plugged the power cable, so the system was off. I plugged in power again, and I have noticed during bootup that fsck has been done. After running a few minutes, the network problem was there again:

Code Select Expand


wfischer@tpw:~$ ssh root@192.168.1.1
Password for root@OPNsense.test.thomas-krenn.com:
Last login: Thu Dec 21 08:43:11 2017 from 192.168.1.100
----------------------------------------------
|      Hello, this is OPNsense 17.7          |         @@@@@@@@@@@@@@@
|                                            |        @@@@         @@@@
| Website: https://opnsense.org/        |         @@@\\\   ///@@@
| Handbook: https://docs.opnsense.org/   |       ))))))))   ((((((((
| Forums: https://forum.opnsense.org/  |         @@@///   \\\@@@
| Lists: https://lists.opnsense.org/  |        @@@@         @@@@
| Code: https://github.com/opnsense  |         @@@@@@@@@@@@@@@
----------------------------------------------

  0) Logout                              7) Ping host
  1) Assign interfaces                   8) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Upgrade from console
  6) Reboot system                      13) Restore a backup

Enter an option: 8

root@OPNsense:~ # ifconfig
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,TXCSUM_IPV6>
ether 00:30:18:cd:e8:54
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
inet6 fe80::1:1%igb0 prefixlen 64 scopeid 0x1
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:30:18:cd:e8:55
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
igb2: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:30:18:cd:ef:80
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
igb3: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:30:18:cd:ef:81
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
igb4: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:30:18:cd:ef:82
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
igb5: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:30:18:cd:ef:83
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
igb6: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:30:18:cd:ec:60
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
igb7: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:30:18:cd:ec:61
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
igb8: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:30:18:cd:ec:62
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
igb9: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,TXCSUM_IPV6>
ether 00:30:18:cd:ec:63
inet6 fe80::230:18ff:fecd:ec63%igb9 prefixlen 64 scopeid 0xa
inet 10.1.102.55 netmask 0xffffff00 broadcast 10.1.102.255
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
enc0: flags=0<> metric 0 mtu 1536
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: enc
pflog0: flags=100<PROMISC> metric 0 mtu 33160
groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
groups: pfsync
syncpeer: 0.0.0.0 maxupd: 128 defer: off
root@OPNsense:~ # arp -a
? (10.1.102.1) at 4c:5e:0c:4b:23:30 on igb9 expires in 1011 seconds [ethernet]
OPNsense.test.thomas-krenn.com (10.1.102.55) at 00:30:18:cd:ec:63 on igb9 permanent [ethernet]
OPNsense.test.thomas-krenn.com (192.168.1.1) at 00:30:18:cd:e8:54 on igb0 permanent [ethernet]
? (192.168.1.100) at f0:de:f1:f3:17:88 on igb0 expires in 1088 seconds [ethernet]
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
^C
--- 10.1.102.1 ping statistics ---
10 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # arp -a
? (10.1.102.1) at 4c:5e:0c:4b:23:30 on igb9 expires in 957 seconds [ethernet]
OPNsense.test.thomas-krenn.com (10.1.102.55) at 00:30:18:cd:ec:63 on igb9 permanent [ethernet]
OPNsense.test.thomas-krenn.com (192.168.1.1) at 00:30:18:cd:e8:54 on igb0 permanent [ethernet]
? (192.168.1.100) at f0:de:f1:f3:17:88 on igb0 expires in 1034 seconds [ethernet]
root@OPNsense:~ # date
Thu Dec 21 13:19:23 UTC 2017
root@OPNsense:~ # freebsd-version -ku
11.0-RELEASE-p17
11.0-RELEASE-p17
root@OPNsense:~ # sysctl hw.igb.num_queues
hw.igb.num_queues: 0
root@OPNsense:~ # sysctl hw.pci.enable_msix
hw.pci.enable_msix: 1
root@OPNsense:~ # sysctl hw.igb.enable_msix
hw.igb.enable_msix: 1
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
^C
--- 10.1.102.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # arp -a
? (10.1.102.1) at 4c:5e:0c:4b:23:30 on igb9 expires in 781 seconds [ethernet]
OPNsense.test.thomas-krenn.com (10.1.102.55) at 00:30:18:cd:ec:63 on igb9 permanent [ethernet]
OPNsense.test.thomas-krenn.com (192.168.1.1) at 00:30:18:cd:e8:54 on igb0 permanent [ethernet]
? (192.168.1.100) at f0:de:f1:f3:17:88 on igb0 expires in 858 seconds [ethernet]
root@OPNsense:~ # arp -a && date
? (10.1.102.1) at 4c:5e:0c:4b:23:30 on igb9 expires in 695 seconds [ethernet]
OPNsense.test.thomas-krenn.com (10.1.102.55) at 00:30:18:cd:ec:63 on igb9 permanent [ethernet]
OPNsense.test.thomas-krenn.com (192.168.1.1) at 00:30:18:cd:e8:54 on igb0 permanent [ethernet]
? (192.168.1.100) at f0:de:f1:f3:17:88 on igb0 expires in 772 seconds [ethernet]
Thu Dec 21 13:23:42 UTC 2017
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
^C
--- 10.1.102.1 ping statistics ---
7 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # ifconfig igb9 down
root@OPNsense:~ # ifconfig igb9 up
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
64 bytes from 10.1.102.1: icmp_seq=2 ttl=64 time=0.316 ms
64 bytes from 10.1.102.1: icmp_seq=3 ttl=64 time=0.298 ms
64 bytes from 10.1.102.1: icmp_seq=4 ttl=64 time=0.377 ms
64 bytes from 10.1.102.1: icmp_seq=5 ttl=64 time=0.294 ms
^C
--- 10.1.102.1 ping statistics ---
6 packets transmitted, 4 packets received, 33.3% packet loss
round-trip min/avg/max/stddev = 0.294/0.321/0.377/0.033 ms
root@OPNsense:~ # arp -a && date
? (10.1.102.1) at 4c:5e:0c:4b:23:30 on igb9 expires in 1198 seconds [ethernet]
OPNsense.test.thomas-krenn.com (10.1.102.55) at 00:30:18:cd:ec:63 on igb9 permanent [ethernet]
OPNsense.test.thomas-krenn.com (192.168.1.1) at 00:30:18:cd:e8:54 on igb0 permanent [ethernet]
? (192.168.1.100) at f0:de:f1:f3:17:88 on igb0 expires in 1198 seconds [ethernet]
Thu Dec 21 13:24:22 UTC 2017
root@OPNsense:~ #


Another user of this system (I think he is using pfSense 2.4) switched EEE off via the driver, and at the same time he has set hw.igb.num_queues=1. Up until now, he did not see any issues. I will try this, too. I'll keep you updated.

Thank you for the hint. I have downloaded the tool (although I'm not sure if the tool should be used with I211-AT chips, as the Intel download site does not list the I211-AT as valid product for this download). In the doc file bootutil.txt I have found this hint regarding -WOLD:

Code Select Expand


POWER MANAGEMENT OPTIONS:
-WOLENABLE or -WOLE
  Enables Wake On LAN (WOL) functionality on the selected port.
-WOLDISABLE or -WOLD
  Disables Wake On LAN (WOL) functionality on the selected port.


The I211 data sheet - see https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/i211-ethernet-controller-datasheet.pdf?asset=9567 - lists 10 different power management features in Table 1-9.

I'm not sure how -WOLD really affects those 10 different power management features. So just in case that you as a user are experiencing link down issues, and you are not sure how you could fix it, ask your hardware vendor if there is a firmware which has the power management deactivated.

Hi Franco & Team,

as it now turned out the NIC issue was really somehow caused by the power management function of the I211.

Turning EEE off via the driver did not help, as outlined in https://www.thomas-krenn.com/de/wiki/OPNsense_igb_EEE_Funktion_deaktivieren

We now received a BIOS update for the system, where the power management of the LAN ports has been switched off via firmware. Up until now, we did not detect any problems any more.

We will do q&a testing of the new BIOS/UEFI-firmware and provide the firmware once the tests are finished in the Downloads-section of our site: https://www.thomas-krenn.com/de/download.html?product=15417

Thank you all for your help.

PS: In case that you are reading this because you are experiencing issues with FreeBSD 11.0/11.1 based systems with embedded I211 NICs, check with your hardware/firmware vendor and ask for a firmware which has the power management functions deactivated  ;)

Best regards,
Werner

I have now the HBJC385F551-63U-B up and running (it comes with an Intel Core i5-6300U CPU).

I'm using the following NICs:

• igb0 as LAN (the first port of the quard-port i350 NIC chip of the system)
• igb4 as WAN (the I211 NIC of the system
• I do not use igb1/2/3 (the other three ports of the i350 NIC chip) and I do not use em0 (the i219 NIC chip of the system)

Currently I'm running the default OPNsense 17.7.8-amd64 with FreeBSD 11.0-RELEASE-p15. No issues so far. I'll keep you updated.

I have kept limiting hw.igb.num_queues to 1 and having both hw.pci.enable_msix and hw.igb.enable_msix set 0 and have updated to OPNsense 18.1 Beta (using FreeBSD 11.1).

I did not help. As sure as I have started testing again, the problem occured. Starting a speed test on fast.com on a client led immediately to the problem. Only running "ifconfig igb9 down" and "ifconfig igb9 up" again helped:

Code Select Expand


root@OPNsense:~ # opnsense-update -bkgr 18.1.b -n "snapshots\/beta"
Fetching base-18.1.b-amd64.obsolete: ... done
Fetching base-18.1.b-amd64.txz: .........................................^C
root@OPNsense:~ # ifconfig igb9 down
root@OPNsense:~ # ifconfig igb9 up
root@OPNsense:~ # opnsense-update -bkgr 18.1.b -n "snapshots\/beta"
Fetching base-18.1.b-amd64.obsolete: ... done
Fetching base-18.1.b-amd64.txz: .......................... done
Fetching kernel-dbg-18.1.b-amd64.txz: ................................ done
!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!
! A critical upgrade is in progress. !
! Please do not turn off the system. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Installing kernel-dbg-18.1.b-amd64.txz... done
Installing base-18.1.b-amd64.txz... done
Installing base-18.1.b-amd64.obsolete... done
Please reboot.
root@OPNsense:~ #
root@OPNsense:~ # /usr/local/etc/rc.reboot
>>> Invoking stop script 'beep'
>>> Invoking stop script 'freebsd'
>>> Invoking stop script 'backup'
Cannot 'stop' flowd_aggregate. Set flowd_aggregate_enable to YES in /etc/rc.conf or use 'onestop' instead of 'stop'.
Shutdown NOW!
shutdown: [pid 63573]
root@OPNsense:~ #                                                                               
*** FINAL System shutdown message from root@OPNsense.test.thomas-krenn.com ***
                                                                             

System going down IMMEDIATELY                                                 

                                                                               

System shutdown time has arrived
Connection to 192.168.1.1 closed by remote host.
Connection to 192.168.1.1 closed.
wfischer@tpw:~$ ssh root@192.168.1.1
Password for root@OPNsense.test.thomas-krenn.com:
Last login: Tue Dec  5 09:24:52 2017 from 192.168.1.100
----------------------------------------------
|      Hello, this is OPNsense 17.7          |         @@@@@@@@@@@@@@@
|                                            |        @@@@         @@@@
| Website: https://opnsense.org/        |         @@@\\\   ///@@@
| Handbook: https://docs.opnsense.org/   |       ))))))))   ((((((((
| Forums: https://forum.opnsense.org/  |         @@@///   \\\@@@
| Lists: https://lists.opnsense.org/  |        @@@@         @@@@
| Code: https://github.com/opnsense  |         @@@@@@@@@@@@@@@
----------------------------------------------

  0) Logout                              7) Ping host
  1) Assign interfaces                   8) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Upgrade from console
  6) Reboot system                      13) Restore a backup

Enter an option: 8

root@OPNsense:~ # freebsd-version -ku
11.1-RELEASE-p2
11.1-RELEASE-p2
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
^C
--- 10.1.102.1 ping statistics ---
8 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # arp -a && date
? (10.1.102.1) at 4c:5e:0c:4b:23:30 on igb9 expires in 1168 seconds [ethernet]
OPNsense.test.thomas-krenn.com (10.1.102.55) at 00:30:18:cd:ec:63 on igb9 permanent [ethernet]
OPNsense.test.thomas-krenn.com (192.168.1.1) at 00:30:18:cd:e8:54 on igb0 permanent [ethernet]
? (192.168.1.100) at f0:de:f1:f3:17:88 on igb0 expires in 1152 seconds [ethernet]
Tue Dec  5 10:23:35 UTC 2017
root@OPNsense:~ #
root@OPNsense:~ #
root@OPNsense:~ # clog /var/log/system.log
[...]
Dec  5 10:22:13 OPNsense kernel: aesni0: No AESNI support.
Dec  5 10:22:13 OPNsense kernel:
Dec  5 10:22:13 OPNsense kernel: igb9: link state changed to DOWN
Dec  5 10:22:13 OPNsense sshlockout[15498]: sshlockout/webConfigurator v3.0 starting up
Dec  5 10:22:13 OPNsense configd.py: [499c4346-ad71-4e2b-9e64-ffce20ce3d3c] Linkup stopping igb9
Dec  5 10:22:18 OPNsense kernel: igb9: link state changed to UP
Dec  5 10:22:18 OPNsense configd.py: [6df83b34-21a5-4d42-a53b-f492e8b7193b] Linkup starting igb9
Dec  5 10:22:18 OPNsense opnsense: /usr/local/etc/rc.bootup: Accept router advertisements on interface igb9
Dec  5 10:22:18 OPNsense kernel: igb0: link state changed to DOWN
Dec  5 10:22:18 OPNsense configd.py: [c7a56dbc-d156-4b0a-9022-97f35d436b47] Linkup stopping igb0
Dec  5 10:22:19 OPNsense kernel: pflog0: promiscuous mode enabled
Dec  5 10:22:19 OPNsense kernel: .done.
Dec  5 10:22:19 OPNsense sshd[40530]: Server listening on :: port 22.
Dec  5 10:22:19 OPNsense sshd[40530]: Server listening on 0.0.0.0 port 22.
Dec  5 10:22:19 OPNsense configd.py: [2bb1592c-4e7e-4285-a882-2a110317d983] generate template OPNsense/WebGui
Dec  5 10:22:19 OPNsense kernel: done.
Dec  5 10:22:19 OPNsense configd.py: generate template container OPNsense/WebGui
Dec  5 10:22:19 OPNsense lighttpd[41414]: (log.c.217) server started
Dec  5 10:22:20 OPNsense opnsense: /usr/local/etc/rc.bootup: ROUTING: setting IPv4 default route to 10.1.102.1
Dec  5 10:22:20 OPNsense kernel: done.
Dec  5 10:22:20 OPNsense kernel: done.
Dec  5 10:22:20 OPNsense kernel: done.
Dec  5 10:22:21 OPNsense kernel: done.
Dec  5 10:22:21 OPNsense kernel: done.
Dec  5 10:22:21 OPNsense configd.py: [4ac9229b-5738-4acf-8e67-ba3af24f9232] generate template *
Dec  5 10:22:21 OPNsense kernel: ....done.
Dec  5 10:22:22 OPNsense configd.py: generate template container OPNsense/Auth
Dec  5 10:22:22 OPNsense configd.py: generate template container OPNsense/Captiveportal
Dec  5 10:22:22 OPNsense configd.py: generate template container OPNsense/Cron
Dec  5 10:22:22 OPNsense configd.py: generate template container OPNsense/IDS
Dec  5 10:22:23 OPNsense configd.py: generate template container OPNsense/IPFW
Dec  5 10:22:23 OPNsense kernel: igb0: link state changed to UP
Dec  5 10:22:23 OPNsense configd.py: [2a3a9380-edb6-4a8a-9940-b38c2068244a] Linkup starting igb0
Dec  5 10:22:23 OPNsense configd.py: generate template container OPNsense/Macros
Dec  5 10:22:23 OPNsense configd.py: generate template container OPNsense/Netflow
Dec  5 10:22:24 OPNsense configd.py: generate template container OPNsense/Proxy
Dec  5 10:22:25 OPNsense configd.py: generate template container OPNsense/Sample
Dec  5 10:22:25 OPNsense configd.py: generate template container OPNsense/Sample/sub1
Dec  5 10:22:25 OPNsense configd.py: generate template container OPNsense/Sample/sub2
Dec  5 10:22:25 OPNsense configd.py: generate template container OPNsense/Syslog
Dec  5 10:22:25 OPNsense configd.py: generate template container OPNsense/WebGui
Dec  5 10:22:27 OPNsense configd.py: [58e260da-2e89-4290-a9a1-e985c024ff15] generate template OPNsense/Syslog
Dec  5 10:22:27 OPNsense kernel: done.
Dec  5 10:22:28 OPNsense configd.py: generate template container OPNsense/Syslog
Dec  5 10:22:28 OPNsense kernel: done.
Dec  5 10:22:31 OPNsense configd.py: [831530b7-a519-4d60-b14e-2d35f351ad66] restarting cron
Dec  5 10:22:31 OPNsense sshlockout[15018]: sshlockout/webConfigurator v3.0 starting up
Dec  5 10:22:31 OPNsense kernel: OK
Dec  5 10:22:33 OPNsense kernel:
Dec  5 10:22:54 OPNsense sshd[27160]: Postponed keyboard-interactive for root from 192.168.1.100 port 52728 ssh2 [preauth]
Dec  5 10:22:57 OPNsense opnsense: user 'root' authenticated successfully
Dec  5 10:22:57 OPNsense sshd[27160]: Postponed keyboard-interactive/pam for root from 192.168.1.100 port 52728 ssh2 [preauth]
Dec  5 10:22:57 OPNsense sshd[27160]: Accepted keyboard-interactive/pam for root from 192.168.1.100 port 52728 ssh2
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
^C
--- 10.1.102.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # sysctl hw.igb.num_queues
hw.igb.num_queues: 1
root@OPNsense:~ # sysctl hw.pci.enable_msix
hw.pci.enable_msix: 0
root@OPNsense:~ # sysctl hw.igb.enable_msix
hw.igb.enable_msix: 0
root@OPNsense:~ # cat /boot/loader.conf.local
hw.igb.num_queues=1
hw.pci.enable_msix=0
hw.igb.enable_msix=0
root@OPNsense:~ # rm /boot/loader.conf.local
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
^C
--- 10.1.102.1 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # ifconfig down igb9
ifconfig: interface down does not exist
root@OPNsense:~ # ifconfig igb9 down
root@OPNsense:~ # ifconfig igb9 up
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
64 bytes from 10.1.102.1: icmp_seq=3 ttl=64 time=0.354 ms
64 bytes from 10.1.102.1: icmp_seq=4 ttl=64 time=0.279 ms
64 bytes from 10.1.102.1: icmp_seq=5 ttl=64 time=26.825 ms
64 bytes from 10.1.102.1: icmp_seq=6 ttl=64 time=16.797 ms
^C
--- 10.1.102.1 ping statistics ---
7 packets transmitted, 4 packets received, 42.9% packet loss
round-trip min/avg/max/stddev = 0.279/11.064/26.825/11.317 ms
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
^C
--- 10.1.102.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
^C
--- 10.1.102.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # ifconfig igb9 down
root@OPNsense:~ # ifconfig igb9 up
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
64 bytes from 10.1.102.1: icmp_seq=4 ttl=64 time=0.333 ms
64 bytes from 10.1.102.1: icmp_seq=5 ttl=64 time=0.263 ms
64 bytes from 10.1.102.1: icmp_seq=6 ttl=64 time=0.342 ms
64 bytes from 10.1.102.1: icmp_seq=7 ttl=64 time=0.284 ms
64 bytes from 10.1.102.1: icmp_seq=8 ttl=64 time=0.285 ms
64 bytes from 10.1.102.1: icmp_seq=9 ttl=64 time=0.299 ms
64 bytes from 10.1.102.1: icmp_seq=10 ttl=64 time=0.314 ms
64 bytes from 10.1.102.1: icmp_seq=11 ttl=64 time=0.364 ms
^C
--- 10.1.102.1 ping statistics ---
26 packets transmitted, 8 packets received, 69.2% packet loss
round-trip min/avg/max/stddev = 0.263/0.310/0.364/0.032 ms
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
^C
--- 10.1.102.1 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # ifconfig igb9 down
root@OPNsense:~ # ifconfig igb9 up
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
64 bytes from 10.1.102.1: icmp_seq=3 ttl=64 time=0.415 ms
64 bytes from 10.1.102.1: icmp_seq=4 ttl=64 time=0.258 ms
64 bytes from 10.1.102.1: icmp_seq=5 ttl=64 time=0.258 ms
^C
--- 10.1.102.1 ping statistics ---
6 packets transmitted, 3 packets received, 50.0% packet loss
round-trip min/avg/max/stddev = 0.258/0.310/0.415/0.074 ms
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
^C
--- 10.1.102.1 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # ifconfig igb9 down
root@OPNsense:~ # ifconfig igb9 up
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
64 bytes from 10.1.102.1: icmp_seq=3 ttl=64 time=0.443 ms
64 bytes from 10.1.102.1: icmp_seq=4 ttl=64 time=0.255 ms
64 bytes from 10.1.102.1: icmp_seq=5 ttl=64 time=0.341 ms
64 bytes from 10.1.102.1: icmp_seq=6 ttl=64 time=0.290 ms
64 bytes from 10.1.102.1: icmp_seq=7 ttl=64 time=0.288 ms
64 bytes from 10.1.102.1: icmp_seq=8 ttl=64 time=0.350 ms
64 bytes from 10.1.102.1: icmp_seq=9 ttl=64 time=0.318 ms
64 bytes from 10.1.102.1: icmp_seq=10 ttl=64 time=0.376 ms
64 bytes from 10.1.102.1: icmp_seq=11 ttl=64 time=0.301 ms
64 bytes from 10.1.102.1: icmp_seq=12 ttl=64 time=0.324 ms
64 bytes from 10.1.102.1: icmp_seq=13 ttl=64 time=0.287 ms
64 bytes from 10.1.102.1: icmp_seq=14 ttl=64 time=0.285 ms
64 bytes from 10.1.102.1: icmp_seq=15 ttl=64 time=0.279 ms
64 bytes from 10.1.102.1: icmp_seq=16 ttl=64 time=0.326 ms
64 bytes from 10.1.102.1: icmp_seq=17 ttl=64 time=0.267 ms
64 bytes from 10.1.102.1: icmp_seq=18 ttl=64 time=0.474 ms
64 bytes from 10.1.102.1: icmp_seq=19 ttl=64 time=0.264 ms
64 bytes from 10.1.102.1: icmp_seq=20 ttl=64 time=0.234 ms
64 bytes from 10.1.102.1: icmp_seq=21 ttl=64 time=0.339 ms
64 bytes from 10.1.102.1: icmp_seq=22 ttl=64 time=0.369 ms
64 bytes from 10.1.102.1: icmp_seq=23 ttl=64 time=0.476 ms
64 bytes from 10.1.102.1: icmp_seq=24 ttl=64 time=0.293 ms
64 bytes from 10.1.102.1: icmp_seq=25 ttl=64 time=0.413 ms
64 bytes from 10.1.102.1: icmp_seq=26 ttl=64 time=0.429 ms
64 bytes from 10.1.102.1: icmp_seq=27 ttl=64 time=0.345 ms
64 bytes from 10.1.102.1: icmp_seq=28 ttl=64 time=0.411 ms
64 bytes from 10.1.102.1: icmp_seq=29 ttl=64 time=0.292 ms
64 bytes from 10.1.102.1: icmp_seq=30 ttl=64 time=0.268 ms
64 bytes from 10.1.102.1: icmp_seq=31 ttl=64 time=0.237 ms
64 bytes from 10.1.102.1: icmp_seq=32 ttl=64 time=0.281 ms
64 bytes from 10.1.102.1: icmp_seq=33 ttl=64 time=0.385 ms
64 bytes from 10.1.102.1: icmp_seq=34 ttl=64 time=0.371 ms
64 bytes from 10.1.102.1: icmp_seq=35 ttl=64 time=0.332 ms
64 bytes from 10.1.102.1: icmp_seq=36 ttl=64 time=0.343 ms
64 bytes from 10.1.102.1: icmp_seq=37 ttl=64 time=0.314 ms
64 bytes from 10.1.102.1: icmp_seq=38 ttl=64 time=0.329 ms
64 bytes from 10.1.102.1: icmp_seq=39 ttl=64 time=0.712 ms
64 bytes from 10.1.102.1: icmp_seq=40 ttl=64 time=0.340 ms
64 bytes from 10.1.102.1: icmp_seq=41 ttl=64 time=0.328 ms
64 bytes from 10.1.102.1: icmp_seq=42 ttl=64 time=0.387 ms
64 bytes from 10.1.102.1: icmp_seq=43 ttl=64 time=0.252 ms
64 bytes from 10.1.102.1: icmp_seq=44 ttl=64 time=0.343 ms
64 bytes from 10.1.102.1: icmp_seq=45 ttl=64 time=0.368 ms
64 bytes from 10.1.102.1: icmp_seq=46 ttl=64 time=0.245 ms
64 bytes from 10.1.102.1: icmp_seq=47 ttl=64 time=0.466 ms
64 bytes from 10.1.102.1: icmp_seq=48 ttl=64 time=0.414 ms
64 bytes from 10.1.102.1: icmp_seq=49 ttl=64 time=0.302 ms
64 bytes from 10.1.102.1: icmp_seq=50 ttl=64 time=0.464 ms
64 bytes from 10.1.102.1: icmp_seq=51 ttl=64 time=0.262 ms
64 bytes from 10.1.102.1: icmp_seq=52 ttl=64 time=0.524 ms
64 bytes from 10.1.102.1: icmp_seq=53 ttl=64 time=0.421 ms
64 bytes from 10.1.102.1: icmp_seq=54 ttl=64 time=0.226 ms
^C
--- 10.1.102.1 ping statistics ---
55 packets transmitted, 52 packets received, 5.5% packet loss
round-trip min/avg/max/stddev = 0.226/0.346/0.712/0.087 ms
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
64 bytes from 10.1.102.1: icmp_seq=0 ttl=64 time=0.325 ms
64 bytes from 10.1.102.1: icmp_seq=1 ttl=64 time=0.356 ms
^C
--- 10.1.102.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.325/0.341/0.356/0.015 ms
root@OPNsense:~ #


After that, I have deleted /boot/loader.conf.local (to get the default values after the next boot). I have powered off the OPNsense system, powered it on again and now when I start a speed test on fast.com on a client, I only see on the OPNsense system that the ping times increase - when the speed test is finished the ping times go down again:

Code Select Expand


root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
64 bytes from 10.1.102.1: icmp_seq=0 ttl=64 time=0.422 ms
64 bytes from 10.1.102.1: icmp_seq=1 ttl=64 time=0.319 ms
64 bytes from 10.1.102.1: icmp_seq=2 ttl=64 time=0.471 ms
64 bytes from 10.1.102.1: icmp_seq=3 ttl=64 time=23.658 ms
64 bytes from 10.1.102.1: icmp_seq=4 ttl=64 time=32.818 ms
64 bytes from 10.1.102.1: icmp_seq=5 ttl=64 time=31.154 ms
64 bytes from 10.1.102.1: icmp_seq=6 ttl=64 time=27.961 ms
64 bytes from 10.1.102.1: icmp_seq=7 ttl=64 time=18.703 ms
64 bytes from 10.1.102.1: icmp_seq=8 ttl=64 time=31.381 ms
64 bytes from 10.1.102.1: icmp_seq=9 ttl=64 time=33.733 ms
64 bytes from 10.1.102.1: icmp_seq=10 ttl=64 time=0.243 ms
64 bytes from 10.1.102.1: icmp_seq=11 ttl=64 time=0.357 ms
64 bytes from 10.1.102.1: icmp_seq=12 ttl=64 time=0.290 ms
64 bytes from 10.1.102.1: icmp_seq=13 ttl=64 time=0.246 ms
^C
--- 10.1.102.1 ping statistics ---
14 packets transmitted, 14 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.243/14.411/33.733/14.528 ms
root@OPNsense:~ #


I'll continue with some more tests with a HBJC385F551-63U-B - see http://www.jetwaycomputer.com/JBC385F551.html - and check the 4 i350, the one i211 and the one i219. As the current system has 10x i211 I'm curious how things will run on this other system on the i211 NIC.

I'll keep you updated.

Thank you dcol for your hints. According to your reports you see different speeds. I think this is another separate topic, as for my situation the interface is somehow completely down.

I my current tests I'm running OPNsense 17.7.8-amd64 using FreeBSD 11.0-RELEASE-p15. Unfortunately limiting hw.igb.num_queues to 1 did not help. About 5 minutes after boot, I got the issues again. Only an ifconfig down/up did help:

Code Select Expand


root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
^C
--- 10.1.102.1 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # arp -a && date
? (10.1.102.1) at (incomplete) on igb9 expired [ethernet]
? (10.1.102.55) at 00:30:18:cd:ec:63 on igb9 permanent [ethernet]
? (192.168.1.1) at 00:30:18:cd:e8:54 on igb0 permanent [ethernet]
? (192.168.1.100) at f0:de:f1:f3:17:88 on igb0 expires in 1151 seconds [ethernet]
Tue Dec  5 09:25:35 UTC 2017
root@OPNsense:~ # ifconfig igb9 down && date
Tue Dec  5 09:26:08 UTC 2017
root@OPNsense:~ # ifconfig igb9 up && date
Tue Dec  5 09:26:18 UTC 2017
root@OPNsense:~ # arp -a && date
OPNsense.test.thomas-krenn.com (192.168.1.1) at 00:30:18:cd:e8:54 on igb0 permanent [ethernet]
? (192.168.1.100) at f0:de:f1:f3:17:88 on igb0 expires in 1168 seconds [ethernet]
Tue Dec  5 09:26:22 UTC 2017
root@OPNsense:~ # ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
64 bytes from 10.1.102.1: icmp_seq=0 ttl=64 time=0.467 ms
64 bytes from 10.1.102.1: icmp_seq=1 ttl=64 time=0.358 ms
64 bytes from 10.1.102.1: icmp_seq=2 ttl=64 time=0.276 ms
64 bytes from 10.1.102.1: icmp_seq=3 ttl=64 time=0.384 ms
^C
--- 10.1.102.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.276/0.371/0.467/0.068 ms
root@OPNsense:~ # arp -a && date
? (10.1.102.1) at 4c:5e:0c:4b:23:30 on igb9 expires in 1198 seconds [ethernet]
OPNsense.test.thomas-krenn.com (10.1.102.55) at 00:30:18:cd:ec:63 on igb9 permanent [ethernet]
OPNsense.test.thomas-krenn.com (192.168.1.1) at 00:30:18:cd:e8:54 on igb0 permanent [ethernet]
? (192.168.1.100) at f0:de:f1:f3:17:88 on igb0 expires in 1193 seconds [ethernet]
Tue Dec  5 09:26:34 UTC 2017
root@OPNsense:~ # freebsd-version -ku
11.0-RELEASE-p15
11.0-RELEASE-p15
root@OPNsense:~ # sysctl hw.igb.num_queues
hw.igb.num_queues: 1
root@OPNsense:~ # sysctl hw.pci.enable_msix
hw.pci.enable_msix: 0
root@OPNsense:~ # sysctl hw.igb.enable_msix
hw.igb.enable_msix: 0
root@OPNsense:~ #


I'll continue to switch to OPNsense 18.1 Beta as described by Franco here: https://forum.opnsense.org/index.php?topic=6257.0

I'll keep you updated.

After some further research, I'll try these settings for the upcoming days:

Code Select Expand


root@OPNsense:~ # sysctl hw.igb.num_queues
hw.igb.num_queues: 0
root@OPNsense:~ # sysctl hw.pci.enable_msix
hw.pci.enable_msix: 1
root@OPNsense:~ # sysctl hw.igb.enable_msix
hw.igb.enable_msix: 1
root@OPNsense:~ # echo "Settings for next boots, try to fix nic issues:"
Settings for next boots, try to fix nic issues:
root@OPNsense:~ # cat /boot/loader.conf.local
hw.igb.num_queues=1
hw.pci.enable_msix=0
hw.igb.enable_msix=0


As the issues arise only from time to time, it could again take some days until I can say whether or not these settings could help. I'll keep you updated.

Regarding the igb issues: I have no solution yet, but got some hints from different people - I just summarize them here (I have not tested them yet):

• Reports and some resolution hints for problems with igb on pfSense 2.4 (which is based on FreeBSD 11.1):  https://www.administrator.de/wissen/pfsense-2-4-1-port-flapping-packet-loss-354821.html - mentioning "hw.igb.num_queues=1"
• Problems with igb on FreeBSD 11.0: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212413 (The crash seem to be related to a certain type and amount of packets. Removing ALTQ support from kernel fix the issue.) - Some more info on ALTQ can be found here: https://www.freebsdnews.com/2016/04/01/enable-altq-igb-driver-freebsd/
• https://redmine.pfsense.org/issues/7149 (also mentioning "hw.igb.num_queues=1")
• Commit on github/pfSense https://github.com/pfsense/FreeBSD-src/commit/215ddb035593bc4cee275b9dbbf8fc3a7579aee1 - but as I now also got reports from another user that he has seen the issues with PFsense 2.4 (FreeBSD11) and PFsense 2.4.2 (FreeBSD 11.1p4) I don't think that this change in the source of the driver made the situation better for the described problem

I will do further testing within the next days and keep you updated.

I have now installed a FreeBSD 11.1 Kernel (got instruction for that from Franco via PM):

Code Select Expand


root@OPNsense:~ # freebsd-version -k
11.1-RELEASE-p1
root@OPNsense:~ # freebsd-version -u
11.0-RELEASE-p12
root@OPNsense:~ #


Also with FreeBSD 11.1 the NICs show "GIANT-LOCKED" in the dmesg output (as they do also with FreeBSD 11.0, but not with pfSense 2.3/FreeBSD 10.3 (which does not have the issue)):

Code Select Expand


...
pcib18: <PCI-PCI bridge> irq 18 at device 3.0 on pci12
pcib18: [GIANT-LOCKED]
pci15: <PCI bus> on pcib18
igb8: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x6000-0x601f mem 0xd0700000-0xd071ffff,0xd0720000-0xd0723fff irq 18 at device 0.0 on pci15
igb8: Using MSIX interrupts with 3 vectors
igb8: Ethernet address: 00:30:18:cd:ec:62
igb8: Bound queue 0 to cpu 0
igb8: Bound queue 1 to cpu 1
igb8: netmap queues/slots: TX 2/1024, RX 2/1024
pcib19: <PCI-PCI bridge> irq 19 at device 4.0 on pci12
pcib19: [GIANT-LOCKED]
pci16: <PCI bus> on pcib19
igb9: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x5000-0x501f mem 0xd0600000-0xd061ffff,0xd0620000-0xd0623fff irq 19 at device 0.0 on pci16
igb9: Using MSIX interrupts with 3 vectors
igb9: Ethernet address: 00:30:18:cd:ec:63
igb9: Bound queue 0 to cpu 2
igb9: Bound queue 1 to cpu 3
igb9: netmap queues/slots: TX 2/1024, RX 2/1024
...


But anyway I will stay with this setup for the next days and will watch out whether the problem shows up again or not. I'll keep you updated.

Since my last posting on Aug, 31st, I've been running pfSense 2.3.4 on the system, without having any issues. So I'm rather sure, that my problem has to do with FreeBSD 11.0 vs. FreeBSD 10.3. I have attached a ZIP with the logs of pfSense 2.3.4.

I have searched for differences and I have found:

• sysctl -a shows for pfSense 2.3.4 the item "hw.igb.buf_ring_size: 4096". This item is missing on pfSense 2.4-RC. Could this be causing the issue? (UPDATE: on OPNsense 17.7.2 "hw.igb.buf_ring_size: 4096" is present - so I think this is not the root cause for my problem)
• dmesg shows on pfSense 2.4-RC pci entries with "[GIANT-LOCKED]". pfSense 2.3.4 does not list this kind of items, see the code below. Could this be causing the issue? (UPDATE: also OPNsense 17.7.2 shows "[GIANT-LOCKED]")

Here is the code for the mentioned dmesg part:

Code Select Expand


pfSense 2.3.4 dmesg:
  ...
  pcib19: <PCI-PCI bridge> irq 19 at device 4.0 on pci15
  pci19: <PCI bus> on pcib19
  igb9: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x5000-0x501f mem 0xd0600000-0xd061ffff,0xd0620000-0xd0623fff irq 19 at device 0.0 on pci19
  ...

pfSense 2.4.0-RC dmesg:
  ...
  pcib19: <PCI-PCI bridge> irq 19 at device 4.0 on pci12
  pcib19: [GIANT-LOCKED]
  pci16: <PCI bus> on pcib19
  igb9: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x5000-0x501f mem 0xd0600000-0xd061ffff,0xd0620000-0xd0623fff irq 19 at device 0.0 on pci16
  ...


Tomorrow, I will switch back to OPNsense and I will install the FreeBSD 11.1 kernel. I'll keep you updated.

[pfSense 2.4 RC]

I'm currently still having pfSense 2.4 RC on the system, and today in the morning I got exactly the same problem like I get with OPNsense 17.1/17.7: after some time of operation (about 1-3 minutes after boot) I run into the problem. EEE was _not_ deactivated, but as I have seen the issue on OPNsense with EEE having deactivated, too, I won't do currently any tests with EEE deactivated with pfSense 2.4 RC.

Attached you find a full bunch of logs and command output (in case it helps us to analyze the root cause of the issue).

One thing I want to show you right here - as you can see an "ifconfig igb9 down" followed by an "ifconfig igb9 up" fixes the issue:

Code Select Expand


[2.4.0-RC][admin@pfSense24.test.thomas-krenn.com]/root: ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
^C
--- 10.1.102.1 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
[2.4.0-RC][admin@pfSense24.test.thomas-krenn.com]/root: arp -a
? (10.1.102.1) at (incomplete) on igb9 expired [ethernet]
? (10.1.102.55) at 00:30:18:cd:ec:63 on igb9 permanent [ethernet]
pfSense24.test.thomas-krenn.com (192.168.1.1) at 00:30:18:cd:e8:54 on igb0 permanent [ethernet]
? (192.168.1.100) at f0:de:f1:f3:17:88 on igb0 expires in 480 seconds [ethernet]
[2.4.0-RC][admin@pfSense24.test.thomas-krenn.com]/root: date
Thu Aug 31 09:55:44 CEST 2017
[2.4.0-RC][admin@pfSense24.test.thomas-krenn.com]/root: ifconfig igb9 down
[2.4.0-RC][admin@pfSense24.test.thomas-krenn.com]/root: date
Thu Aug 31 09:56:26 CEST 2017
[2.4.0-RC][admin@pfSense24.test.thomas-krenn.com]/root: ifconfig igb9 up
[2.4.0-RC][admin@pfSense24.test.thomas-krenn.com]/root: date
Thu Aug 31 09:56:34 CEST 2017
[2.4.0-RC][admin@pfSense24.test.thomas-krenn.com]/root: date
Thu Aug 31 09:56:41 CEST 2017
[2.4.0-RC][admin@pfSense24.test.thomas-krenn.com]/root: arp -a
? (10.1.102.1) at 4c:5e:0c:4b:23:30 on igb9 expires in 1196 seconds [ethernet]
? (10.1.102.55) at 00:30:18:cd:ec:63 on igb9 permanent [ethernet]
pfSense24.test.thomas-krenn.com (192.168.1.1) at 00:30:18:cd:e8:54 on igb0 permanent [ethernet]
? (192.168.1.100) at f0:de:f1:f3:17:88 on igb0 expires in 1199 seconds [ethernet]
[2.4.0-RC][admin@pfSense24.test.thomas-krenn.com]/root: date
Thu Aug 31 09:56:47 CEST 2017
[2.4.0-RC][admin@pfSense24.test.thomas-krenn.com]/root: ping 10.1.102.1
PING 10.1.102.1 (10.1.102.1): 56 data bytes
64 bytes from 10.1.102.1: icmp_seq=0 ttl=64 time=0.340 ms
64 bytes from 10.1.102.1: icmp_seq=1 ttl=64 time=0.246 ms
64 bytes from 10.1.102.1: icmp_seq=2 ttl=64 time=0.238 ms
64 bytes from 10.1.102.1: icmp_seq=3 ttl=64 time=0.225 ms
^C
--- 10.1.102.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.225/0.262/0.340/0.046 ms
[2.4.0-RC][admin@pfSense24.test.thomas-krenn.com]/root:


Before I continue to test with the upcoming 17.7.1 and the preliminary Kernel, I think I'll test with pfSense 2.3 (which is based on FreeBSD 10.3) and watch out if it really works rock-solid (to have more evidence that FreeBSD 11.0 is causing the issue, while FreeBSD 10.3 brings no issues).

When I see that pfSense 2.3 indeed runs solid like expected, I'll grab all the output for analysis (especially "sysctl -a") and will compare it to the outputs that I have attached in this post. Maybe we find some settings, which differ, that then could maybe be the reason for this problem.

I will keep you updated  ;)