Messages

Thank you for the online update. I will check it today/tomorrow. With the first beta I've run into an issue with a LTE modem connection. I have tried https://www.thomas-krenn.com/de/wiki/OPNsense_LTE_Verbindung with a Quectel Modem, and got a crash with an automated reboot afterwards. I will re-check this after applying the online update.

In case I still encounter the issue with the LTE connection: should I report it here or should I open an issue on GitHub?

Best Regards,
Werner

Just for reference - this issue has been a bug, which will be fixed with OPNsense 20.1.3:

https://github.com/opnsense/core/issues/3961#issuecomment-597536162

I have created the pull request: https://github.com/opnsense/docs/pull/198

Thank you, I'll update the wiki article https://www.thomas-krenn.com/en/wiki/OPNsense_HA_Cluster_configuration.

I'll check whether I can prepare a pull request for the OPNsense documentation. The example configuration files should be updated there, too (as the current configuration file of the backup node does not have pfsync enabled).

Hi all,

when you are setting up a firewall cluster, the documentation https://docs.opnsense.org/manual/how-tos/carp.html#setup-ha-sync-xmlrpc-and-pfsync currently says:

• First we should enable pfSync using our dedicated interface using the master firewall. Go to System ‣ High Availability ‣ Settings, enable pfSync and select the interface used for pfSync.

Also a diff of the sample configuration shows that pfsync is only enabled on node 1 (master):

• https://docs.opnsense.org/_downloads/64fce6febca41b922ab9906c47078aa9/Carp_example_master.xml
• https://docs.opnsense.org/_downloads/5b64c2fa6e30519189630e5dd22f0e58/Carp_example_backup.xml

My question:

• What happens e.g. when you are doing a firmware update and you switch the master role from node 1 to node 2?
• I assume that pf states are not synchronized again from node 2 -> node 1 when node 1 comes back up.
• I _think_ that pfsync should be enabled on node 2, too. pfsense suggests it in this way too, according to https://docs.netgate.com/pfsense/en/latest/book/highavailability/pfsync-overview.html#pfsync-overview "When pfsync is in use, pfsync settings must be enabled on all nodes participating in state synchronization, including secondary nodes, or it will not function properly."

Should we update the OPNsense documentation?

Best regards,
Werner

Hi all,

I have some feedback regarding CARP:

• a) pfsync: I think in https://docs.opnsense.org/manual/how-tos/carp.html#setup-ha-sync-xmlrpc-and-pfsync it should be stated that on the backup firewall the "Synchronize States" option should be set, too.
• b) Enter Persistent CARP Maintenance Mode: when clicking this on firwall 1 this sets currently net.inet.carp.demotion=240 and leaves advskew as long there is no reboot. After a reboot of firewall 1, net.inet.carp.demotion=0, but advskew is set to 254 - because of https://github.com/opnsense/core/blob/master/src/etc/inc/interfaces.inc#L1713. When following the steps described at https://docs.opnsense.org/manual/how-tos/carp.html#example-updating-a-carp-ha-cluster the advskew setting is still set to 254 on firewall 1 even after clicking "Leave Persistent CARP Maintenance Mode". When testing a WAN outage (unplug igb1) afterwards, only for the WAN IP, the other firewall gets MASTER, leaving LAN (igb0) as BACKUP there. So the Internet connectivity gets lost for clients. Only rebooting the firewall 1 or manually setting advskew back to 0 solves the issue. I'm not sure what would be the best way to fix this behavior. Any ideas?

Steps to reproduce issue b):

• Build an OPNsense HA cluster with two nodes, firewall 1 as MASTER and firewall 2 as BACKUP
• Click "Enter Persistent CARP Maintenance Mode" on firewall 1. The sysctl "net.inet.carp.demotion" will be set to 240. advskew is still 0 for all configured CARP interfaces.
• Do a reboot of firewall 1.
• After the reboot, on firewall 1 "net.inet.carp.demotion" is now 0 (not 240), but advskew for all CARP interfaces is set to 254 (query by "ifconfig | grep carp"). So advskew is set to 254, but the web interface shows still values of 0 in "Firewall -> Virtual IPs -> Settings".
• Clicking "Leave Persistent CARP Maintenance Mode" on firewall 1 does _not_ switch back the CARP IPs to firewall 1. firewall 2 is still MASTER, although I would expect that now there should be a switch-back to firewall 1 - according to the doc https://docs.opnsense.org/manual/how-tos/carp.html#example-updating-a-carp-ha-cluster
• Only after another reboot of firewall 1, advskew is again set to 0. But in my opinion this additional reboot of firewall 1 is unecessary when updating an OPNsense firewall cluster.

Best regards,
Werner

The status site "status_interfaces.php" shows the vendor of a MAC address.

We have now acquired the MAC vendor range DC58BC, which is now listed at IEEE, see
https://regauth.standards.ieee.org/standards-ra-web/pub/view.html#registries and http://standards-oui.ieee.org/oui/oui.txt

My question is: where does OPNsense store this oui.txt? (I'd like to contribute a patch, so that future OPNsense version show the vendor "Thomas-Krenn.AG" for MAC addresses starting with DC:58:BC  ;)

Thx and best regards,
Werner

I can reproduce this issue, too. I'm also getting the error message:

Sep 3 13:59:36    suricata: [100118] <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "drop http $HOME_NET any -> $EXTERNAL_NET any (msg:"URLhaus Known malware download URL detected"; flow:established,from_client; content:"GET"; http_method; content:"/url=http://wordpress.p364918.webspaceconfig.de/614tiscfz/com/us|26|data=02|01|rcorm1@jcp.com|ec2a6ed25318490bd27608d6077bf11e|9c0ac0b90217468aa4322649cd6ed297|0|0|636704626242706015|26|sdata=g3qlynktc59ma3fllqbbfs0uwnigsem1mwi/cdfotvu=|26|reserved=0"; http_uri; depth:250; isdataat:!1,relative; content:"na01.safelinks.protection.outlook.com"; http_host; depth:37; isdataat:!1,relative; metadata:created_at 2018_08_21; reference:url, urlhaus.abuse.ch/url/45633/; classtype:trojan-activity;sid:80908733; rev:1;)^M" from file /usr/local/etc/suricata/opnsense.rules/abuse.ch.urlhaus.rules at line 5298

And in "clog /var/log/system.log":

Sep  3 14:01:45 OPNsense kernel: pid 6613 (suricata), uid 0: exited on signal 6 (core dumped)

Switching from Pattern matcher "Hyperscan" to "Aho-Corasick" resolves the issue that Suricata dies, but on a 100G WAN link the speed is dropping to 20G (small system with SSE3 capable Atom-CPU).

Disabling "abuse.ch/URLhaus" also fixes the issues, too (leaving "Hyperscan" activated).

I just wanted to provide those details, maybe this helps you to find the root cause of the issue (is not time critical to me).

Best regards,
Werner

[Updated as I have answered in the wrong thread]:

I'm having issues here, too.

See https://forum.opnsense.org/index.php?topic=9512.msg43639#msg43639 for my details.

We have just plugged in a X710-DA2 card in our FrontIO system - here you find the driver version for ixl:

Code Select Expand


root@OPNsense82:~/hw-analyse-frontio-mit-x710 # sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'
dev.igb.5.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.4.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.3.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.2.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.ixl.1.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.9.9-k
dev.ixl.0.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.9.9-k
dev.ix.1.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k
dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k
root@OPNsense82:~/hw-analyse-frontio-mit-x710 # sysctl dev.ixl.0.fw_version
dev.ixl.0.fw_version: fw 6.0.48442 api 1.7 nvm 6.01 etid 800035cf oem 1.262.0
root@OPNsense82:~/hw-analyse-frontio-mit-x710 #


So this updated ixl version 1.9.9-k comes from FreeBSD 11.2: https://www.freebsd.org/releases/11.2R/relnotes.html#drivers-device and https://svnweb.freebsd.org/base?view=revision&revision=333343

Driver 1.9.9-k fixes together with the X710 firmware NVM 6.01 issues with LACP - see https://www.intel.com/content/dam/www/public/us/en/documents/brief/lacp-config-guide.pdf - see page 17 which says:

Each Intel ® Ethernet 700 Series Network Adapter has a built-in hardware LLDP engine, which is
enabled by default. The LLDP Engine is responsible for receiving and consuming LLDP frames, and also
replies to the LLDP frames that it receives. The LLDP engine does not forward LLDP frames to the
network stack of the Operating System.
LACP may not function correctly in certain environments that require LLDP frames containing LCAP
information to be forwarded to the network stack. To avoid this situation, the user must disable the
Intel ® Ethernet 700 Series Network Adapter's hardware LLDP engine.
The following subsections provide instructions on how to disable the Intel ® Ethernet 700 Series
Network Adapter's hardware LLDP engine.

As deactivating the included LLDP engine had issues with firmwares < 6.01 - see issue #70 on page 50 of https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xl710-10-40-controller-spec-update.pdf - you need to have firmware version NVM 6.01 on such a nic (like me in the example output above). See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221530 for details on that.

And here you can find further information on the 1.9.9-k driver: https://reviews.freebsd.org/D14985

@mimugmail: I'm not sure, but maybe this (by default enabled) LLDP engine of the X710 cards could be a cause of the IPS issues of this card. I haven't tested IPS yet with this LLDP engine switched of, but maybe this could help.

GBICs (SFP modules) must be from Intel to be fully supported by the Intel NIC chips:

Intel® Ethernet SFP+ SR Optics and Intel® Ethernet SFP+ LR Optics are the only 10-Gbps optical modules supported.

Sources:

• SFP+ Modules, SFP Modules, and Cables Compatible with Intel® Ethernet Converged Network Adapter X520 Series: https://www.intel.com/content/www/us/en/support/articles/000005528/network-and-i-o/ethernet-products.html
• Compatible SFP+ Modules, SFP Modules, and Cables for Intel® Ethernet Server Adapter X710 Series: https://www.intel.com/content/www/us/en/support/articles/000007045/network-and-i-o/ethernet-products.html

It seems that the FW of the NICs checks the installed SFP module EEPROM for 03h SFF-8472 identifier and refuse to work if that check failed. See https://www.thomas-krenn.com/de/wiki/Intel_10_Gigabit_X710-DA2_SFP%2B_state_DOWN_beheben and https://communities.intel.com/message/402917 for further background information (although these links refer to X710 cards, they should be valid for the other Intel 10Gbit cards, too).

Thx for the feedback. And no, currently I don't have any problem reports right now 8)
I'm just curious...

Thank you Franco,

OPNsense 18.7-amd64 shows these versions for me:

Code Select Expand


# sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'
dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.6.1-k
dev.igb.4.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k


Does anybody have a system with ix driver by the hand to check?

Or is there also a way to check the version when the driver is not loaded (like in my case when I want to check the ix driver version, but I don't have a system with an appropriate NIC by the hand)?

Is there an easy way to query driver versions in FreeBSD? OPNsense 18.7 has "several of its [FreeBSD 11.2] Intel NIC driver updates included".

Now I'd like to query the driver versions of ix, igb, ...
Is there an easy way to do this? (like it can be done under Linux using "modinfo")

Best regards,
Werner

Thank you for your answer Franco (sorry for being late with my reply)

I happens when I run the installer locally (not via SSH).

I did not understand how I could trace it / what steps I should do to trace it.

I could run tests on Friday when you write me the commands which I should run.

Best regards,
Wwerner