Topics

[Reduced power consumption]

I happened to notice that FreeBSD has SATA Link Power Management disabled by default.

This is also the case with the Linux vanilla Kernel so far. However, numerous Linux distributions (e.g. Ubuntu as of Linux kernel 5.15) enable this for mobile/embedded chipsets using "SATA_LPM_POLICY=3".

For the upstream Linux kernel "SATA_LPM_POLICY=3" was also already under discussion: https://www.spinics.net/lists/linux-ide/msg62918.html Due to a hot-plug problem with AMD-based systems (there the SATA controller is identical for EPYC and Ryzen - thus it can't be determined via the PCIe IDs of the SATA controller whether it is a mobile/embedded chipset) this has not been implemented yet.

What are the advantages of enabled SATA Link Power Management?

• Reduced power consumption (approx. 1-1.8 watts for a system with one SATA SSD.)
• Lower waste heat, cooler SSD controller chips (e.g. 30 °C lower temperature of the SSD controller for an ATP AF120GSTIC-T22). Especially in summer this can be very advantageous for small fanless firewall systems, if such systems are located in unair-conditioned rooms and possibly the sun shines on the device.
  

What are the disadvantages of enabled SATA Link Power Management?

• Possible problems with hot-plug. This is not relevant for smaller/embedded firewalls with a M.2 SATA SSDs. Likewise, it does not affect rack systems that boot from M.2 SATA. The hot-plug limitation really only affects rack systems with hot-swap drive bays that have two SATA SSDs in a ZFS mirror.
• Slightly increased SATA I/O latency. When the link is in "Partial" power saving state, this is max. 10 microseconds. In the "Slumber" power saving state, this is 10ms max (but this is only activated if no I/O happens for a long time). See SATA AHCI Spec https://www.intel.com/content/www/us/en/io/serial-ata/serial-ata-ahci-spec-rev1-3-1.html chapter 8.2 Power State Mappings (Partial - Phy logic is powered, but in a reduced state. Exit latency is no longer than 10µs / Slumber - Phy logic is powered, but in a reduced state. Exit latency can be up to 10ms)

Under FreeBSD there are several SATA link power management options available - see https://man.freebsd.org/cgi/man.cgi?ahci(4)

• 0 -> Off (default)
• 1 -> Device Initiated Power Management (DIPM)
• 2 to 5 -> different options for Host Initiated Power Management (HIPM)

However, a combination of DIPM+HIPM would be ideal. This would achieve the highest energy savings. Linux has offered this since kernel 4.15 - see https://hansdegoede.livejournal.com/18412.html and https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f4ac6476945ff62939420bcf8266e39f8d5d54bd. This emulates the functionality of the Intel RST driver from Windows, which is explained in detail in this PDF: https://www-ssl.intel.com/content/dam/doc/reference-guide/sata-devices-implementation-recommendations.pdf. In the mid and long term, it would be ideal to implement this functionality (combination of DIPM+HIPM) in FreeBSD as well. However, this is beyond my qualifications. For now, it would already bring a lot to support DIPM.

Currently it is possible to enable DIPM for a SSD via Tuneable: hint.ahcich.0.pm_level = 1 (Attention: a system can have multiple SATA ports - if in doubt set the value for all of them). I have documented all details about this: https://www.thomas-krenn.com/en/wiki/Activate_SATA_Link_Power_Management_in_OPNsense

My questions to you in the OPNsense development:

• (for OPNsense 23.7): Currently OPNsense shows at tuneables hint.ahcich.0.pm_level, hint.ahcich.1.pm_level, hint.ahcich.2.pm_level, ... as "Type" the info "unsupported" in red. Could you change this to "boot-time" (in black)? Background: the red display of "unsupported" could discourage users to use this function.
• (later after OPNsense 23.7 release): Option in web interface to enable SATA link power management. Would probably also be solvable via plugin. So far I haven't developed a plugin, but I could imagine it.
  
• (for discussion in the medium term): Enable DIPM by default for mobile/embedded SATA chipsets. However, since there is the limitation with hot-plug, this would have to be well-considered. However, the advantage would be that thousands (or tens/hundreds of thousands - I don't know how many OPNsense systems with SATA are in use worldwide) would then potentially need 1 watt less energy each. Globally, this would result in a relevant CO2 saving. OPNsense could call itself "Green IT deleoped Software" or something like that or just emphasize that energy optimization (in this case practically without costs on network performance) is an important point.

What do you think?
Best regards, Werner

[WARNING: attempt to domain_add(netgraph) after domainfinalize]

Hi OPNsense team,

I did some in-depth testing today with two LTE modems (Quectel EG25-G, Quectel EG25-E, Huawei ME909u-521). All those modems work fine with OPNsense 20.1 (and they did with 19.7, 19.1, 18.7). With the current OPNsense 20.7 beta, the firewall reboots after I try to set the WAN connection to the LTE interface.

Steps to reproduce:

• Install the current OPNsense 20.7 beta using https://pkg.opnsense.org/FreeBSD:12:amd64/snapshots/OPNsense-devel-20.7.b-OpenSSL-vga-amd64.img.bz2
• Use igb0 as LAN, igb1 as WAN and have a working wired Internet connection on igb1
• Apply all updates as of today (May, 26th). Versions are then "OPNsense 20.7.b_108-amd64", "FreeBSD 12.1-RELEASE-p4-HBSD"
• Configure the modem via "Interfaces -> Point-to-Point -> Devices" as described in https://www.thomas-krenn.com/de/wiki/OPNsense_LTE_Verbindung#Konfiguration_Modem. For a Quectel modem use /dev/cuaU0.2, for a Huawei ME909u-521 use /dev/cuaU0.0
• Switch to "Interfaces -> Assignments" and configure for "WAN" the network port "ppp0". Click "Save"

Immediately after that, on the console there is the following output (in bold):
WARNING: attempt to domain_add(netgraph) after domainfinalize

About 2 seconds after that, a lot of outputs runs through the console and after a while the firewall reboots. On the next login the dashboard mentions "A problem was detected. Click here for more information." (I did this and submitted the output).

I have attached the four files here, too.

Any ideas what the root cause of this issue could be? (I think this is a general ppp0/LTE issue, as both the Quectel and Huawei modems show the same issue)

Best regards,
Werner

Hi all,

when you are setting up a firewall cluster, the documentation https://docs.opnsense.org/manual/how-tos/carp.html#setup-ha-sync-xmlrpc-and-pfsync currently says:

• First we should enable pfSync using our dedicated interface using the master firewall. Go to System ‣ High Availability ‣ Settings, enable pfSync and select the interface used for pfSync.

Also a diff of the sample configuration shows that pfsync is only enabled on node 1 (master):

• https://docs.opnsense.org/_downloads/64fce6febca41b922ab9906c47078aa9/Carp_example_master.xml
• https://docs.opnsense.org/_downloads/5b64c2fa6e30519189630e5dd22f0e58/Carp_example_backup.xml

My question:

• What happens e.g. when you are doing a firmware update and you switch the master role from node 1 to node 2?
• I assume that pf states are not synchronized again from node 2 -> node 1 when node 1 comes back up.
• I _think_ that pfsync should be enabled on node 2, too. pfsense suggests it in this way too, according to https://docs.netgate.com/pfsense/en/latest/book/highavailability/pfsync-overview.html#pfsync-overview "When pfsync is in use, pfsync settings must be enabled on all nodes participating in state synchronization, including secondary nodes, or it will not function properly."

Should we update the OPNsense documentation?

Best regards,
Werner

Hi all,

I have some feedback regarding CARP:

• a) pfsync: I think in https://docs.opnsense.org/manual/how-tos/carp.html#setup-ha-sync-xmlrpc-and-pfsync it should be stated that on the backup firewall the "Synchronize States" option should be set, too.
• b) Enter Persistent CARP Maintenance Mode: when clicking this on firwall 1 this sets currently net.inet.carp.demotion=240 and leaves advskew as long there is no reboot. After a reboot of firewall 1, net.inet.carp.demotion=0, but advskew is set to 254 - because of https://github.com/opnsense/core/blob/master/src/etc/inc/interfaces.inc#L1713. When following the steps described at https://docs.opnsense.org/manual/how-tos/carp.html#example-updating-a-carp-ha-cluster the advskew setting is still set to 254 on firewall 1 even after clicking "Leave Persistent CARP Maintenance Mode". When testing a WAN outage (unplug igb1) afterwards, only for the WAN IP, the other firewall gets MASTER, leaving LAN (igb0) as BACKUP there. So the Internet connectivity gets lost for clients. Only rebooting the firewall 1 or manually setting advskew back to 0 solves the issue. I'm not sure what would be the best way to fix this behavior. Any ideas?

Steps to reproduce issue b):

• Build an OPNsense HA cluster with two nodes, firewall 1 as MASTER and firewall 2 as BACKUP
• Click "Enter Persistent CARP Maintenance Mode" on firewall 1. The sysctl "net.inet.carp.demotion" will be set to 240. advskew is still 0 for all configured CARP interfaces.
• Do a reboot of firewall 1.
• After the reboot, on firewall 1 "net.inet.carp.demotion" is now 0 (not 240), but advskew for all CARP interfaces is set to 254 (query by "ifconfig | grep carp"). So advskew is set to 254, but the web interface shows still values of 0 in "Firewall -> Virtual IPs -> Settings".
• Clicking "Leave Persistent CARP Maintenance Mode" on firewall 1 does _not_ switch back the CARP IPs to firewall 1. firewall 2 is still MASTER, although I would expect that now there should be a switch-back to firewall 1 - according to the doc https://docs.opnsense.org/manual/how-tos/carp.html#example-updating-a-carp-ha-cluster
• Only after another reboot of firewall 1, advskew is again set to 0. But in my opinion this additional reboot of firewall 1 is unecessary when updating an OPNsense firewall cluster.

Best regards,
Werner

The status site "status_interfaces.php" shows the vendor of a MAC address.

We have now acquired the MAC vendor range DC58BC, which is now listed at IEEE, see
https://regauth.standards.ieee.org/standards-ra-web/pub/view.html#registries and http://standards-oui.ieee.org/oui/oui.txt

My question is: where does OPNsense store this oui.txt? (I'd like to contribute a patch, so that future OPNsense version show the vendor "Thomas-Krenn.AG" for MAC addresses starting with DC:58:BC  ;)

Thx and best regards,
Werner

Is there an easy way to query driver versions in FreeBSD? OPNsense 18.7 has "several of its [FreeBSD 11.2] Intel NIC driver updates included".

Now I'd like to query the driver versions of ix, igb, ...
Is there an easy way to do this? (like it can be done under Linux using "modinfo")

Best regards,
Werner

Hi all,

although I have no C3000 system yet, I'm just curious and would like to know whether OPNsense 18.1 has an updated ix/ixv driver to support the integrated X553 10GBit NICs of the C3000 Denverton CPUs? For CPU infos see https://ark.intel.com/products/codename/63508/Denverton and https://www.intel.com/content/www/us/en/design/products-and-solutions/processors-and-chipsets/denverton/ns/atom-processor-c3000-series.html

FreeNAS 11.1 has included updated drivers: https://redmine.ixsystems.com/issues/26292

FreeBSD 11.1 does not support the X553 NICs, it seems that FreeBSD 11-STABLE has support for them included:
https://lists.freebsd.org/pipermail/freebsd-announce/2017-September/001803.html (search for "553")
https://lists.freebsd.org/pipermail/freebsd-stable/2017-July/087387.html

Best regards,
Werner

[WAN link goes away]

(UPDATE #1: the issue has finally been fixed through a BIOS/UEFI-Firmware update, see posting #44 in this thread for details: https://forum.opnsense.org/index.php?topic=5511.msg28681#msg28681)

(UPDATE #2: after many days of tests, the problem came up again - see https://forum.opnsense.org/index.php?topic=5511.msg28781#msg28781 for details)

Hi all,

I'm trying to analyze a strange issue: sometimes (very rare, I was only able to reproduce the issue 2 times), the WAN link goes away. From a laptop behind the OPNsense firewall I am not able to ping the WAN's default gateway anymore. I still can access the OPNsense system.

Details:

• Hardware based on JBC390F541AA-19-B (System: http://www.jetwaycomputer.com/JBC390F541AA.html, Mainboard: http://www.jetwaycomputer.com/NF541.html, 2 4quad Port NICs: http://www.jetwayipc.com/content/?ADPIE1ILAN4_3868.html)
• current OPNsense version 17.1.9 with FreeBSD 11.0-RELEASE-p10
• LAN is on igb0, WAN on igb9
• An "ifconfig igb9 down" followed by an "ifconfig igb9 up" fixes the issue - like also described in this thread here: https://forum.opnsense.org/index.php?topic=4850.msg21238#msg21238
• I was able to reproduce the issue only two times. It occurred soon after booting the OPNsense machine. I do not get any hints in /var/log/system.log.
• I have attached a zip file with the output of various tools.
• Unfortunately I have not checked the ARP cache when the problem occurred. In case I can reproduce it again, I'll check that.
• I have also tried pfSense 2.3.4. I was not able to reproduce the error there, but this may have been just luck, or it works there 'cause they use the older FreeBSD 10.3 compared to FreeBSD 11.0.

Have you ever seen an issue like this?
Do you have any hints what I could do to further analyze the issue?

Thanks in advance and best regards,
Werner

Hi everyone,

I just noticed that "pfSense Community Edition version 2.5" will depend on a Intel CPU with AES-NI support:
https://www.netgate.com/blog/pfsense-2-5-and-aes-ni.html

Are there any plans at OPNsense, to remove support for non-AES-NI CPUs in the future, too?

Best regards,
Werner

Hi all,

first of all thank you for providing OPNsense :-)
I've installed OPNsense 17.1.4-amd64 on a server with two SATA SSDs, and I have set up a GEOM Mirror during installation. I have used the target "mirror/OPNSenseMirror" as you can see in this screenshot:
https://www.thomas-krenn.com/de/wikiDE/images/9/9b/OPNsense-Installation-GEOM-07-Select-a-Disk.png

I'm now looking for a way to monitor the status of the gmirror, and I have searched in the web interface for such an option (but I did not find a site or widget for that). I only found this https://github.com/mibuthu/opnsense-core/blob/master/src/sbin/gmirror_status_check.php on a GitHub fork.
In pfSense, there are options to monitor the status: https://www.thomas-krenn.com/de/wiki/PfSense_Software_RAID_1#.C3.9Cberwachung

My questions:

• Are there currently any options to monitor the status of a gmirror?
• If not: are there any options planned for this?
• If there are no options planned currently: should I open a github issue here: https://github.com/opnsense/core/issues ?

Thank you & best regards,
Werner