Messages

1

22.1 Production Series / Re: OPNsense router keeps crashing / becoming unresponsive

« on: February 09, 2022, 09:14:45 am »

I also had issues with PowerD on a recent i7 CPU (2019, same year as yours) - disabling it resolved a similar-ish problem for me, relying on the CPU scaling instead,

Had no issues with a much older Celeron J1900 using PowerD, until I upgraded to the above.

2

Hardware and Performance / Re: Deciso DEC850 - CPU speed goes up only to 1500MHz instead of 3100MHz?

« on: February 01, 2022, 11:48:52 am »

My Intel i7-9700 does as well:

Code: [Select]

dev.cpu.7.freq_levels: 3000/-1
dev.cpu.7.freq: 4497
dev.cpu.6.freq_levels: 3000/-1
dev.cpu.6.freq: 4497
dev.cpu.5.freq_levels: 3000/-1
dev.cpu.5.freq: 4497
dev.cpu.4.freq_levels: 3000/-1
dev.cpu.4.freq: 4497
dev.cpu.3.freq_levels: 3000/-1
dev.cpu.3.freq: 4497
dev.cpu.2.freq_levels: 3000/-1
dev.cpu.2.freq: 4497
dev.cpu.1.freq_levels: 3000/-1
dev.cpu.1.freq: 4497
dev.cpu.0.freq_levels: 3000/-1
dev.cpu.0.freq: 4497
I'm just using hwpstate_intel, rather than PowerD - I had issues with that with this CPU, in 21.7 yet to try PowerD again on 22.1

3

Tutorials and FAQs / Re: Check_MK Agent setup

« on: January 24, 2022, 02:28:19 pm »

Wonderful, many thanks @NilsS

4

General Discussion / Re: Virgin Media UK - 'Super' Hub 4 - Modem Mode

« on: January 22, 2022, 10:30:50 pm »

Replacement UPS should arrive later, so I'll have to restart the Hub again. 

Well I swapped in the new UPS, I left the Virgin Hub 4 powered off for about 25 minutes as it was last night when the previous UPS failed, poweedr it up...

... opnsense got an IP address first time using the settings above. 

Fingers crossed.  Guess time will tell, or if anyone else who has/is having problems can test.

5

22.1 Production Series / Re: RSS Support Yet?

« on: January 22, 2022, 07:06:22 pm »

Works for me.

root@fw00:~ # grep "isr\|rss" /boot/loader.conf.local
net.isr.bindthreads=1
net.isr.maxthreads=8
net.inet.rss.enabled=1
net.inet.rss.bits=3

Sticky post at the top of this very forum?

6

General Discussion / Re: Virgin Media UK - 'Super' Hub 4 - Modem Mode

« on: January 22, 2022, 01:11:59 pm »

From: https://www.freebsd.org/cgi/man.cgi?query=dhclient.conf&sektion=5&n=1

opnsense defaults:

- Timeout: 60
- Retry: 15
- Select-timeout: 0
- Reboot: 1
- backoff cutoff: empty
- initial interval: 1

So, for as much my benefit to work through this as potentially anyone else, this means:

- 60 seconds must pass, before the firewall decides it's not going to be able to contact a server

- After the 60 second timeout has passed, if there are static leases (which we shouldn't have) or any leases not yet expired (possibly/likely), client will loop through them trying to validate them

- After the 60 second timeout has passed, if there are no valid static leases or expired leases, client will restart the protocol after retry (15) interval

- 15 seconds must pass after opnsense has determined there is no DHCP server present, before it tries again to contact a DHCP server (default is 5 minutes in FreeBSD)

- select-timeout only seems to be relevant if there is more than one DHCP server on a network, not sure this is relevant in this case, especially as we are excluding the local requests from 192.168.100.254 so we want the first 'other' offer we receive (but get it with a zero setting anyway)

- When the client is restarted, 1 second must pass before it gives up trying to get its old address again (default 10 seconds)

- To try to prevent many clients making requests in lockstep, opnsense uses a backoff algorithm with some
randomness, default for this is 2 minutes.  This is called backoff cutoff.

- There is 1 second between the first attempt to reach a server and the second attempt, each message sent the interval is incremented by twice the current interval multiplied by a random number between 0 and 1.  If it is greater than the backoff cut off (2 minutes) it is set to that amount (2 minutes)

If we accept as 'true' that a device connecting in modem mode must make a DHCP request within a 15 second window, which perhaps could be 30 seconds to a few minutes after the Hub 4 starts up, we need much more consistent DHCP requests over this period to ensure we make at least 1 request in that 15 second window.

I believe it is this continually sliding and partially random interval that causes problems and perhaps adds some weight to why Asus also call theirs 'continuous' - in that it makes requests in a metronomic type fashion, to ensure a request is made within that 15 second window.

So if we assume that 'Asus 12Hz Continous DHCP' is not 12Hz, so not 12 requests a second (!!!) but more likely 12 per minute/every 5 seconds, perhaps reasonable options would be:

- Timeout: 4
- Retry: 1
- Select-timeout: 0
- Reboot: 1
- backoff cutoff: 1
- initial interval: 1
- Reject Leases From: 192.168.100.254

.. this should also potentially help as/when the DHCP IP address offer changes, due to reboot being set to 1.   

7

General Discussion / Re: Virgin Media UK - 'Super' Hub 4 - Modem Mode

« on: January 22, 2022, 11:50:57 am »

After doing a bit more digging, some seem to suggest that the Asus 12Hz is 12 per minute - every 5 seconds.... which means it isn't 12Hz at all...

But I don't have an Asus router to check.  Replacement UPS should arrive later, so I'll have to restart the Hub again. 

Will perhaps try modifying my DHCP options to replicate every 5 seconds, rather than every 1, I think the opnsense default is 15 seconds (but then extends the re-try if no response, so you have to time it correctly - repeatedly every X seconds would be better potentially for the Hub 4).

.. especially if it is true that there is a 15 second window when the Hub 4 will hand out an IP in modem mode and then ignores all others.

8

General Discussion / Re: Google Fiber 2GBps

« on: January 22, 2022, 09:50:29 am »

Whilst I cannot say for sure, I'm still waiting for a 2.5Gbps port cable modem... somethings to perhaps consider:

You'd need an SFP+ module in the card for anything over 1, but I believe some are picky about 2.5Gbps.

Additionally, possibly a firmware upgrade to the X710.  This can be done 'online' from FreeBSD/opnsense using the 'intel-nvmupdate' package from ports:

https://www.freshports.org/sysutils/intel-nvmupdate/

Code: [Select]

/usr/local/sbin/nvmupdate

Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.35.33.4
Copyright (C) 2013 - 2020 Intel Corporation.


WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.
Inventory in progress. Please wait [.....|****]


Num Description                          Ver.(hex)  DevId S:B    Status
=== ================================== ============ ===== ====== ==============
01) Intel(R) Ethernet Converged         8.00(8.00)   1572 00:001 Up to date
    Network Adapter X710
There does appear to be a later version on the Intel site, although when I recently got my box with an X710 card I used the ports version:

https://www.intel.com/content/www/us/en/download/18636/non-volatile-memory-nvm-update-utility-for-intel-ethernet-adapters-700-series-freebsd.html

Link state can be lost, requiring a reboot after completed, so if it is only your WAN on the card it shouldn't be too much of a problem - as you could still access the firewall - if not, completing over a local/console connection is probably advisable, so you can tell when it has actually finished if you need to reboot.

9

General Discussion / Re: SMTP server on Monit

« on: January 22, 2022, 09:36:25 am »

Hi there,

I use a gmail account for mine - configured with:

https://support.google.com/mail/answer/7126229?hl=en-GB#zippy=%2Cstep-change-smtp-other-settings-in-your-email-client

... port 465, SSL enabled, etc.

However, if you just put your regular gmail password in, you'll see the below in the Monit logs:

Code: [Select]

SMTP: Mailserver response error -- 534 5.7.9 https://support.google.com/mail/?p=InvalidSecondFactor o18sm2679679ejb.111 - gsmtp
Once you configure an App specific password, and use this in Monit instead of your main/actual gmail password, then it works:

https://support.google.com/mail/search?q=application+specific+password&from_promoted_search=true

10

General Discussion / Virgin Media UK - 'Super' Hub 4 - Modem Mode

« on: January 22, 2022, 09:00:03 am »

Recently upgraded my Virgin Media connection to Gig1 and with it (unfortunately) I was provided with a Hub 4 - from day 1, it has been a PITA.  It took me hours and hours to finally get an IP address from modem mode to opnsense.

Running the device in Modem Mode, it is extremely picky about when/if it will let opnsense have an IP - or the DHCP servers are - certainly with the default DHCP settings, even though the opnsense defaults request more frequently than the FreeBSD defaults.

There are various reports of problems, with people using opnsense, pfSense, Asus, you name it.  It seems to be 'pot luck' whether the device behind the modem gets an IP address or not, the suggested steps of:

- Power off the modem
- Restart your main firewall/router
- Give it a few minutes
- Power on the modem

.. seems to work in some cases, but not others.  I do firmly believe there is still an element of luck to whether this works or not. 

I read somewhere that if the DHCP request doesn't complete in a 15 second window, after the modem has booted, it basically ignores all other requests after.  This was quoted from a Virgin engineer, so who knows....

I thought I had cracked it last time I had problems, when I filtered 192.168.100.254 in 'Reject Leases From' - otherwise opnsense ends up getting a 192.168.100.x address instead.  Note, previous modems you needed to filter 192.168.100.1 (I believe).

Running tcpdump, filtering for DHCP requests/replies, the requests are being made, but just no response or not a completed response:

Code: [Select]

tcpdump -i eth0 port 67 or port 68 -e -n -vv
Last night the UPS on my cable modem died, opnsense failed over beautifully to 4G and until I heard a whining noise on my way to bed coming from the garage I didn't realise it had failed (Muted PushOver notifications after 10PM).  So I removed the UPS, powered on the cable modem and went to bed - this morning, it still hadn't got an IP and was still running on 4G.

I tried restarting the modem, and opnsense, using the basic procedure above that is reported to (sometimes) work.  But it didn't.

In the end, I discovered that Asus now has some '12Hz DHCP' option - to supposedly fix similar issues, where the device doesn't get an IP - which requests an IP 12 times a second (12Hz)?!?!

https://www.asus.com/my/support/FAQ/1043591/

Frustrated, I ended up putting '1' in every box in the DHCP options for the WAN interface - along with filtering 192.168.100.254 from DHCP replies - to try to make DHCP requests as frequent as possible.  Rebooted the Hub 4....and it got an IP address first time.

... I don't really want to test this again, just yet, but will update this post with any further developments next time I come across it.

Aside from this, does anyone have any idea how to replicate the Asus 12Hz option?  I do see some mention that others with Starlink and other cable modems also need to use this feature, so it doesn't seem to be Virgin specific.

11

Hardware and Performance / Re: Hardware Opinion? Supermicro Xeon D-1518 Mini 1U Rackmount

« on: January 21, 2022, 07:15:12 pm »

I am very fond of the Xeon-D series of CPUs. They are not cheap - you can get a better bang for the buck if you pick AMD. But all my applications on FreeBSD and ESXi run just rock solid. If you don't run Linux but FreeBSD, i.e. OPNsense, TrueNAS, ... I can really recommend them. Especially the embedded boards by Supermicro ...

I'll second this - I have 2 Broadberry devices, basically Supermicro, in a Proxmox cluster, in a rack under the stairs, they're powerful enough, don't draw too much power, quiet/cool enough, 2 x 10Gb ports and 2 x 1Gb ports...whats not to like.

Albeit I don't run Opnsense on the proxmox cluster, have a stand alone device for that (and house automation....been burnt waaaay too many times when tinkering and not being able to turn the lights off!!!!

... Xeon D, solid choice.

12

22.1 Production Series / i7 9700 (presumed) CPU issues resolved with 22.1

« on: January 21, 2022, 08:11:47 am »

Just adding this here, for anyone in a similar situation...

Was previously running a Qotom J1900 for the last 7+ years, happily on 21.7.  Recently upgraded the firewall, now with an i7 9700 CPU.

Upgrade was easy enough, export the config, edit the interface names for the new device in the export file, import into the new box.  Initially started out with 21.7 on the new box.

But, whilst using PowerD with the J1900 was not a problem, with the i7 9700 it caused it to crash usually under a reasonably small amount of load - on 21.7 - fans would go to 100% and required physically power cycling. 

Turning off PowerD resolved the issue, but the CPU frequency (dev.cpu.0.freq) would always be at 1400 - even using 'stress' the CPU would seemingly not scale.

Upgrading to 22.1 which I believe uses hwpstate_intel instead for power management, resolves the issue. 

dev.cpu.x.freq when running stress shows it is also using up to the 'Turbo Boost' frequency '4497', so the CPU can work at maximum when required and drop back down (dev.cpu.0.freq: 897). 

Performance vs power can be tuned with dev.hwpstate_intel.*.epp

I did come across a FreeBSD post, where they were essentially saying it was legacy and should probably be removed - so I haven't attempted PowerD again, but will for testing again at some point.  So it seems 'ok' for older CPUs, but for more modern CPUs PState seems to be the way to go.

13

22.1 Production Series / Re: Throughput performance issues/shaper problems

« on: January 20, 2022, 08:35:26 pm »

My understanding is that with net.inet.ip.dummynet.io_fast enabled, as it is by default, that when the pipe bandwidth is not exhausted packets are forwarded directly without actually going through the shaper.

The problem arises when you're at capacity/saturation, some packets will be forwarded 'fast', some won't and that:

'saturation/desaturation of pipe occurs several times in one second so it seems then we get reorders on every transition)'

Ideally, we could turn off io_fast to always force all packets through the shaper, but this option is commented in the code.  So the only way to work around it, is to force all packets to always go through the shaper - adding a 1ms delay to all pipes makes that happen.

14

22.1 Production Series / Re: Throughput performance issues/shaper problems

« on: January 20, 2022, 05:59:15 pm »

Might be worthwhile experimenting with the 1ms delay on the pipes - to prevent out of order packets - I found it helped.  Unless you're one of these gamers that goes nuts over 1ms

It seems the option to disable IO Fast still doesn't do anything, as it is commented out in line 946 in the code (same as earlier versions of FreeBSD)

https://github.com/opnsense/src/blob/stable/13/sys/netpfil/ipfw/ip_dn_io.c

As the chap in the pfsense post mentions originally:

"Since net.inet.ip.dummynet.io_fast does split path of packets for saturated/unsaturated pipe mode, then this setting is likely to be responsible for packet reordering. (traffic is very bursty for TCP without pacing or IPERF3 UDP test, so saturation/desaturation of pipe occurs several times in one second, so it seems then we get reorders on every transition)

But setting of net.inet.ip.dummynet.io_fast=0 has no effect, net.inet.ip.dummynet.io_pkt_fast is still increasing. Explanation is very simple:
io_fast check is commented in dummynet source code:

if (/*dn_cfg.io_fast &&*/ m *m0 && (dir & PROTO_LAYER2) 0 ) {"

15

22.1 Production Series / Re: Throughput performance issues/shaper problems

« on: January 20, 2022, 05:16:19 pm »

Interesting, in the link they ended up with more or less what I did.

However, whilst I might be wrong, I was under the assumption that it reserved some bandwidth for new connections/sessions?

So maybe with more concurrent bandwidth demands from devices, you might find the connection overloaded (and bufferbloat coming back again) if you're close to the upper end of the available bandwidth?

With the maximum throughput of gigabit ethernet at around 940Mbps, I'd perhaps question the 1000Mbits set in the link - unless the cable modem has a 2.5Gb port on the inside.... which is what I'm waiting for